[Feature] Maximum memory usage

[shirshak55]

We don't to use memory like an infinite resource and want to release memory if possible. Lets say if there are more than 100000 request it should only store latest 1000 request (can be set by humans).

I think we should provide this settings? And it seems httptoolkit also has memory leaks?

Today it was occupying 3 GB, I cleared it from ui. But the memory was not released as indicated by task manager?

[pimterry]

Lets say if there are more than 100000 request it should only store latest 1000 request (can be set by humans).

I'm not sure we need to actively limit this - people can easily clear the requests if they want to reduce data. I'm open to making it easier to manually clear some requests while keeping others, but I don't think HTTP Toolkit should automatically lose data for you (there are many cases where you really do want to record 10000s of requests for hours, and then look back through all of that data - if it disappears that would be really bad).

There could be an argument for showing a warning like "HTTP Toolkit has recorded X,000 requests and may be using more memory that you need. Would you like to clear some of them? [No] [Clear older than 5 minutes] [Clear all requests larger than 10KB]" or something like that.

it seems httptoolkit also has memory leaks

If you can reproduce this, where HTTP Toolkit is using lots of memory but containing very few requests, can you please capture a memory snapshot and share it? You can do so from the app dev tools, there's instructions here: https://developer.chrome.com/docs/devtools/memory-problems/heap-snapshots/.

From that data, we can see exactly what data is being held and why. That should make this easy to fix, but we definitely need a recording to be able to do that.

One thing to note though is that the memory is internally managed by Electron here, and that won't necessarily immediately release it when HTTP Toolkit does. If HTTP Toolkit stops using lots of memory, but there's lots of other free memory on the machine, Electron will keep holding the memory for a little bit of extra time in case we want to use it again. To see the real memory usage, it's best to clear the stored requests, then wait a minute or two.

[shirshak55]

@pimterry I think it should be the user who should decide it? I agree with you that there are scenarios where HT should save data. But, there are a few cases where it shouldn't. Maybe we can use current behavior as default and provide settings to the user? We see this in terminals too where users can specify the number of lines etc. And, these days I use httptoolkit to bypass TLS fingerprinting and I don't want to see memory to explode after long usage.

Yea, whenever the user clicks on the clear button, I think Electron must release memory because in my opinion there is no point in keeping such data in RAM? And, unfortunately, I don't see HTTP Toolkit memory being released even after 1-2 minutes.

[pimterry]

these days I use httptoolkit to bypass TLS fingerprinting and I don't want to see memory to explode after long usage.

If you're purely using HTTP Toolkit in use cases like that, you can hit the 'pause' icon on the view page. All traffic will keep being intercepted exactly as normal, but HTTP Toolkit just won't store or show it.

unfortunately, I don't see HTTP Toolkit memory being released even after 1-2 minutes

Can you share a heap snapshot please from a moment when HTTP Toolkit is using much more memory than expected? A snapshot before and after clearing memory would be even better if possible. I think it sounds like there's a memory leak somewhere, and that's probably the larger problem here.

[shirshak55]

@pimterry
I thought it would stop proxy too. Its a good thing that we have this feature :)

To be honest, HTTPToolkit hangs whenever I try to take a snapshot. It's impossible I guess.

During snapshot

[pimterry]

I don't think we should close this - there's clearly a memory issue here that's not resolved, and I'd like to get this fixed! I'm going to reopen it.

I thought it would stop proxy too. Its a good thing that we have this feature :)

👍

HTTPToolkit hangs whenever I try to take a snapshot. It's impossible I guess.

Ah yes, if you've got a 3.6GB heap that's going to be difficult to capture!

Is there way to reproduce the issue, but capture the heap snapshot when it starts growing unexpectedly but it's still just a few hundred MB? I.e. capture a snapshot initially, then intercept enough traffic to noticeably increase the JS heap size, clear the data in HTTP Toolkit, and if the heap size doesn't go down then capture a second snapshot. If you can do that, we can compare the two snapshots to see where the unexpected extra memory usage is coming from.

Is there a clear cause that triggers this? What kind of traffic are you typically intercepting when you see this issue? Does HTTP Toolkit use lots of memory on your machine if you intercept just one request, and leave the app open for a while?

Any memory leaks are absolutely fixable, we just need to work out what's actually being held in that extra memory.

[shirshak55]

Sure, I will test this tomorrow.

[shirshak55]

@pimterry

Unfortunately, there is no way to get a heap snapshot. HTTP Tool kit simply gets crashed.

And the way it uses memory kinda makes me sad. 4GB(which contains snapshot probably) and all the data were cleared.

I used this to reproduce this issue. https://github.com/shirshak55/reproduce-httptoolkit-249

[pimterry]

I've run the HTTP Toolkit UI in a normal Chrome window, and taken snapshots from a smaller heap (collecting just a couple of hundred requests, and then clearing) and I think I've found a couple of possible memory leak causes and patched those. I've just pushed a fix for this that seems to have improved things so that clearing the data properly releases all the relevant memory - can you give that a test and see if it improves this at all for you?

HTTP Toolkit should update in the background next time you start it, and you can see the current UI version by hovering over the HTTP icon. As long as it's at least 0da70674 or newer (see the commit log: https://github.com/httptoolkit/httptoolkit-ui/commits/master) then the fixes should be active.

[shirshak55]

Thanks. It looks better.