Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems with large files #24

Closed
marcusbuffett opened this issue Jul 30, 2020 · 8 comments
Closed

Problems with large files #24

marcusbuffett opened this issue Jul 30, 2020 · 8 comments
Labels
help wanted Extra attention is needed wontfix This will not be worked on

Comments

@marcusbuffett
Copy link

marcusbuffett commented Jul 30, 2020

I'm trying to proxy to a local web server, but it looks like ktunnel doesn't behave well with large files. Here's some output from curl:

❯ curl https://proxy.mbuffett.com/_next/static/chunks/main.js | wc -c
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  8 4295k    8  377k    0     0  73862      0  0:00:59  0:00:05  0:00:54 73862
curl: (18) transfer closed with 4012193 bytes remaining to read
386672

❯ curl localhost:8085/_next/static/chunks/main.js | wc -c            
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4295k  100 4295k    0     0   209M      0 --:--:-- --:--:-- --:--:--  209M
4398865

Please, no judgement for the 4MB JS file 😄 I do see these logs from ktunnel about buffer sizes:

INFO[2020-07-30 07:59:34] increasing buffer size to 65536              
INFO[2020-07-30 07:59:34] increasing buffer size to 131072

Wonder if there's anything I can do about this limitation? Thanks!

And awesome program, exactly what I was looking for and easy to set up

@omrikiei
Copy link
Owner

omrikiei commented Aug 2, 2020

Hi @marcusbuffett, thank you for the kind words!
I'm 98% positive that the failure is related to buffer limitations in GRPC, so I limited the buffer size and should release it under v1.3.0 sometimes this week. however, if you'd like to test it out before that would be great! you can do so by cloning and building from branch v_1_3, and explicitly call the server version with the -i flag
for example:

./ktunnel expose -i quay.io/omrikiei/ktunnel:v_1_3

@marcusbuffett
Copy link
Author

No dice, unfortunately :/ I tried with a 400kb file:

❯ fallocate -l 400000 400k.txt
❯ curl http://proxy.mbuffett.com/static/400k.txt --output /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 32  390k   32  127k    0     0   4237      0  0:01:34  0:00:30  0:01:04     0

curl gets stuck there and stays at the same % received. Then eventually gives:

curl: (18) transfer closed with 269227 bytes remaining to read

@omrikiei omrikiei added help wanted Extra attention is needed wontfix This will not be worked on labels Aug 9, 2020
@omrikiei
Copy link
Owner

hi @marcusbuffett, I'm finding it difficult to make time to troubleshoot and debug this issue, so it might take longer. I'm sorry :(

@marcusbuffett
Copy link
Author

No worries at all, I’m happily using ktunnel for other purposes currently

@jaredallard
Copy link
Contributor

I'm also running into this. Happy to help, but not 100% sure where to go 😢

@jaredallard
Copy link
Contributor

I fixed this here: https://github.com/jaredallard/ktunnel/tree/jaredallard/feat/context

However, I put a lot of other changes in this branch, so I need to clean it up to make an actual PR.

@omrikiei
Copy link
Owner

@marcusbuffett, with @jaredallard 's contribution now merged and released in v1.3.0 - could you kindly check if the issue persists?

@omrikiei
Copy link
Owner

Just managed to test with 1.8G file - it works. so closing the issue..
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

3 participants