New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

keter failing to deliver content on load "peaks" #75

Closed
jgreve-github opened this Issue Feb 27, 2015 · 20 comments

Comments

Projects
None yet
3 participants
@jgreve-github

jgreve-github commented Feb 27, 2015

When refreshing a keter-deployed yesod webapp rapidly, keter fails to deliver content, bailing out with

Error connecting to gateway:

InvalidStatusLine "0"

This seems to be indepent of ssl, I tested both. Take a look at a quite raw out-of-the-box-stripped-down example at http://phreeknet.de -- refreshing this repeatedly results in sometimes no css (due to this problem) and sometime no html, instead the following message.
because keter does not seem to bind the hosted web app's 'real' instance to localhost, the raw yesod instance can be accessed at http://phreeknet.de:43125. There, only the css problem occurs (because that still gets served via keter), which sorts out the yesod instance itself as the problem.

I suspect this not only happens on rapid refresh but also on load "peaks", whereas if some clients simultaneously request data.

Please let me know if you need any further information.

$ bin/keter --version
keter version: 1.3.8

Great work so far, though :)

Jan

@creichert

This comment has been minimized.

Show comment
Hide comment
@creichert

creichert Feb 27, 2015

Collaborator

Interesting, I haven't experienced this. It seems related to yesodweb/wai#249 and maybe yesodweb/wai#266.

I have quite a few sites up with Keter now and I can't reproduce this on a single one. I can reproduce it on your site, though.

It would be nice if you could provide the source for a Keter bundle we can reproduce the issue with.

Collaborator

creichert commented Feb 27, 2015

Interesting, I haven't experienced this. It seems related to yesodweb/wai#249 and maybe yesodweb/wai#266.

I have quite a few sites up with Keter now and I can't reproduce this on a single one. I can reproduce it on your site, though.

It would be nice if you could provide the source for a Keter bundle we can reproduce the issue with.

@creichert creichert added the bug label Feb 27, 2015

@creichert

This comment has been minimized.

Show comment
Hide comment
@creichert

creichert Feb 27, 2015

Collaborator

Can you verify which version of warp-tls, warp and wai you have installed?

Collaborator

creichert commented Feb 27, 2015

Can you verify which version of warp-tls, warp and wai you have installed?

@creichert

This comment has been minimized.

Show comment
Hide comment
@creichert

creichert Feb 27, 2015

Collaborator

One more note. I think the cause is the chunked transfer-encoding (as stated by the issue). Once we can verify your versions I can look at possibly pinning the version to incorporate the fix in warp-tls. It does look like you are using a pretty new version of Warp, though.

Collaborator

creichert commented Feb 27, 2015

One more note. I think the cause is the chunked transfer-encoding (as stated by the issue). Once we can verify your versions I can look at possibly pinning the version to incorporate the fix in warp-tls. It does look like you are using a pretty new version of Warp, though.

@jgreve-github

This comment has been minimized.

Show comment
Hide comment
@jgreve-github

jgreve-github Feb 28, 2015

The following versions are installed in the sandbox that compiled keter:

warp-3.0.9.2
warp-tls-3.0.2
wai-3.0.2.3

As to the yesod source, I will tar it, upload it and post a link here, yet: it is yesod init --bare, I only edited the homepage.hamlet file, the homepage.julius file and removed the route and handler for posting to /.

edit: And yes, I made sure to get everything up-to-date before submitting a possibly already fixed bug. I first experienced the problems with an "up to date" haskell-platform from ubuntu 14.04 LTS, which is naturally quite out of date; I then installed the current haskell platform release and recompiled everything to make sure to have the most current cabal had to offer.

jgreve-github commented Feb 28, 2015

The following versions are installed in the sandbox that compiled keter:

warp-3.0.9.2
warp-tls-3.0.2
wai-3.0.2.3

As to the yesod source, I will tar it, upload it and post a link here, yet: it is yesod init --bare, I only edited the homepage.hamlet file, the homepage.julius file and removed the route and handler for posting to /.

edit: And yes, I made sure to get everything up-to-date before submitting a possibly already fixed bug. I first experienced the problems with an "up to date" haskell-platform from ubuntu 14.04 LTS, which is naturally quite out of date; I then installed the current haskell platform release and recompiled everything to make sure to have the most current cabal had to offer.

@jgreve-github

This comment has been minimized.

Show comment
Hide comment
@jgreve-github

jgreve-github Feb 28, 2015

A link to the demo source requested is here: https://www.dropbox.com/s/40n988dg84z31r4/demosource.tgz?dl=0

This is a full snapshot of the directory the yesod instance is at home for development. As soon as the upload is complete, the link should be working fine. Please let me know when I can remove the file again.

jgreve-github commented Feb 28, 2015

A link to the demo source requested is here: https://www.dropbox.com/s/40n988dg84z31r4/demosource.tgz?dl=0

This is a full snapshot of the directory the yesod instance is at home for development. As soon as the upload is complete, the link should be working fine. Please let me know when I can remove the file again.

@jgreve-github

This comment has been minimized.

Show comment
Hide comment
@jgreve-github

jgreve-github Mar 1, 2015

BTW, I re-enabled ssl and it feels like this increases the probability of bailing out. I thus suspect some sort of a timing problem.

jgreve-github commented Mar 1, 2015

BTW, I re-enabled ssl and it feels like this increases the probability of bailing out. I thus suspect some sort of a timing problem.

@creichert

This comment has been minimized.

Show comment
Hide comment
@creichert

creichert Mar 2, 2015

Collaborator

@snoyberg do you think this is related to yesodweb/wai#249? He seems to have an up-to-date warp and warp-tls.

Collaborator

creichert commented Mar 2, 2015

@snoyberg do you think this is related to yesodweb/wai#249? He seems to have an up-to-date warp and warp-tls.

@snoyberg

This comment has been minimized.

Show comment
Hide comment
@snoyberg

snoyberg Mar 2, 2015

Owner

It sounds more likely to be a bug in http-client; it looks like with connection sharing in the connection manager, the final chunked header is not being flushed from the socket and is being read as the first chunk in the next connection. I did recently fix a bug or two in shared connection, can you confirm the version of http-client, http-client-tls, and http-conduit used to build keter?

Owner

snoyberg commented Mar 2, 2015

It sounds more likely to be a bug in http-client; it looks like with connection sharing in the connection manager, the final chunked header is not being flushed from the socket and is being read as the first chunk in the next connection. I did recently fix a bug or two in shared connection, can you confirm the version of http-client, http-client-tls, and http-conduit used to build keter?

@snoyberg

This comment has been minimized.

Show comment
Hide comment
@snoyberg

snoyberg Mar 2, 2015

Owner

@creichert The issue you linked to could theoretically be related, but it's highly unlikely. That was a very specific and ugly bug that should be completely fixed.

Owner

snoyberg commented Mar 2, 2015

@creichert The issue you linked to could theoretically be related, but it's highly unlikely. That was a very specific and ugly bug that should be completely fixed.

@jgreve-github

This comment has been minimized.

Show comment
Hide comment
@jgreve-github

jgreve-github Mar 2, 2015

@snoyberg, here you go:

http-client-0.4.7.1
http-client-tls-0.2.2
http-conduit-2.1.5

jgreve-github commented Mar 2, 2015

@snoyberg, here you go:

http-client-0.4.7.1
http-client-tls-0.2.2
http-conduit-2.1.5
@jgreve-github

This comment has been minimized.

Show comment
Hide comment
@jgreve-github

jgreve-github Mar 2, 2015

By the way, I noticed this also happens with yesod devel; without the yesod instance going down for compile, the site refreshing and dishing out motivational messages comes up sometimes.

I suspect that the development server is (partly) using the same technique keter does. Maybe that info helps narrowing it down further. The http-* versions are the same as for the keter sandbox.

jgreve-github commented Mar 2, 2015

By the way, I noticed this also happens with yesod devel; without the yesod instance going down for compile, the site refreshing and dishing out motivational messages comes up sometimes.

I suspect that the development server is (partly) using the same technique keter does. Maybe that info helps narrowing it down further. The http-* versions are the same as for the keter sandbox.

snoyberg added a commit that referenced this issue Mar 3, 2015

@snoyberg

This comment has been minimized.

Show comment
Hide comment
@snoyberg

snoyberg Mar 3, 2015

Owner

You're correct in your guess, it's mostly handled by the http-reverse-proxy library.

It's going to be difficult to debug this, as I've never encountered these issues myself, and can't reproduce them on my system. I'd like to start off by narrowing down where the problem is. I've pushed a commit to the separate-manager branch of keter, which uses a separate connection manager for each request. It's a huge pessimization, and therefore shouldn't be released, but if it solves the issue for you, it will give us a much better idea of where the bug is coming from.

Can you try using that version of keter and see what happens?

Owner

snoyberg commented Mar 3, 2015

You're correct in your guess, it's mostly handled by the http-reverse-proxy library.

It's going to be difficult to debug this, as I've never encountered these issues myself, and can't reproduce them on my system. I'd like to start off by narrowing down where the problem is. I've pushed a commit to the separate-manager branch of keter, which uses a separate connection manager for each request. It's a huge pessimization, and therefore shouldn't be released, but if it solves the issue for you, it will give us a much better idea of where the bug is coming from.

Can you try using that version of keter and see what happens?

@jgreve-github

This comment has been minimized.

Show comment
Hide comment
@jgreve-github

jgreve-github Mar 3, 2015

@snoyberg, I cloned the repo, switched branches and compiled, deployed and: everything seems fine.

jgreve-github commented Mar 3, 2015

@snoyberg, I cloned the repo, switched branches and compiled, deployed and: everything seems fine.

@snoyberg

This comment has been minimized.

Show comment
Hide comment
@snoyberg

snoyberg Mar 3, 2015

Owner

OK, so then this is seemingly an http-client bug. Next step will be
creating a reproducing test case. I'll see if I can put something together
tomorrow. If I can't reproduce, I may ask you to try running such a program
on your system.

On Tue, Mar 3, 2015 at 8:28 PM jgreve-github notifications@github.com
wrote:

@snoyberg https://github.com/snoyberg, I cloned the repo, switched
branches and compiled, deployed and: everything seems fine.


Reply to this email directly or view it on GitHub
#75 (comment).

Owner

snoyberg commented Mar 3, 2015

OK, so then this is seemingly an http-client bug. Next step will be
creating a reproducing test case. I'll see if I can put something together
tomorrow. If I can't reproduce, I may ask you to try running such a program
on your system.

On Tue, Mar 3, 2015 at 8:28 PM jgreve-github notifications@github.com
wrote:

@snoyberg https://github.com/snoyberg, I cloned the repo, switched
branches and compiled, deployed and: everything seems fine.


Reply to this email directly or view it on GitHub
#75 (comment).

@jgreve-github

This comment has been minimized.

Show comment
Hide comment
@jgreve-github

jgreve-github Mar 3, 2015

@snoyberg, you mentioned you were not able to reproduce on your system; I cleaned up everything before, so a vanilla current Haskell-platform and a "cabal sandbox init;cabal install keter" should work for reproducing. It's a vanilla, up-to-date Ubuntu amd64 lts on both my dev server and phreeknet.de.

jgreve-github commented Mar 3, 2015

@snoyberg, you mentioned you were not able to reproduce on your system; I cleaned up everything before, so a vanilla current Haskell-platform and a "cabal sandbox init;cabal install keter" should work for reproducing. It's a vanilla, up-to-date Ubuntu amd64 lts on both my dev server and phreeknet.de.

@snoyberg

This comment has been minimized.

Show comment
Hide comment
@snoyberg

snoyberg Mar 3, 2015

Owner

This is most likely a race condition, which means that it's not just a
software issue, but a hardware-triggered issue.

On Tue, Mar 3, 2015 at 8:37 PM jgreve-github notifications@github.com
wrote:

@snoyberg https://github.com/snoyberg, you mentioned you were not able
to reproduce on your system; I cleaned up everything before, so a vanilla
current Haskell-platform and a "cabal sandbox init;cabal install keter"
should work for reproducing. It's a vanilla, up-to-date Ubuntu amd64 lts on
both my dev server and phreeknet.de.


Reply to this email directly or view it on GitHub
#75 (comment).

Owner

snoyberg commented Mar 3, 2015

This is most likely a race condition, which means that it's not just a
software issue, but a hardware-triggered issue.

On Tue, Mar 3, 2015 at 8:37 PM jgreve-github notifications@github.com
wrote:

@snoyberg https://github.com/snoyberg, you mentioned you were not able
to reproduce on your system; I cleaned up everything before, so a vanilla
current Haskell-platform and a "cabal sandbox init;cabal install keter"
should work for reproducing. It's a vanilla, up-to-date Ubuntu amd64 lts on
both my dev server and phreeknet.de.


Reply to this email directly or view it on GitHub
#75 (comment).

@jgreve-github

This comment has been minimized.

Show comment
Hide comment
@jgreve-github

jgreve-github Mar 3, 2015

I see. Both my dev server (a atom) and phreeknet.de (a tiny vserver) experience this. Would be a strange coincidence that both environments are affected.

jgreve-github commented Mar 3, 2015

I see. Both my dev server (a atom) and phreeknet.de (a tiny vserver) experience this. Would be a strange coincidence that both environments are affected.

@snoyberg

This comment has been minimized.

Show comment
Hide comment
@snoyberg

snoyberg Mar 8, 2015

Owner

I'm investigating a different avenue entirely right now: the 416 status codes on one of the static/tmp files. That looks like it may be contributing significantly to the problem.

Owner

snoyberg commented Mar 8, 2015

I'm investigating a different avenue entirely right now: the 416 status codes on one of the static/tmp files. That looks like it may be contributing significantly to the problem.

snoyberg referenced this issue in yesodweb/wai Mar 8, 2015

Improve range handling to return 416
Prevent parsing negative range end-points, truncate ranges to file size, and
correctly return 416 when requested range is outside file bounds.

snoyberg added a commit to yesodweb/wai that referenced this issue Mar 8, 2015

@snoyberg

This comment has been minimized.

Show comment
Hide comment
@snoyberg

snoyberg Mar 8, 2015

Owner

OK, apologies for the mis-diagnosis, this was all because of a bad commit I merged into warp a month back. The bug should now be completely fixed in warp 3.0.9.3. You'll need to update your project to use the new version of warp. Upgrading keter should be neither necessary nor sufficient (though upgrading isn't a bad idea). Can you give that a shot and let me know if it fixes your problem? I can no longer reproduce any of the symptoms I saw on my machine.

Owner

snoyberg commented Mar 8, 2015

OK, apologies for the mis-diagnosis, this was all because of a bad commit I merged into warp a month back. The bug should now be completely fixed in warp 3.0.9.3. You'll need to update your project to use the new version of warp. Upgrading keter should be neither necessary nor sufficient (though upgrading isn't a bad idea). Can you give that a shot and let me know if it fixes your problem? I can no longer reproduce any of the symptoms I saw on my machine.

@creichert

This comment has been minimized.

Show comment
Hide comment
@creichert

creichert Jul 25, 2015

Collaborator

@jgreve-github I'm going to close this issue, please don't hesitate to reopen if the issue persists.

Collaborator

creichert commented Jul 25, 2015

@jgreve-github I'm going to close this issue, please don't hesitate to reopen if the issue persists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment