Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Webhooks randomly fail with 408 timeout #5470

Closed
sbrl opened this issue Dec 4, 2018 · 11 comments

Comments

4 participants
@sbrl
Copy link

commented Dec 4, 2018

  • Gitea version (or commit ref): 1.6.0 (latest stable release
  • Git version: git version 2.17.1 (on the server)
  • Operating system: Ubuntu Server 18.04
  • Database (use [x]):
    • PostgreSQL
    • MySQL
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
    • No
    • Not relevant
  • Log gist: ?

Description

I've setup a webhook between my server and a server at a remote location, I'm getting random failures. Some webhooks go through fine, and others die for some strange reason.

Setup:

  • Git Server:
    • Gitea behind nginx
  • Remote server:

Gitea claims the following in the webhook log:

Delivery: Post https://ci.starbeamrainbowlabs.com/hooks/laminar-config-check: read tcp 5.196.73.75:54504->x.y.z.w:443: i/o timeout

...but the remote server's nginx claims that it's Gitea's fault for not sending a complete request in time:

5.196.73.75 - - [04/Dec/2018:20:48:05 +0000] "POST /hooks/laminar-config-check HTTP/1.1" 408 0 "-" "GiteaServer"

The remote server has a good-quality fibre line. The git server has a stable 100mb/sec connection from KimSufi / OVH. Why the timeout?

Annoyingly, it always works if I hit "test delivery" - it's only genuine push events that are causing problems.

Screenshots

selection_094

Edit: Nope, it doesn't always work with the "test delivery" button. I've just got this:

selection_095

....clicking the "test delivery" button again causes the next one to succeed though.

@sbrl

This comment has been minimized.

Copy link
Author

commented Jan 13, 2019

Anyone at all? @lunny? Someone?

@sbrl sbrl referenced this issue Jan 16, 2019

Open

Resend webhook payloads #4589

3 of 3 tasks complete
@barryp

This comment has been minimized.

Copy link

commented Jan 21, 2019

I'm getting a similar thing sort of randomly

Delivery: Post http://www.home:9100/hooks/deploy-site: read tcp 10.0.0.107:12078->10.0.0.101:9100: i/o timeout

If I try a Test Delivery, it usually works fine, although that could be because the first invocation already did some time-consuming downloads - and the second invocation can complete much quicker. I'm using the same webhook @sbrl is using, although not behind nginx. The script it invokes can take maybe 5-15 seconds, perhaps even longer. It'd be nice to know what the timeout was, and even better if it could be configurable.

@sbrl

This comment has been minimized.

Copy link
Author

commented Jan 23, 2019

I think the timeout is already configurable, @barryp. It's this in custom/app.ini:

[webhook]
DELIVER_TIMEOUT = 60

....60 is the value I have set. If I try the "test" delivery, it does work fine some of the time. If I try it manually with curl from the same host, it always works. I think it's definitely a bug in Gitea.

@sbrl

This comment has been minimized.

Copy link
Author

commented Jan 23, 2019

I can confirm that this is still an issue with the latest v1.7.0 release of Gitea.

@sbrl

This comment has been minimized.

Copy link
Author

commented Feb 4, 2019

Furthermore, I have a tcpdump capture that demonstrates the issue. It's a HTTPS connection, but hopefully still proves useful. Contact me (my email address is bugs at starbeamrainbowlabs dot com) to obtain said dump.

(Note to self: It's in my downloads)

@lunny

This comment has been minimized.

Copy link
Member

commented Feb 5, 2019

Can not reproduce on try.gitea.io and my instances.

@sbrl

This comment has been minimized.

Copy link
Author

commented Feb 6, 2019

@lunny Oh dear. That's awkward. Do you have an email address I can send this packet capture to, if you think it'd help?

If further details are needed, I could try doing it over regular http. If you've got any other suggestions as to what could potentially be the cause, I'd love to hear it and I'll investigate.

I'm currently investigating using the git post-receive hook instead and writing a bash script as a workaround.

@joyqat

This comment has been minimized.

Copy link

commented Mar 1, 2019

Have the same problem
image
It might be a network problem. Does Gitea have a "retry when failed" config?

@lunny

This comment has been minimized.

Copy link
Member

commented Mar 1, 2019

@sbrl You can send message to me on discord.

@stale

This comment has been minimized.

Copy link

commented Apr 30, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

@stale stale bot added the stale label Apr 30, 2019

@sbrl

This comment has been minimized.

Copy link
Author

commented May 1, 2019

Looks like I have issues with GitHub sending Webhooks to the same server, so I think it's probably a network issue - or something else on my end. Thanks for investigating though!

@sbrl sbrl closed this May 1, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.