New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittently seeing wrong port #12

Closed
chmac opened this Issue Nov 29, 2013 · 7 comments

Comments

Projects
None yet
3 participants
@chmac

chmac commented Nov 29, 2013

On our production server, with mod_rpaf installed, we intermittently see requests where the port has been set to 443, even though the request has no X-Port or X-Forwarded-Port header, and reached apache via port 80.

Both the server logs and PHP's $_SERVER['SERVER_PORT']; variable show 443. It happens anywhere from 3 in 20 to 12 in 20 times.

Any advice on how to debug the issue?

@chmac

This comment has been minimized.

Show comment
Hide comment
@chmac

chmac Dec 16, 2013

Managed to reproduce this issue consistently.

  • Install a fresh Ubuntu 12.04 server (EC2 / Amazon / Virtualbox / whatever)
  • sudo apt-get install libapache2-mod-php5
  • Build and install mod_rpaf
  • sudo rm /var/www/index.html
  • echo '<?php usleep(rand(1000,100000)); var_dump($_SERVER);' | sudo tee /var/www/index.php

Run a couple of tests:

  • for i in {1..1000}; do curl -s http://localhost/ | grep '"80"'; done | wc -l && for i in {1..1000}; do curl -s http://localhost/ | grep '"443"'; done | wc -l
  • Should output 1000 and 0

Use siege to simulate traffic. Easiest option is to run 2 simultaneous siege processes in two terminals:

  • siege --time=20M -c 15 -H 'X-Forwarded-For: 123.123.123.123' -H 'X-Forwarded-Port: 443' http://localhost/
  • siege --time=20M -c 15 http://localhost/

Let siege run for a minute or two, then repeat the tests:

  • for i in {1..1000}; do curl -s http://localhost/ | grep '"443"'; done | wc -l
  • for i in {1..1000}; do curl -s http://localhost/ | grep '"80"'; done | wc -l

These final tests return some odd numbers in my tests. At first, instead of 1000 and 0, I get ~990 and ~10. So 99% of the requests are getting the wrong port. Some tests show 100% of requests with the wrong port.

Happy to provide more debugging information if necessary.

chmac commented Dec 16, 2013

Managed to reproduce this issue consistently.

  • Install a fresh Ubuntu 12.04 server (EC2 / Amazon / Virtualbox / whatever)
  • sudo apt-get install libapache2-mod-php5
  • Build and install mod_rpaf
  • sudo rm /var/www/index.html
  • echo '<?php usleep(rand(1000,100000)); var_dump($_SERVER);' | sudo tee /var/www/index.php

Run a couple of tests:

  • for i in {1..1000}; do curl -s http://localhost/ | grep '"80"'; done | wc -l && for i in {1..1000}; do curl -s http://localhost/ | grep '"443"'; done | wc -l
  • Should output 1000 and 0

Use siege to simulate traffic. Easiest option is to run 2 simultaneous siege processes in two terminals:

  • siege --time=20M -c 15 -H 'X-Forwarded-For: 123.123.123.123' -H 'X-Forwarded-Port: 443' http://localhost/
  • siege --time=20M -c 15 http://localhost/

Let siege run for a minute or two, then repeat the tests:

  • for i in {1..1000}; do curl -s http://localhost/ | grep '"443"'; done | wc -l
  • for i in {1..1000}; do curl -s http://localhost/ | grep '"80"'; done | wc -l

These final tests return some odd numbers in my tests. At first, instead of 1000 and 0, I get ~990 and ~10. So 99% of the requests are getting the wrong port. Some tests show 100% of requests with the wrong port.

Happy to provide more debugging information if necessary.

@y-ken

This comment has been minimized.

Show comment
Hide comment
@y-ken

y-ken Dec 16, 2013

I have hit same problem on my environment, too (CentOS 6.4).

y-ken commented Dec 16, 2013

I have hit same problem on my environment, too (CentOS 6.4).

@gnif

This comment has been minimized.

Show comment
Hide comment
@gnif

gnif Dec 17, 2013

Owner

Hi,

This is a known issue, it is caused by a race condition between threads setting the virtual host port. A simple way to work around this would be to run two virtual hosts, one for HTTPS requests, and the other for HTTP on separate ports. Currently there is no known way to fix this issue with Apache's internal design.

Owner

gnif commented Dec 17, 2013

Hi,

This is a known issue, it is caused by a race condition between threads setting the virtual host port. A simple way to work around this would be to run two virtual hosts, one for HTTPS requests, and the other for HTTP on separate ports. Currently there is no known way to fix this issue with Apache's internal design.

@gnif gnif closed this Dec 17, 2013

@y-ken

This comment has been minimized.

Show comment
Hide comment
@y-ken

y-ken Dec 17, 2013

I see. thank you for telling me the details.

y-ken commented Dec 17, 2013

I see. thank you for telling me the details.

@chmac

This comment has been minimized.

Show comment
Hide comment
@chmac

chmac Dec 17, 2013

Ahh, interesting. Thanks for the response. I've been testing exactly that, splitting the virtual hosts between http and https. It didn't seem to be 100% reliable, but I'll conduct some further tests. Open to a pull request explaining the issue in the readme?

chmac commented Dec 17, 2013

Ahh, interesting. Thanks for the response. I've been testing exactly that, splitting the virtual hosts between http and https. It didn't seem to be 100% reliable, but I'll conduct some further tests. Open to a pull request explaining the issue in the readme?

y-ken added a commit to y-ken/mod_rpaf that referenced this issue Dec 20, 2013

Remove support of RPAFsetport
ref. gnif/mod_rpaf#12 (comment)

> This is a known issue, it is caused by a race condition between
threads setting the virtual host port. A simple way to work around this
would be to run two virtual hosts, one for HTTPS requests, and the
other for HTTP on separate ports. Currently there is no known way to
fix this issue with Apache's internal design.

@gnif gnif reopened this Oct 29, 2014

@gnif

This comment has been minimized.

Show comment
Hide comment
@gnif

gnif Oct 29, 2014

Owner

Can someone please re-test this, it should be corrected.

Owner

gnif commented Oct 29, 2014

Can someone please re-test this, it should be corrected.

@gnif

This comment has been minimized.

Show comment
Hide comment
@gnif

gnif Oct 13, 2015

Owner

Closing due to inactivity, assuming fixed until told otherwise.

Owner

gnif commented Oct 13, 2015

Closing due to inactivity, assuming fixed until told otherwise.

@gnif gnif closed this Oct 13, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment