New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hanging POST requests with no data (potential for DDOS) #1331

Closed
SebastianEdwards opened this Issue Dec 20, 2014 · 4 comments

Comments

Projects
None yet
2 participants
@SebastianEdwards

SebastianEdwards commented Dec 20, 2014

$ curl -v -XPOST --data "" http://0.0.0.0:5000/                                                                                                                  
* Hostname was NOT found in DNS cache
*   Trying 0.0.0.0...
* Connected to 0.0.0.0 (127.0.0.1) port 5000 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.37.1
> Host: 0.0.0.0:5000
> Accept: */*
> Content-Length: 0
> Content-Type: application/x-www-form-urlencoded
>
Hangs here indefinitely
@SebastianEdwards

This comment has been minimized.

Show comment
Hide comment
@SebastianEdwards

SebastianEdwards Dec 21, 2014

Using 5.0.0-beta2 by the way

SebastianEdwards commented Dec 21, 2014

Using 5.0.0-beta2 by the way

@FooBarWidget

This comment has been minimized.

Show comment
Hide comment
@FooBarWidget

FooBarWidget Dec 22, 2014

Member

I can't reproduce this issue. Does this happen with every app, and with every URL endpoint? If not, what is the specific app/endpoint doing? When it hangs indefinitely, can you show me what passenger-status and passenger-status --show=requests say?

Member

FooBarWidget commented Dec 22, 2014

I can't reproduce this issue. Does this happen with every app, and with every URL endpoint? If not, what is the specific app/endpoint doing? When it hangs indefinitely, can you show me what passenger-status and passenger-status --show=requests say?

@SebastianEdwards

This comment has been minimized.

Show comment
Hide comment
@SebastianEdwards

SebastianEdwards Dec 22, 2014

The issue occurs both with my app and with this minimal example (with 5.0.0.beta2 specified). It can occur at any endpoint and the request never hits the actual application. Encountered this on both my development machine and Heroku.

Here is the show-requests output.

Version : 5.0.0.beta2
Date    : 2014-12-23 10:14:24 +1300
Instance: Ozqqf6Sd (Phusion_Passenger/5.0.0.beta2)

{
   "thread1" : {
      "active_client_count" : 0,
      "active_clients" : {},
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 128,
      "free_request_count" : 1,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 1,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 512,
            "human_readable" : "512 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 1,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread2" : {
      "active_client_count" : 0,
      "active_clients" : {},
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 128,
      "free_request_count" : 1,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 1,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 512,
            "human_readable" : "512 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 1,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread3" : {
      "active_client_count" : 1,
      "active_clients" : {
         "3.1" : {
            "connected_at" : {
               "local" : "Tue Dec 23 10:13:34 2014",
               "relative" : "50s ago",
               "timestamp" : 1419282814.395125
            },
            "connection_state" : "ACTIVE",
            "current_request" : {
               "flags" : {
                  "dechunk_response" : false,
                  "https" : false,
                  "request_body_buffering" : false
               },
               "http_state" : "PARSING_HEADERS",
               "refcount" : 1,
               "session_checkout_try" : 0,
               "state" : "ANALYZING_REQUEST",
               "sticky_session" : false
            },
            "lingering_request_count" : 0,
            "name" : "3.1",
            "number" : 1,
            "refcount" : 2,
            "requests_begun" : 0
         }
      },
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 127,
      "free_request_count" : 0,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 1,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 512,
            "human_readable" : "512 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 1,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread4" : {
      "active_client_count" : 1,
      "active_clients" : {
         "4.1" : {
            "connected_at" : {
               "local" : "Tue Dec 23 10:14:09 2014",
               "relative" : "15s ago",
               "timestamp" : 1419282849.559246
            },
            "connection_state" : "ACTIVE",
            "current_request" : {
               "app_response_body_already_read" : 0,
               "app_response_body_fully_read" : true,
               "app_response_body_type" : "NO_BODY",
               "app_response_http_major" : 1,
               "app_response_http_minor" : 0,
               "app_response_http_state" : "PARSING_HEADERS",
               "app_response_want_keep_alive" : false,
               "flags" : {
                  "dechunk_response" : false,
                  "https" : false,
                  "request_body_buffering" : false
               },
               "host" : "0.0.0.0:3000",
               "http_major" : 1,
               "http_minor" : 1,
               "http_state" : "COMPLETE",
               "method" : "POST",
               "path" : "/",
               "refcount" : 1,
               "request_body_already_read" : 0,
               "request_body_fully_read" : true,
               "request_body_type" : "NO_BODY",
               "response_begun" : false,
               "session" : {
                  "gupid" : "168f13a-6xCjFVP5vQ",
                  "pid" : 75172
               },
               "session_checkout_try" : 1,
               "started_at" : {
                  "local" : "Tue Dec 23 10:14:09 2014",
                  "relative" : "15s ago",
                  "timestamp" : 1419282849.559246
               },
               "state" : "WAITING_FOR_APP_OUTPUT",
               "sticky_session" : false,
               "want_keep_alive" : true
            },
            "lingering_request_count" : 0,
            "name" : "4.1",
            "number" : 1,
            "refcount" : 2,
            "requests_begun" : 1
         }
      },
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 127,
      "free_request_count" : 0,
      "mbuf_pool" : {
         "active_blocks" : 1,
         "active_memory" : {
            "bytes" : 512,
            "human_readable" : "512 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 1,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 512,
            "human_readable" : "512 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 145,
      "total_clients_accepted" : 1,
      "total_requests_begun" : 1,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread5" : {
      "active_client_count" : 0,
      "active_clients" : {},
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 128,
      "free_request_count" : 0,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 0,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 0,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread6" : {
      "active_client_count" : 0,
      "active_clients" : {},
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 128,
      "free_request_count" : 0,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 0,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 0,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread7" : {
      "active_client_count" : 0,
      "active_clients" : {},
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 128,
      "free_request_count" : 0,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 0,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 0,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread8" : {
      "active_client_count" : 0,
      "active_clients" : {},
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 128,
      "free_request_count" : 0,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 0,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 0,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "threads" : 8
}

SebastianEdwards commented Dec 22, 2014

The issue occurs both with my app and with this minimal example (with 5.0.0.beta2 specified). It can occur at any endpoint and the request never hits the actual application. Encountered this on both my development machine and Heroku.

Here is the show-requests output.

Version : 5.0.0.beta2
Date    : 2014-12-23 10:14:24 +1300
Instance: Ozqqf6Sd (Phusion_Passenger/5.0.0.beta2)

{
   "thread1" : {
      "active_client_count" : 0,
      "active_clients" : {},
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 128,
      "free_request_count" : 1,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 1,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 512,
            "human_readable" : "512 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 1,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread2" : {
      "active_client_count" : 0,
      "active_clients" : {},
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 128,
      "free_request_count" : 1,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 1,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 512,
            "human_readable" : "512 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 1,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread3" : {
      "active_client_count" : 1,
      "active_clients" : {
         "3.1" : {
            "connected_at" : {
               "local" : "Tue Dec 23 10:13:34 2014",
               "relative" : "50s ago",
               "timestamp" : 1419282814.395125
            },
            "connection_state" : "ACTIVE",
            "current_request" : {
               "flags" : {
                  "dechunk_response" : false,
                  "https" : false,
                  "request_body_buffering" : false
               },
               "http_state" : "PARSING_HEADERS",
               "refcount" : 1,
               "session_checkout_try" : 0,
               "state" : "ANALYZING_REQUEST",
               "sticky_session" : false
            },
            "lingering_request_count" : 0,
            "name" : "3.1",
            "number" : 1,
            "refcount" : 2,
            "requests_begun" : 0
         }
      },
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 127,
      "free_request_count" : 0,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 1,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 512,
            "human_readable" : "512 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 1,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread4" : {
      "active_client_count" : 1,
      "active_clients" : {
         "4.1" : {
            "connected_at" : {
               "local" : "Tue Dec 23 10:14:09 2014",
               "relative" : "15s ago",
               "timestamp" : 1419282849.559246
            },
            "connection_state" : "ACTIVE",
            "current_request" : {
               "app_response_body_already_read" : 0,
               "app_response_body_fully_read" : true,
               "app_response_body_type" : "NO_BODY",
               "app_response_http_major" : 1,
               "app_response_http_minor" : 0,
               "app_response_http_state" : "PARSING_HEADERS",
               "app_response_want_keep_alive" : false,
               "flags" : {
                  "dechunk_response" : false,
                  "https" : false,
                  "request_body_buffering" : false
               },
               "host" : "0.0.0.0:3000",
               "http_major" : 1,
               "http_minor" : 1,
               "http_state" : "COMPLETE",
               "method" : "POST",
               "path" : "/",
               "refcount" : 1,
               "request_body_already_read" : 0,
               "request_body_fully_read" : true,
               "request_body_type" : "NO_BODY",
               "response_begun" : false,
               "session" : {
                  "gupid" : "168f13a-6xCjFVP5vQ",
                  "pid" : 75172
               },
               "session_checkout_try" : 1,
               "started_at" : {
                  "local" : "Tue Dec 23 10:14:09 2014",
                  "relative" : "15s ago",
                  "timestamp" : 1419282849.559246
               },
               "state" : "WAITING_FOR_APP_OUTPUT",
               "sticky_session" : false,
               "want_keep_alive" : true
            },
            "lingering_request_count" : 0,
            "name" : "4.1",
            "number" : 1,
            "refcount" : 2,
            "requests_begun" : 1
         }
      },
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 127,
      "free_request_count" : 0,
      "mbuf_pool" : {
         "active_blocks" : 1,
         "active_memory" : {
            "bytes" : 512,
            "human_readable" : "512 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 1,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 512,
            "human_readable" : "512 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 145,
      "total_clients_accepted" : 1,
      "total_requests_begun" : 1,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread5" : {
      "active_client_count" : 0,
      "active_clients" : {},
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 128,
      "free_request_count" : 0,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 0,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 0,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread6" : {
      "active_client_count" : 0,
      "active_clients" : {},
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 128,
      "free_request_count" : 0,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 0,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 0,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread7" : {
      "active_client_count" : 0,
      "active_clients" : {},
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 128,
      "free_request_count" : 0,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 0,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 0,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "thread8" : {
      "active_client_count" : 0,
      "active_clients" : {},
      "disconnected_client_count" : 0,
      "disconnected_clients" : {},
      "free_client_count" : 128,
      "free_request_count" : 0,
      "mbuf_pool" : {
         "active_blocks" : 0,
         "active_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         },
         "chunk_size" : 512,
         "free_blocks" : 0,
         "offset" : 448,
         "spare_memory" : {
            "bytes" : 0,
            "human_readable" : "0 bytes"
         }
      },
      "pid" : 75111,
      "server_state" : "ACTIVE",
      "total_bytes_consumed" : 0,
      "total_clients_accepted" : 0,
      "total_requests_begun" : 0,
      "turbocaching" : {
         "fetches" : 0,
         "hit_ratio" : nan,
         "hits" : 0,
         "store_success_ratio" : nan,
         "store_successes" : 0,
         "stores" : 0
      }
   },
   "threads" : 8
}
@FooBarWidget

This comment has been minimized.

Show comment
Hide comment
@FooBarWidget

FooBarWidget Jan 2, 2015

Member

Yes, problem reproduced. Thank you.

Member

FooBarWidget commented Jan 2, 2015

Yes, problem reproduced. Thank you.

FooBarWidget added a commit that referenced this issue Jan 2, 2015

Do not forward CONTENT_LENGTH if it is 0.
Also closes GH-1331, but in a better way than the previous fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment