Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

write starvation fixes #545

Closed
wants to merge 9 commits into from
Closed

write starvation fixes #545

wants to merge 9 commits into from

Commits on Apr 11, 2015

  1. A fix the write starvation problem that we see with tornado and pika

    By default (in base_connection) Pika buffers all writes and only
    attempts to send the next time it drops into the ioloop and detects
    the socket as writable. This causes some odd behaviour in a number
    of cases.
    1. A process generating large numbers of messages will not actually
    send them until it finishes the processing and drops into the ioloop.
    2. A process that is consuming a large queue will only send messages
    when it's read buffer is empty. If the messages are small this means
    it may end up consuming 000s of messages for every one it manages to
    publish. This behaviour stalls pipelines of processes.
    
    This patch tries to send the data on the socket as soon as it's
    generated and to avoid timeouts that might be generated by the use
    of sendall() it uses send() and handles partial writes by requeing
    the data.
    Will Slater authored and Will Slater committed Apr 11, 2015
    Configuration menu
    Copy the full SHA
    1f22edc View commit details
    Browse the repository at this point in the history
  2. Use non-blocking socket in Tornado

    Will Slater authored and Will Slater committed Apr 11, 2015
    Configuration menu
    Copy the full SHA
    69d17df View commit details
    Browse the repository at this point in the history
  3. Make _handle_write fixes to base_connection

    Move the _handle_write changes from tornado_connection to
    base_connection. They're equally applicable to all connections
    and this fixes a bug in master where BlockingConnections wouldn't
    send messages until close was called.
    Will Slater authored and Will Slater committed Apr 11, 2015
    Configuration menu
    Copy the full SHA
    a11ce5f View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2015

  1. Make _handle_write fixes to base_connection

    Move the _handle_write changes from tornado_connection to
    base_connection. They're equally applicable to all connections
    and this fixes a bug in master where BlockingConnections wouldn't
    send messages until close was called.
    Will Slater authored and Will Slater committed Apr 12, 2015
    Configuration menu
    Copy the full SHA
    576c1f0 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2015

  1. Merge branch 'master' into fix_write_handling

    Will Slater authored and Will Slater committed Apr 13, 2015
    Configuration menu
    Copy the full SHA
    cca26df View commit details
    Browse the repository at this point in the history
  2. Various cleanups as suggested by @vitaly-krugl

    Will Slater authored and Will Slater committed Apr 13, 2015
    Configuration menu
    Copy the full SHA
    f2dc430 View commit details
    Browse the repository at this point in the history
  3. Minor cleanups

    remove unused import, dodgy error handling
    Will Slater authored and Will Slater committed Apr 13, 2015
    Configuration menu
    Copy the full SHA
    7b474bb View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    7d457f5 View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2015

  1. SSL fix for non-blocking reads

    wjps committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    635957f View commit details
    Browse the repository at this point in the history