Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

m broker <-> n threads #679

Commits on Jun 1, 2016

  1. m broker <-> n threads

    - create pre-configured number of broker threads (these threads do everything that broker threads used to do in 1:1 broker to thread model). The way they pace computation is different though, because they are multiplexing many brokers now.
    - new entity called rd_kafka_broker_thread_t (rkbt), which gets brokers assigned in a way that tries to balances number of brokers assigned per-thread.
    - termination and freeing of rkb is performed on rkbt
    - rkbt has a separate rkb array owned by the thread which is re-populated every time a broker is added or removed. Here addition or removal use rkbt broker-assignment lock, but this array is exclusively owned by rkbt. The rkb pointers are copied over this array and then used lock-lessly.
    - this introduces a new config parameter called 'broker.thread.count' (defaults to 1) and identifies number of threads user intends to run for all the broker related work, which allows user to configure n (thread count) independent of m (broker count)
    
    Note: have tried to fix windows support in transport (WSApoll call), but not sure if that is the right thing to do. That area needs a very close review.
    janmejay committed Jun 1, 2016
    Configuration menu
    Copy the full SHA
    30be48e View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2016

  1. Connect used to enable POLLOUT for polled events, but io-serve used t…

    …o unset/clear it, if connect was not successful by the first poll call. So this was a race-condition between connect finishing successfully and first poll call returning. Sometimes poll call returned first which lead to POLLOUT event-mask being cleared from pollfd, causing broker to remain in RD_KAFKA_BROKER_STATE_CONNECT perpetually. This fixes it by not trying to set state while connecting, but instead recognizing before poll that RD_KAFKA_BROKER_STATE_CONNECT state requires POLLOUT enabled for connect-completion to be recorded and hence state-transition to UP state.
    janmejay committed Aug 24, 2016
    Configuration menu
    Copy the full SHA
    f227c26 View commit details
    Browse the repository at this point in the history