Skip to content

Broker/Coordinator nodes start only after inventory is initialized#4770

Closed
niketh wants to merge 3 commits intoapache:masterfrom
niketh:wait-for-views-toload
Closed

Broker/Coordinator nodes start only after inventory is initialized#4770
niketh wants to merge 3 commits intoapache:masterfrom
niketh:wait-for-views-toload

Conversation

@niketh
Copy link
Copy Markdown
Contributor

@niketh niketh commented Sep 9, 2017

Broker queries return wrong results just after startup because the inventory is not fully initialized and BrokerServerView is incomplete. The solution is to block CuratorInventoryManager start() method till inventory is fully available.

@niketh niketh force-pushed the wait-for-views-toload branch from ce03121 to 16f3955 Compare September 9, 2017 08:23
@niketh niketh force-pushed the wait-for-views-toload branch from 16f3955 to 6eb80a5 Compare September 9, 2017 17:17
@niketh niketh closed this Sep 9, 2017
@niketh niketh reopened this Sep 9, 2017
@niketh niketh closed this Sep 9, 2017
@niketh niketh reopened this Sep 12, 2017
@himanshug
Copy link
Copy Markdown
Contributor

@niketh I don't think this will fix the problem, how will your clients know not to send requests before broker lifecycle has finished starting ?
With 0.11.0 , you should be able to get list of brokers from coordinator using /druid/coordinator/v1/cluster/broker endpoint , list returned there is populated only after a broker has initialized its inventory. Then your client can query that endpoint to get list of brokers which are ready.

@niketh
Copy link
Copy Markdown
Contributor Author

niketh commented Sep 13, 2017

@himanshug Tested in production and it works as expected.

how will your clients know not to send requests before broker lifecycle has finished starting ?

Jetty will not start until the full view is loaded

@niketh niketh closed this Sep 13, 2017
@niketh niketh reopened this Sep 13, 2017
}

// Block start() till PathChildrenCache is fully loaded
while (!containerCacheListener.doneInitializing) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use a CountDownLatch.await(1 minute) here instead? that way not too many logs will be printed and also it wouldn't wait any more than necessary ?

@niketh niketh force-pushed the wait-for-views-toload branch from 7d660d0 to 928b473 Compare September 22, 2017 00:36
@niketh niketh closed this Sep 25, 2017
@niketh niketh reopened this Sep 25, 2017

private volatile PathChildrenCache childrenCache;

private final CountDownLatch latch = new CountDownLatch(1);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we call this initializationDoneLatch or something like that ? also , I think doneInitializing boolean variable can be removed and latch.await(1 ms) can be used to do the boolean check wherever necessary.

@gianm
Copy link
Copy Markdown
Contributor

gianm commented Jan 15, 2019

Same idea as #6742.

@gianm gianm closed this Jan 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants