server gets too busy when the crawler kicks in

[abmargb]

When the crawler kicks in, that's the last thing I see in the logs:

2014-08-14 04:36:23,153 [pool-3-thread-1] DEBUG org.buddycloud.channelserver.queue.InQueueConsumer - Received payload: '<iq to="cs.buddycloud.com" type="get" id="G4Ve9-2277021" from="buddycloud-crawler@buddycloud.org/crawler-2395528746076317171"><query xmlns="http://jabber.org/protocol/disco#items"/></iq>'.
2014-08-14 04:36:23,159 [pool-3-thread-1] DEBUG org.buddycloud.channelserver.packetprocessor.iq.IQProcessor - Finding IQ processor for namespace http://jabber.org/protocol/disco#items
2014-08-14 04:36:23,159 [pool-3-thread-1] INFO  org.buddycloud.channelserver.packetprocessor.iq.namespace.AbstractNamespace - Using processor for packet type: get

Then the server stops responding for the next 4 minutes or so, and then it spits out everything related to the interaction with the crawler.

I agree the crawler needs improvement, but I suppose the server shouldn't behave like this ?

[lloydwatkin]

Are you using a synchronous logger? We've switched to an async one.

As you've got a java background @abmargb maybe it would be worth you looking into the queue runner and thread counts to see if that can be improved (and I have no doubts that it can).

[MrFrety]

I'm not an expert, but...
Could it help to fork the p2p-search engine Yacy (https://github.com/f1ori/yacy) in order to get a good functioning crawler?

[abmargb]

Thanks for the pointer, @MrFrety. I'll take a look at it.

[imaginator]

imho the crawler should just be drinking from the /firehose node on all servers since=<last crawl>

[abmargb]

It seems that the disco#items query takes too long:

2014-09-15 12:38:03,028 [pool-3-thread-1] DEBUG org.buddycloud.channelserver.queue.InQueueConsumer - Received payload: '<iq id="FKQy8-20" from="robot@alps.buddycloud.net/crawler--160259746284017356" type="get" to="cs.buddycloud.net"><query xmlns="http://jabber.org/protocol/disco#items"/></iq>'.
2014-09-15 12:38:03,041 [pool-3-thread-1] DEBUG org.buddycloud.channelserver.packetprocessor.iq.IQProcessor - Finding IQ processor for namespace http://jabber.org/protocol/disco#items
2014-09-15 12:38:03,041 [pool-3-thread-1] INFO  org.buddycloud.channelserver.packetprocessor.iq.namespace.AbstractNamespace - Using processor for packet type: get
2014-09-15 12:43:24,695 [pool-3-thread-1] DEBUG org.buddycloud.channelserver.queue.InQueueConsumer - Payload handled in '321667' milliseconds.

And that's for the server running in si.buddycloud.com, which has ~50 nodes.

[lloydwatkin]

that's a crazy amount of time
On 15 Sep 2014 14:10, "Abmar Barros" notifications@github.com wrote:

It seems that the disco#items query takes too long:

2014-09-15 12:38:03,028 [pool-3-thread-1] DEBUG org.buddycloud.channelserver.queue.InQueueConsumer - Received payload: ''.
2014-09-15 12:38:03,041 [pool-3-thread-1] DEBUG org.buddycloud.channelserver.packetprocessor.iq.IQProcessor - Finding IQ processor for namespace http://jabber.org/protocol/disco#items
2014-09-15 http://jabber.org/protocol/disco#items2014-09-15 12:38:03,041 [pool-3-thread-1] INFO org.buddycloud.channelserver.packetprocessor.iq.namespace.AbstractNamespace - Using processor for packet type: get
2014-09-15 12:43:24,695 [pool-3-thread-1] DEBUG org.buddycloud.channelserver.queue.InQueueConsumer - Payload handled in '321667' milliseconds.

And that's for the server running in si.buddycloud.com, which has ~50
nodes.

—
Reply to this email directly or view it on GitHub
#211 (comment)
.

[lloydwatkin]

Its not returning all nodes from instances maybe?
On 15 Sep 2014 14:10, "Abmar Barros" notifications@github.com wrote:

It seems that the disco#items query takes too long:

2014-09-15 12:38:03,028 [pool-3-thread-1] DEBUG org.buddycloud.channelserver.queue.InQueueConsumer - Received payload: ''.
2014-09-15 12:38:03,041 [pool-3-thread-1] DEBUG org.buddycloud.channelserver.packetprocessor.iq.IQProcessor - Finding IQ processor for namespace http://jabber.org/protocol/disco#items
2014-09-15 http://jabber.org/protocol/disco#items2014-09-15 12:38:03,041 [pool-3-thread-1] INFO org.buddycloud.channelserver.packetprocessor.iq.namespace.AbstractNamespace - Using processor for packet type: get
2014-09-15 12:43:24,695 [pool-3-thread-1] DEBUG org.buddycloud.channelserver.queue.InQueueConsumer - Payload handled in '321667' milliseconds.

And that's for the server running in si.buddycloud.com, which has ~50
nodes.

—
Reply to this email directly or view it on GitHub
#211 (comment)
.

[abmargb]

I think it is, since I can't see any RSM set.

[lloydwatkin]

@abmargb I'm thinking this is something that is broken by the multi-tenanted set up? As for disco items we just grab all nodes from the database.

[abmargb]

@lloydwatkin I'm trying to replicate it locally so I can give it a best answer.

[lloydwatkin]

I think we need to rethink the way the multi-tenanted setup works because, from what I understand, this will cause data leakage, and I can think of at least one other place this will occur. Lets organise a time to chat about it.

[abmargb]

You mean data leaking among domains?

Sure, let me just finish refactoring the crawler and I'll have more property over the subject :-)

[abmargb]

After the changes from #238, disco#items takes around 300ms in the integration environment.