Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Incoming shows a doc_count of 204717, but status shows 5 #275
I've been trying to test the distribute service. The Alpha node had a doc_count of 5 before I started. I registered a connection on the Sandbox node to have it distribute to Alpha. Sandbox was showing a doc_count of 204717. I ran into a few errors getting distribute to start, but then I got a message that looked like it was set. I can't tell if it's been running, but when I look at /incoming on Alpha, i now see doc_count: 204717. That's great, but at what point do those documents show up in the available doc_count, or in /obtain or /harvest/listrecords?
Looking at the couchdb data files, resource_data.couch is only 173K, while incoming.couch is 1.1GB
Anything else I need to do to finish off the distribute process, or did my first error break the whole thing?
@joehobson Okay, I ran into this issue in the testing. I think I see a possible problem, but even when I try to "publish" around it - it's doesn't seem to be doing anything, so I'm not sure what's happening 100%.
In lr/model/resource_data_monitor/incomping_copy_handler.py I'm noticing that there is:
_DOCUMENT_UPDATE_THRESHOLD = 100
and then around line 79:
if len(self.documents) >= _DOCUMENT_UPDATE_THRESHOLD or len(self.documents) >= database.info()['doc_count']: while len(self.documents) > 0: doc = self.documents.popleft() tname = threadName(doc) t = Thread(target=handleDocument, name=tname, args=(doc,)) self.threads[tname] = t t.start() while len(self.threads) > self.max_threads: time.sleep(.1)
I'm not entirely sure what get's passed in as
So I tried publishing around this - published 105 docs, that should create bump the number of docs > 100, and I'm still not seeing the copies happen. It's almost like the Monitor isn't running.
Do you want me to try and fix this?
okay... think I found root cause... uwsgi - probably the least "stable" thing in the whole house of cards. My guess is the version of uwsgi I'm using is newer (however it shouldn't be) but in any case it starts without threading support; so what happens is upon startup it starts the change monitors, then once uwsgi is fully started - it kills those threads.
I think a change to the startup script should fix this. Will verify.