Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incoming shows a doc_count of 204717, but status shows 5 #275

Open
joehobson opened this issue Apr 7, 2014 · 3 comments

Comments

@joehobson
Copy link
Member

commented Apr 7, 2014

I've been trying to test the distribute service. The Alpha node had a doc_count of 5 before I started. I registered a connection on the Sandbox node to have it distribute to Alpha. Sandbox was showing a doc_count of 204717. I ran into a few errors getting distribute to start, but then I got a message that looked like it was set. I can't tell if it's been running, but when I look at /incoming on Alpha, i now see doc_count: 204717. That's great, but at what point do those documents show up in the available doc_count, or in /obtain or /harvest/listrecords?

Looking at the couchdb data files, resource_data.couch is only 173K, while incoming.couch is 1.1GB

Anything else I need to do to finish off the distribute process, or did my first error break the whole thing?

@jimklo

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2015

@joehobson Okay, I ran into this issue in the testing. I think I see a possible problem, but even when I try to "publish" around it - it's doesn't seem to be doing anything, so I'm not sure what's happening 100%.

In lr/model/resource_data_monitor/incomping_copy_handler.py I'm noticing that there is:

_DOCUMENT_UPDATE_THRESHOLD = 100

and then around line 79:

        if len(self.documents) >= _DOCUMENT_UPDATE_THRESHOLD or len(self.documents) >= database.info()['doc_count']:
            while len(self.documents) > 0:
                doc = self.documents.popleft()
                tname = threadName(doc)
                t = Thread(target=handleDocument, name=tname, args=(doc,))
                self.threads[tname] = t
                t.start()
                while len(self.threads) > self.max_threads:
                    time.sleep(.1)

I'm not entirely sure what get's passed in as database but at the very least, there's needs to be some time has passed trigger as well.

So I tried publishing around this - published 105 docs, that should create bump the number of docs > 100, and I'm still not seeing the copies happen. It's almost like the Monitor isn't running.

Do you want me to try and fix this?

@jimklo

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2015

Looking at this more since it's kinda blocking me. Looks like MonitorChanges is mysteriously dying. A few items get processed, and then once uwsgi completely starts up... it seems to be dead.

@jimklo

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2015

okay... think I found root cause... uwsgi - probably the least "stable" thing in the whole house of cards. My guess is the version of uwsgi I'm using is newer (however it shouldn't be) but in any case it starts without threading support; so what happens is upon startup it starts the change monitors, then once uwsgi is fully started - it kills those threads.

I think a change to the startup script should fix this. Will verify.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.