Always "Checking import_jobs"

[tjwebb]

Even after I refresh the database and flushall in redis, I see this when I boot up the dev server:

20:59:32 resque.1  | ** [20:59:32 2012-12-11] 7022: Checking import_jobs
20:59:32 resque.1  | ** [20:59:32 2012-12-11] 7022: Checking queries_threshold
20:59:32 resque.1  | ** [20:59:32 2012-12-11] 7022: Sleeping for 5.0 seconds
20:59:32 resque.1  | ** [20:59:32 2012-12-11] 7022: resque-1.19.0: Waiting for *
20:59:37 resque.1  | ** [20:59:37 2012-12-11] 7022: Checking import_jobs
20:59:37 resque.1  | ** [20:59:37 2012-12-11] 7022: Checking queries_threshold
20:59:37 resque.1  | ** [20:59:37 2012-12-11] 7022: Sleeping for 5.0 seconds
20:59:37 resque.1  | ** [20:59:37 2012-12-11] 7022: resque-1.19.0: Waiting for *
20:59:42 resque.1  | ** [20:59:42 2012-12-11] 7022: Checking import_jobs
...

ad infinitum. Is this supposed to happen?

[tjwebb]

And then when I load up the web UI, it thinks there's some table somewhere to be loaded, and hangs at "Creating Table" in the bottom left.

[tjwebb]

Ah! It seems this might actually be the cause of this issue: #50

Because always I can import one SHP file, and then all subsequent imports fail until I wipe the database + redis. Very weird.

[demimismo]

The "Checking import_jobs" on redis log is expected.

The permanent "Creating table" message on the UI is a known bug that I hope we'll fix this week.

We also have a known bug on development environment that I think is what is happening to you when trying to import various files. Resque hangs after the first import (successful or not) and then you have to reboot the daemon to process another.

I'll keep this issue until is fixed, but we can't prioritize it since is only happening on development, sorry for that.

[tjwebb]

Can you point me to where the bug is in the code? I can try to fix it if you can point me to where the problem is

[demimismo]

Not yet. I've had this problem for some weeks, seems to be some weird interaction between resque and the debugger gem, I'd start by removing the debugger gem from the local environment and then checking if the problem persist.

But maybe something else, nasty bug :(

[strk]

I've the same problem (also for weeks). If anyone found a fix or a workaround please let me know as the 2-seconds frontend heartbeat makes debugging other issues harder :)

[markov00]

Any news about this issue? I still can't import data because after the first fail, the importer stuck and I can't go further.

[devloic]

I started the importer in production mode with "RAILS_ENV=production VVERBOSE=true QUEUE=* rake environment resque:work" (and the rest of the stack in production mode too) but I still get the error: first import okay and then it fails with cpu of ruby process above 90%. Is there any special configuration in production mode to get rid of it ?

[demimismo]

Seems like connecting the rails app to postgres via pgbouncer solves the issue for some of us.

Could you try with that? We're a little bit lost on this issue.

[devloic]

I installed pgbouncer.... and the imports work without locking on my ubuntu 12.04 VM. Fantastic! Thanks so much.

[strk]

Indeed, same here. Worth writing a note on https://github.com/CartoDB/cartodb20/wiki/Problems-faced-during-CartoDB-install-&-solutions-if-known -- volunteers ?

[demimismo]

Done https://github.com/CartoDB/cartodb20/wiki/Problems-faced-during-CartoDB-install-&-solutions-if-known

[strk]

I think the ticket should stay open until the bug is fixed. Starting pgbouncer is just a workaround.

[Kartones]

This has been fixed a while ago.
As demimismo mentions the checking imports "ad infinitum" log is correct.
And the requirement of PGBouncer is not anymore required, if an import job fails due to bad SSL (a known issue we have but not easily fixable without rewriting the whole Ruby jobs backend) Resque detects it and retries.

[strk]

Is the SSL error also logged ?
I think we can deal with logging in #388

[Kartones]

Yes, the Resque logs would log this SSL errors, like:

[10:22:22 2014-04-15] 27617: resque-1.23.0: Processing imports since 1397550142
Job got a DatabaseDisconnectError: PG::Error: SSL error: decryption failed or bad record mac
Retrying job

But Importer logs should be definetly improved, yes