Error when uploading SHP file on local dev instance #50

tjwebb · 2012-12-07T19:41:19Z

First of all, thank you guys so much for helping me finally get this thing running. Here's another one for ya:

I'm trying to upload a .zip file that contains a SHP file and all of the usual accessories. It uploads great through my account at cartodb.com. But when I try to upload it on my new dev instance, I get this error:

19:39:03 resque.1  | ERROR 6: No translation for Mercator to PROJ.4 format is known.
19:39:03 resque.1  | ERROR 6: No translation an empty SRS to PROJ.4 format is known.

Any help is appreciated, thanks.

The text was updated successfully, but these errors were encountered:

tjwebb · 2012-12-10T15:55:45Z

I still can't make this work. I reinstalled PostGIS because the proj4 seemed to be a postgis problem. So I no longer see the particular errors I mentioned above.

But now the "processing" of the file seems to fall into an infinite loop of some kind. I now see no error message, and 100% CPU usage. I let it try to process a 10MB file for 3 hours before I gave up. Uploading this same data to cartodb.com takes less than 10 seconds to process.

Any ideas?

tjwebb · 2012-12-10T20:42:58Z

I'm not sure why, but when I tested this again today, I get a new error:

NoMethodError (undefined method `>' for nil:NilClass):
app/controllers/api/json/imports_controller.rb:30:in `create'

btw, this happens with my custom SHP data, as well as the built-in sample data e.g. World borders.

Any help is appreciated, thanks.

demimismo · 2012-12-11T08:28:51Z

The infinite loop thing is a known issue, the importer doesn't detect some crashed imports. We'll hopefully push a fix this week.

What branch of Cartodb/Cartodb20 are you using in your instance? I'm not able to reproduce this last error on my develop branch as of fb2ae81

tjwebb · 2012-12-11T10:14:51Z

I'm on develop. I just pulled the latest and now I can get the sample data to import, but my data will not import (http://dev.nbtsolutions.net/~tjwebb/mi.zip). Nothing ever happens; it says "importing" and the client just keeps checking forever and the server never says done and I see no errors. But this data imports fine at cartodb.com

demimismo · 2012-12-11T10:23:37Z

Mmmm, any feedback on the development log?

Also, you can take a look at the data_imports table, a record should be created there with some log information.

Thanks for all the feedback!

tjwebb · 2012-12-11T20:47:30Z

There is nothing in the development log, and the only entries in the data_imports table are from the other successful imports.

demimismo · 2012-12-11T22:48:47Z

So I think this is the same bug I explained on #51. Did you try to reboot the resque daemon and then import the file again?

How are you running the daemon QUEUE=* bundle exec rake resque:work?

tjwebb · 2012-12-11T23:37:26Z

bundle exec foreman start -p 3000

On Tue, Dec 11, 2012 at 5:48 PM, David Arango notifications@github.comwrote:

So I think this is the same bug I explained on #51 #51.
Did you try to reboot the resque daemon and then import the file again?

How are you running the daemon QUEUE=* bundle exec rake resque:work?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/50#issuecomment-11268354.

demimismo · 2012-12-11T23:49:58Z

Try to import the file after a full app restart, if it works then is the same bug I said before.

amnesic · 2012-12-15T19:19:47Z

Just in case it can help.
"But this data imports fine at cartodb.com"
with the mi.zip file coming from your URL http://dev.nbtsolutions.net/~tjwebb/mi.zip , it's failed on at cartodb.com with an Error 3102 (Unsupported projection). On local deployment, I've no error message but it's failed and a full app restart is need to be able to import again.

after transform you to the LAT/LON projection, you should be able to import it :
ogr2ogr -f "ESRI Shapefile" mi_4326.shp mi_zip.shp -t_srs EPSG:4326

tjwebb · 2012-12-15T19:27:22Z

Worked for me at cartodb.com. This data is there right now. Looks like they
have more problems surrounding this issue than they realize...

On Sat, Dec 15, 2012 at 2:19 PM, amnesic notifications@github.com wrote:

Just in case it can help.
"But this data imports fine at cartodb.com"
with the mi.zip file coming from your URL
http://dev.nbtsolutions.net/~tjwebb/mi.zip , it's failed on at cartodb.comwith an Error 3102 (Unsupported projection). On local deployment, I've no
error message but it's failed and a full app restart is need to be able to
import again.

after transform you to the LAT/LON projection, you should be able to
import it :
ogr2ogr -f "ESRI Shapefile" mi_4326.shp mi_zip.shp -t_srs EPSG:4326

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/50#issuecomment-11409152.

demimismo · 2012-12-17T10:48:47Z

Hey Travis,

As I said before, this bug seems to happen only on development mode. Did you manage to import the file after a full app restart?

Actually, I spent some time with it past week, I seems that you need to add realine pkg to rvm (see here)

amnesic · 2012-12-20T04:25:04Z

I've the same problem here (in development mode too). The first import works well, but the second failed and fall into an infinite loop.
Looking inside the data_import table, the failure row logger is


Begin:
TRANSITION: preprocessing => failure, 2012-12-20 04:39:25 +0100
ERROR!
UPDATE: table created
UPDATE: renaming cartodb_id from import file
UPDATE: cleaning supplied cartodb_id

The 'table_id' and 'data_source' art strangely empty only for this import.

Also I copy the resque workers log for this failed import in case it can help :


Worker
loic-VirtualBox:3986 on IMPORT_JOBS at just now Retry or Remove
Class
Resque::ImporterJobs
Arguments
"3b80b35f31a9039ce8b37543b25f644d777a6490"
{"user_id"=>2, "table_name"=>nil, "data_source"=>"http://cartodb.s3.amazonaws.com/static/50m-urban-area.zip", "table_id"=>nil, "append"=>false, "table_copy"=>nil, "from_query"=>nil}
Exception
NoMethodError
Error
undefined method `map' for nil:NilClass
/home/loic/Development/cartodb20/app/models/data_import.rb:129:in `block in after_create'
/home/loic/Development/cartodb20/app/models/data_import.rb:127:in `each'
/home/loic/Development/cartodb20/app/models/data_import.rb:127:in `after_create'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/model/base.rb:1585:in `block (2 levels) in _save'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/model/base.rb:867:in `around_create'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/model/base.rb:1578:in `block in _save'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/model/base.rb:867:in `around_save'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/state_machine-1.1.2/lib/state_machine/machine.rb:2067:in `block in around_save'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:149:in `block in run_actions'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:169:in `catch_exceptions'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:147:in `run_actions'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:132:in `run_callbacks'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:211:in `run_callbacks'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:63:in `block (2 levels) in perform'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:63:in `catch'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:63:in `block in perform'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:185:in `within_transaction'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:62:in `perform'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/state_machine-1.1.2/lib/state_machine/machine.rb:2067:in `around_save'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/model/base.rb:1573:in `_save'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/model/base.rb:1254:in `block (2 levels) in save'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/model/base.rb:1722:in `block in checked_transaction'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/database/query.rb:323:in `_transaction'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/database/query.rb:285:in `block in transaction'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/database/connecting.rb:236:in `block in synchronize'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/connection_pool/threaded.rb:104:in `hold'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/database/connecting.rb:236:in `synchronize'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/database/query.rb:278:in `transaction'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/model/base.rb:1722:in `checked_transaction'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/model/base.rb:1254:in `block in save'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/model/base.rb:1713:in `checked_save_failure'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/model/base.rb:1254:in `save'
/home/loic/.rvm/gems/ruby-1.9.3-p327@cartodb/gems/sequel-3.42.0/lib/sequel/model/base.rb:158:in `create'
/home/loic/Development/cartodb20/lib/resque/importer_jobs.rb:21:in `perform'

demimismo · 2012-12-20T15:11:13Z

Thank you so much for the log trace, @amnesic.

I spent more time with this today, but can't find the problem.

Tried compiling Ruby with readline support since I thought the problem could be related to the debugger gem, which gave us problems with Resque in the past, but the workers keep freezing after the first run. Also tried upgrading our gems (Rails, resque...), but the problem still persist.

I also spent some time exploring Resque issues unsuccessfully. Found this possibly related issue (https://github.com/defunkt/resque/issues/288), closed without much detail.

I can't spend much more time with this this week since this isn't a production issue. Maybe next week we'll find something, I'll keep you updated.

demimismo · 2012-12-20T15:16:23Z

Ah, I also debugged some import jobs and found that the perform method doesn't get invoked, so this isn't an issue with our Importer class, I'm pretty sure the problem is on Resque.

jcballesteros · 2013-03-04T10:17:42Z

Can you fix that problem? I have the same when i try to add shapefile in my local instalation of cartodb20.

demimismo · 2013-03-04T10:19:42Z

Could you share your shapefile with us? That's a different error message.

jcballesteros · 2013-03-04T10:24:01Z

Is a shapefile from cartodb open data (European Countries) . (All of the common data gives me that error).

I´m looking development.log (no errors appears) and postgresql-9.1-main.log (configured to write all sentences).

I only see that in the postgresql:

(0.000653s) UPDATE "data_imports" SET "user_id" = 2, "table_id" = NULL, "data_source" = 'http://cartodb.s3.amazonaws.com/static/european_countries.zip', "data_type" = 'url', "table_name" = NULL, "state" = 'importing', "success" = false, "logger" = '39985be6-8267-11e2-adfd-080027431acf', "updated_at" = '2013-03-01 12:57:45.449928+0100', "created_at" = '2013-03-01 12:57:43.467775+0100', "error_code" = 3005, "queue_id" = 'd464384fe83818394ede329bb17479fe97ca2378', "tables_created_count" = NULL, "table_names" = NULL WHERE ("id" = 1)
(0.001620s) UPDATE "data_imports" SET "user_id" = 2, "table_id" = NULL, "data_source" = 'http://cartodb.s3.amazonaws.com/static/european_countries.zip', "data_type" = 'url', "table_name" = NULL, "state" = 'importing', "success" = false, "logger" = '39985be6-8267-11e2-adfd-080027431acf', "updated_at" = '2013-03-01 12:57:45.453297+0100', "created_at" = '2013-03-01 12:57:43.467775+0100', "error_code" = 3005, "queue_id" = 'd464384fe83818394ede329bb17479fe97ca2378', "tables_created_count" = NULL, "table_names" = NULL WHERE ("id" = 1)
(0.000434s) UPDATE "data_imports" SET "user_id" = 2, "table_id" = NULL, "data_source" = 'http://cartodb.s3.amazonaws.com/static/european_countries.zip', "data_type" = 'url', "table_name" = NULL, "state" = 'importing', "success" = false, "logger" = '39985be6-8267-11e2-adfd-080027431acf', "updated_at" = '2013-03-01 12:57:45.458018+0100', "created_at" = '2013-03-01 12:57:43.467775+0100', "error_code" = 3005, "queue_id" = 'd464384fe83818394ede329bb17479fe97ca2378', "tables_created_count" = NULL, "table_names" = NULL WHERE ("id" = 1)
(0.000265s) UPDATE "data_imports" SET "user_id" = 2, "table_id" = NULL, "data_source" = 'http://cartodb.s3.amazonaws.com/static/european_countries.zip', "data_type" = 'url', "table_name" = NULL, "state" = 'importing', "success" = false, "logger" = '39985be6-8267-11e2-adfd-080027431acf', "updated_at" = '2013-03-01 12:57:45.460437+0100', "created_at" = '2013-03-01 12:57:43.467775+0100', "error_code" = 3005, "queue_id" = 'd464384fe83818394ede329bb17479fe97ca2378', "tables_created_count" = NULL, "table_names" = NULL WHERE ("id" = 1)

And

2013-03-04 09:51:31 CET LOG: no se pudo recibir datos del cliente: Conexión reinicializada por la máquina remota
(could not receive data from client: Conexión reset by the remote machine)
2013-03-04 09:51:31 CET LOG: se encontró fin de archivo inesperado en la conexión del cliente
(found unexpected EOF on client connection)

demimismo · 2013-03-04T10:46:05Z

Could you please log into the rails console (bundle exec rails c) and pase the output of this snippet?

DataImport[1].log

jcballesteros · 2013-03-04T11:27:00Z

I can´t access to the snippet.

demimismo · 2013-03-04T11:37:01Z

Sorry, I what I wanted is the output of DataImport[1].log.

You'll have to log into the rails console, do you need help with that?

jcballesteros · 2013-03-04T11:43:05Z

Yes i need help. I you have a manual (or a url), i can read and go back later with the log.

Sorry for my unfamiliarity with ruby, and thank you for your patience.

demimismo · 2013-03-04T11:48:44Z

Nice, you successfully logged in! Just run DataImport[1].log on that prompt and it should give you some output of the status of the import job.

Try Ruby is a nice way to learn the basics of the language if you feel like: http://tryruby.org/levels/1/challenges/0

jcballesteros · 2013-03-04T11:58:26Z

I tried again and when i put DataImport[1].log, appears this

When i see the psql log, i see that lines with error, but i dont know where is the problem:

2013-03-04 09:51:31 CET LOG: no se pudo recibir datos del cliente: Conexión reinicializada por la máquina remota
(could not receive data from client: Conexiwn reset by the remote machine)
2013-03-04 09:51:31 CET LOG: se encontró fin de archivo inesperado en la conexión del cliente
(found unexpected EOF on client connection)

jcballesteros · 2013-03-04T13:43:09Z

I tried to upload a CSV, i have 3006 error:

demimismo · 2013-06-28T09:24:02Z

Fixed: #51

…c-map Display header title and description on public visualization

tjwebb mentioned this issue Dec 11, 2012

Always "Checking import_jobs" #51

Closed

ghost assigned demimismo Dec 20, 2012

demimismo closed this as completed Jun 28, 2013

javisantana pushed a commit that referenced this issue Jun 17, 2016

Merge pull request #50 from flibbertigibbet/feature/header-desc-publi…

3228d07

…c-map Display header title and description on public visualization

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error when uploading SHP file on local dev instance #50

Error when uploading SHP file on local dev instance #50

tjwebb commented Dec 7, 2012

tjwebb commented Dec 10, 2012

tjwebb commented Dec 10, 2012

demimismo commented Dec 11, 2012

tjwebb commented Dec 11, 2012

demimismo commented Dec 11, 2012

tjwebb commented Dec 11, 2012

demimismo commented Dec 11, 2012

tjwebb commented Dec 11, 2012

demimismo commented Dec 11, 2012

amnesic commented Dec 15, 2012

tjwebb commented Dec 15, 2012

demimismo commented Dec 17, 2012

amnesic commented Dec 20, 2012

demimismo commented Dec 20, 2012

demimismo commented Dec 20, 2012

jcballesteros commented Mar 4, 2013

demimismo commented Mar 4, 2013

jcballesteros commented Mar 4, 2013

demimismo commented Mar 4, 2013

jcballesteros commented Mar 4, 2013

demimismo commented Mar 4, 2013

jcballesteros commented Mar 4, 2013

demimismo commented Mar 4, 2013

jcballesteros commented Mar 4, 2013

jcballesteros commented Mar 4, 2013

demimismo commented Jun 28, 2013

Error when uploading SHP file on local dev instance #50

Error when uploading SHP file on local dev instance #50

Comments

tjwebb commented Dec 7, 2012

tjwebb commented Dec 10, 2012

tjwebb commented Dec 10, 2012

demimismo commented Dec 11, 2012

tjwebb commented Dec 11, 2012

demimismo commented Dec 11, 2012

tjwebb commented Dec 11, 2012

demimismo commented Dec 11, 2012

tjwebb commented Dec 11, 2012

demimismo commented Dec 11, 2012

amnesic commented Dec 15, 2012

tjwebb commented Dec 15, 2012

demimismo commented Dec 17, 2012

amnesic commented Dec 20, 2012

demimismo commented Dec 20, 2012

demimismo commented Dec 20, 2012

jcballesteros commented Mar 4, 2013

demimismo commented Mar 4, 2013

jcballesteros commented Mar 4, 2013

demimismo commented Mar 4, 2013

jcballesteros commented Mar 4, 2013

demimismo commented Mar 4, 2013

jcballesteros commented Mar 4, 2013

demimismo commented Mar 4, 2013

jcballesteros commented Mar 4, 2013

jcballesteros commented Mar 4, 2013

demimismo commented Jun 28, 2013