Update loader.py to support multiple UPDATEs in a single SQL command #25

Closed
gaurav opened this Issue Dec 22, 2011 · 3 comments

Comments

Projects
None yet
2 participants

gaurav commented Dec 22, 2011

Uploading data line-by-line takes a long time. It should be much faster to send over a single SQL command in the format, "UPDATE x SET ....; UPDATE x SET ....; UPDATE x SET .....;" Shouldn't be too hard to get this working on loader.py. Might be a good idea to get this working on GET for now, or even on non-functional POST, since it's going to be a priority for once we have POST up and running.

@ghost ghost assigned gaurav Dec 22, 2011

gaurav commented Dec 22, 2011

As of @bf40a52, this is working! A test upload managed to upload 2061 rows out of 2066 rows (i.e. 5 rows missing). I think maybe that's one or two rows being too big for upload via GET, but it's hard to tell: it it was a perfect multiple of 6, I'd be pretty sure it'll be fixed once we get POST working (i.e. issue #13), but until then, I'll try to see if I can find a bug in this.

gaurav commented Dec 31, 2011

This appears to be working: I just uploaded 2,066 rows 100 rows at a time without missing a single row. I'll try it out on a few more datasets before closing this bug, but I've merged this code into the 'develop' branch for testing (as of @d5e0092225c).

gaurav commented Jan 14, 2012

Okay, the log file on our 141,250 uploads has a discrepancy of 5 features (the log indicates that 141,255 features were uploaded). So this isn't ready for testing, but this is hard to replicate on smaller datasets. I will leave this as an outstanding bug for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment