Exporting crash, "peewee.OperationalError: too many SQL variables" #8

kelson42 · 2014-07-21T09:02:49Z

command:
./dump-gutenberg.py -l fr,es -f pdf,epub

log:

GET /4/6/3/1/46314/46314-h.htm HTTP/1.1" 404 252
EXPORTING ebooks to satic folder (and JSON)
        Filtered book collection size: 2808
        Filtered book collection, PDF: 45
        Filtered book collection, ePUB: 2778
        Filtered book collection, HTML: 2786
                Dumping full_by_popularity.js
                Dumping full_by_title.js
                Dumping lang_fr_by_popularity.js
                Dumping lang_fr_by_title.js
                Dumping authors_lang_fr.js
                Dumping lang_es_by_popularity.js
                Dumping lang_es_by_title.js
                Dumping authors_lang_es.js
Traceback (most recent call last):
  File "./dump-gutenberg.py", line 150, in <module>
    main(docopt(help, version=0.1))
  File "./dump-gutenberg.py", line 137, in main
    only_books=BOOKS)
  File "/media/data/tmp/gutenberg/gutenberg/export.py", line 96, in export_all_books
    formats=formats)
  File "/media/data/tmp/gutenberg/gutenberg/export.py", line 576, in export_to_json_helpers
    for author in authors:
  File "/home/kelson/.virtualenvs/gut/local/lib/python2.7/site-packages/peewee.py", line 2139, in __iter__
    return iter(self.execute())
  File "/home/kelson/.virtualenvs/gut/local/lib/python2.7/site-packages/peewee.py", line 2132, in execute
    self._qr = ResultWrapper(model_class, self._execute(), query_meta)
  File "/home/kelson/.virtualenvs/gut/local/lib/python2.7/site-packages/peewee.py", line 1838, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File "/home/kelson/.virtualenvs/gut/local/lib/python2.7/site-packages/peewee.py", line 2414, in execute_sql
    self.commit()
  File "/home/kelson/.virtualenvs/gut/local/lib/python2.7/site-packages/peewee.py", line 2283, in __exit__
    reraise(new_type, new_type(*exc_value.args), traceback)
  File "/home/kelson/.virtualenvs/gut/local/lib/python2.7/site-packages/peewee.py", line 2406, in execute_sql
    cursor.execute(sql, params or ())
peewee.OperationalError: too many SQL variables

The text was updated successfully, but these errors were encountered:

Seb35 · 2014-07-21T09:10:09Z

I had this problem on my computer too for large selections like you have here, but it worked on Renaud’s MacBook probably more powerful. Anyway this is a performance bug which should be resolved.

kelson42 · 2014-09-22T07:27:34Z

I'm stuck on the server with the EN export at this bug. Could someone please fix it urgently?

kelson42 · 2014-09-23T23:17:58Z

There is a problem/regression!

Assuming we make a ZIM file with only --languages=fr then authors.js = authors_lang_fr.js = contains all authors. Both should only contain french speak authors (like before).

kelson42 · 2014-09-24T14:06:57Z

In addition, by doing a full export (all books of Gutenberg)... we still have the problem:

$ rm -rf static/ ; ./dump-gutenberg.py --keep-db --export --zim
'list' object has no attribute 'split'
EXPORTING ebooks to static folder (and JSON)
        Filtered book collection size: 45468
        Filtered book collection, PDF: 960
        Filtered book collection, ePUB: 45331
        Filtered book collection, HTML: 45299
                Dumping full_by_popularity.js
                Dumping full_by_title.js
                Dumping lang_en_by_popularity.js
                Dumping lang_en_by_title.js
                Dumping authors_lang_en.js
Traceback (most recent call last):
  File "./dump-gutenberg.py", line 150, in <module>
    main(docopt(help, version=0.1))
  File "./dump-gutenberg.py", line 137, in main
    only_books=BOOKS)
  File "/media/data/gutenberg/gutenberg/export.py", line 107, in export_all_books
    formats=formats)
  File "/media/data/gutenberg/gutenberg/export.py", line 609, in export_to_json_helpers
    Author.first_names.asc())],
  File "/home/kelson/.virtualenvs/gut/local/lib/python2.7/site-packages/peewee.py", line 2139, in __iter__
    return iter(self.execute())
  File "/home/kelson/.virtualenvs/gut/local/lib/python2.7/site-packages/peewee.py", line 2132, in execute
    self._qr = ResultWrapper(model_class, self._execute(), query_meta)
  File "/home/kelson/.virtualenvs/gut/local/lib/python2.7/site-packages/peewee.py", line 1838, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File "/home/kelson/.virtualenvs/gut/local/lib/python2.7/site-packages/peewee.py", line 2414, in execute_sql
    self.commit()
  File "/home/kelson/.virtualenvs/gut/local/lib/python2.7/site-packages/peewee.py", line 2283, in __exit__
    reraise(new_type, new_type(*exc_value.args), traceback)
  File "/home/kelson/.virtualenvs/gut/local/lib/python2.7/site-packages/peewee.py", line 2406, in execute_sql
    cursor.execute(sql, params or ())
peewee.OperationalError: too many SQL variables

rashiq · 2014-09-25T20:41:54Z

Hi @kelson42

Hmm this is weird. I can't reproduce the bug with ./dump-gutenberg.py --keep-db --export --zim. Everything seems to work fine. I do get a crash with ./dump-gutenberg.py -l fr,es -f pdf,epub though.

I'm investigating.

kelson42 · 2014-09-25T22:09:25Z

You should be able at least to reproduce the problem with the authors list. The other one is a performance problem so related to your HW I guess.

rashiq · 2014-09-25T22:45:41Z

Just tried to run ./dump-gutenberg.py -l te -f pdf,epub (-l te instead of -l fr,es, because there are way to many books for french and spanish than for telugu for example, and I just wanted to quickly test it - but in theory both should work the same) and it worked. I didn't get the error :/

kelson42 · 2014-09-25T22:55:59Z

$ rm -rf static/ ; ./dump-gutenberg.py -l te --export ; ls -la static/authors*.js
'list' object has no attribute 'split'
EXPORTING ebooks to static folder (and JSON)
Filtered book collection size: 5
Filtered book collection, PDF: 0
Filtered book collection, ePUB: 5
Filtered book collection, HTML: 5
Dumping full_by_popularity.js
Dumping full_by_title.js
Dumping lang_te_by_popularity.js
Dumping lang_te_by_title.js
Dumping authors_lang_te.js
Dumping auth_39781_by_popularity.js
Dumping auth_39781_by_title.js
Dumping authors.js
Dumping languages.js
Dumping main_languages.js
Exporting Book #39004.
Exporting to static/శుభలేఖ.39004.html
Copying format file to శుభలేఖ.39004.epub
Creating ePUB at /tmp/tmpEm5Vxp.epub
Exporting to static/శుభలేఖ_cover.39004.html
Exporting Book #39561.
Exporting to static/అగ్నిగుండం.39561.html
Copying format file to అగ్నిగుండం.39561.epub
Creating ePUB at /tmp/tmpaUemp0.epub
Exporting to static/అగ్నిగుండం_cover.39561.html
Exporting Book #40687.
Exporting to static/కొల్లాయి గట్టితే నేమి?.40687.html
Copying format file to కొల్లాయి గట్టితే నేమి?.40687.epub
Creating ePUB at /tmp/tmp3IGrp9.epub
Exporting to static/కొల్లాయి గట్టితే నేమి?_cover.40687.html
Exporting Book #41845.
Exporting to static/ఓనమాలు.41845.html
Copying format file to ఓనమాలు.41845.epub
Creating ePUB at /tmp/tmpbbOEs2.epub
Exporting to static/ఓనమాలు_cover.41845.html
Exporting Book #43220.
Exporting to static/కత్తుల వంతెన.43220.html
Copying format file to కత్తుల వంతెన.43220.epub
Creating ePUB at /tmp/tmpHxlRgS.epub
Exporting to static/కత్తుల వంతెన_cover.43220.html
[]
-rw-rw-r-- 1 kelson kelson 524023 Sep 26 00:51 static/authors.js

-rw-rw-r-- 1 kelson kelson 524023 Sep 26 00:51 static/authors_lang_te.js

Do you seriously believe they are for 500KB of authors in Telegu? ;) Or you get an other result?

rashiq · 2014-09-25T23:23:03Z

Yeah I chose the language telugu because there are so few books in telugu (so I don't have to download for an hour to reproduce the error):

$ du -h static/authors.js 
516K    static/authors.js

There are only 5 books in telugu:

sqlite> SELECT * FROM book WHERE language = 'te';
39004|శుభలేఖ||39781|PD|te|31
39561|అగ్నిగుండం||39781|PD|te|40
40687|కొల్లాయి గట్టితే నేమి?||39781|PD|te|19
41845|ఓనమాలు||39781|PD|te|20
43220|కత్తుల వంతెన||39781|PD|te|16

So the generated file has to be correct.

I'll try to run the english and spanish book-downloading over the night to reproduce the exact error :)

rashiq added the bug label Jul 21, 2014

rashiq assigned rgaudin Jul 21, 2014

kelson42 mentioned this issue Sep 18, 2014

Too many SQL Variables on Export #11

Closed

kelson42 changed the title ~~Exporting crash~~ Exporting crash, "peewee.OperationalError: too many SQL variables" Sep 22, 2014

rgaudin closed this as completed in 34f7618 Sep 22, 2014

kelson42 reopened this Sep 23, 2014

rgaudin closed this as completed in e9ba81e Sep 29, 2014

kelson42 unassigned rgaudin Apr 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exporting crash, "peewee.OperationalError: too many SQL variables" #8

Exporting crash, "peewee.OperationalError: too many SQL variables" #8

kelson42 commented Jul 21, 2014

Seb35 commented Jul 21, 2014

kelson42 commented Sep 22, 2014

kelson42 commented Sep 23, 2014

kelson42 commented Sep 24, 2014

rashiq commented Sep 25, 2014

kelson42 commented Sep 25, 2014

rashiq commented Sep 25, 2014

kelson42 commented Sep 25, 2014

rashiq commented Sep 25, 2014

Exporting crash, "peewee.OperationalError: too many SQL variables" #8

Exporting crash, "peewee.OperationalError: too many SQL variables" #8

Comments

kelson42 commented Jul 21, 2014

Seb35 commented Jul 21, 2014

kelson42 commented Sep 22, 2014

kelson42 commented Sep 23, 2014

kelson42 commented Sep 24, 2014

rashiq commented Sep 25, 2014

kelson42 commented Sep 25, 2014

rashiq commented Sep 25, 2014

kelson42 commented Sep 25, 2014

-rw-rw-r-- 1 kelson kelson 524023 Sep 26 00:51 static/authors_lang_te.js

rashiq commented Sep 25, 2014