Tutorial command no longer works

[petey284]

Issue with command on tutorial on Simon's site.

The following command no longer works, and breaks with the previous too many variables error: #50

> curl "https://data.nasa.gov/resource/y77d-th95.json" | \
    sqlite-utils insert meteorites.db meteorites - --pk=id

Output:

Traceback (most recent call last):
  File "continuum\miniconda3\envs\main\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "continuum\miniconda3\envs\main\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "Continuum\miniconda3\envs\main\Scripts\sqlite-utils.exe\__main__.py", line 9, in <module>
  File "continuum\miniconda3\envs\main\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "continuum\miniconda3\envs\main\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "continuum\miniconda3\envs\main\lib\site-packages\click\core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "continuum\miniconda3\envs\main\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "continuum\miniconda3\envs\main\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "continuum\miniconda3\envs\main\lib\site-packages\sqlite_utils\cli.py", line 434, in insert
    default=default,
  File "continuum\miniconda3\envs\main\lib\site-packages\sqlite_utils\cli.py", line 384, in insert_upsert_implementation
    docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs
  File "continuum\miniconda3\envs\main\lib\site-packages\sqlite_utils\db.py", line 1081, in insert_all
    result = self.db.conn.execute(query, params)
sqlite3.OperationalError: too many SQL variables

My thought is that maybe the dataset grew over the last few years and so didn't run into this issue before.

No error when I reduce the count of entries to 83. Once the number of entries hits 84 the command fails.

// This passes

type meteorite_83.txt | sqlite-utils insert meteorites.db meteorites - --pk=id

// But this fails

type meteorite_84.txt | sqlite-utils insert meteorites.db meteorites - --pk=id

A potential fix might be to chunk the incoming data? I can work on a PR if pointed in right direction.

[simonw]

This should work - the data should be chunked automatically. It looks like this is a bug.

[simonw]

I can't replicate this problem:

/tmp $ sqlite-utils --version
sqlite-utils, version 2.2
/tmp $ curl "https://data.nasa.gov/resource/y77d-th95.json" |     sqlite-utils insert meteorites.db meteorites - --pk=id
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  240k    0  240k    0     0   185k      0 --:--:--  0:00:01 --:--:--  185k

Could you run sqlite-utils --version and tell me what you get?

[simonw]

Closing as can't reproduce.