On Windows, importing or exporting a large number of items (~1000) can fail

[ghost]

Observed behavior

When trying to download content i get a transfer failed error. First channel downloaded in bulk ok but cannot download further channelsin bulk. If i go in and download sub categories i can eventually pull down the whole channel

Expected behavior

channel downloads as a whole not 1 by 1 resource sub sets

Errors and logs

�[33mWARNING  Job d416c174faeb4648b22bd1420a75f2a0 raised an exception: Traceback (most recent call last):
  File "c:\python27\lib\site-packages\kolibri\dist\iceqube\worker\backends\inmem.py", line 75, in handle_finished_future
    result = future.result()
  File "c:\python27\lib\site-packages\kolibri\dist\concurrent\futures\_base.py", line 455, in result
    return self.__get_result()
  File "c:\python27\lib\site-packages\kolibri\dist\concurrent\futures\thread.py", line 63, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\python27\lib\site-packages\kolibri\dist\iceqube\worker\backends\inmem.py", line 149, in wrap
    raise e
OperationalError: too many SQL variables
�[0m
WARNING:iceqube.worker.backends.base:Job d416c174faeb4648b22bd1420a75f2a0 raised an exception: Traceback (most recent call last):
  File "c:\python27\lib\site-packages\kolibri\dist\iceqube\worker\backends\inmem.py", line 75, in handle_finished_future
    result = future.result()
  File "c:\python27\lib\site-packages\kolibri\dist\concurrent\futures\_base.py", line 455, in result
    return self.__get_result()
  File "c:\python27\lib\site-packages\kolibri\dist\concurrent\futures\thread.py", line 63, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\python27\lib\site-packages\kolibri\dist\iceqube\worker\backends\inmem.py", line 149, in wrap
    raise e
OperationalError: too many SQL variables

Context

win 10 running kolibri 0.7.0. all browsers.

[jonboiser]

@edunetck Would you mind sharing details of which channels and sub-categories you attempted to import? Which worked, and which did not work?

[ghost]

Hi
Importing from Koilbri Online. Touchable Earth (337 resources) and Be Strong (65 resources) imoprted ok. The other channels wont bulk import - im guessing it may have something to do with the number of resources in each channel. For example i have nearly finished importing EngageNY but i have had to do it a couple of chapters at a time - each chapter having X number of resources. If i select too many chapters it will fail so have to reduce to 1,2 or 3 chapters at a time and the import works. If i select say 5 chapters and it fails i can go back and import each chapter individually andf it runs ok......

[jonboiser]

@edunetck Thanks for the info. We will try to reproduce it on our side. I'll check the Engage NY channel to see what 1-5 chapters translates to in terms of resources.

To clarify, the largest number of chapters on Engage NY has been 3? But attempting 5 chapters fails?

[ghost]

it seems to fail if the total number of resources, irrespective of the number of chapters gets close to 1000 in one download. EngageNY has 8678 resources, each chapter is between 250 and 450 resources. sometimes i can get 3, sometimes 2 chapters. ANd yes - have not been able to get more than 3 chapters but ihavent speciffically gone and looked for all the 200 resource chapters just to try that scenario.

[ghost]

Grade 5 mathematics in EnganeNY (EN) wont import at all, get same failed error. All other chapters imported ok.

[ghost]

And now when Exporting facing the same issue.

[jonboiser]

@edunetck I just tried an import of about 1200 resources from EngageNY and it was successful (on a MacOS computer). We will try to reproduce this on a Windows machine.

[jonboiser]

@edunetck @radinamatic Was able to reproduce the same exact error on a Windows 10 machine. In her case, the threshold was slightly under 1000 resources from EngageNY.

I will rename this issue to make it easier for others with the same problem to find it and we'll keep you posted on when it's resolved.

[lauradanforth]

@jonboiser for reference, this was raised by a couple people on the forum as well: Threads are here: https://community.learningequality.org/t/ck-12-and-ka-lite-channels-failing-to-download-0-7-0/467 and https://community.learningequality.org/t/is-kolibri-central-server-down/482/3

[jamalex]

Looks like it's due to the implementation of _get_leaves_ids in https://github.com/learningequality/kolibri/pull/2703/files (CC @lyw07 @aronasorman). We shouldn't be creating a list of leaf IDs, we should leverage django-mptt's hierarchical queries. I'll give it a shot; do we have any test coverage on this?

[jamalex]

Fixed in #3163