Create bulk_add_nodes internal endpoint for initiating bulk creation of nodes

[rtibbles]

Desired behavior

A new API endpoint that will take a bulk set of nodes with either channel information, or a parent_id and initiate an asynchronous task to create a new tree or insert them into an existing ricecooker tree. It should return the task_id of the generated task in order to allow ricecooker to poll for updates to the task from the task endpoint.

The main data for the POST request should be a JSON array of objects, each of which represents a ContentNode.

This data should be in the same format as that currently used by

studio/contentcuration/contentcuration/views/internal.py

Line 271 in 9d84374

def api_add_nodes_to_tree(request):

but with an additional optional children key that will contain other ContentNodes.

Stretch goal:

Generate file upload URLs 'as you go' for any file information generated, which can then be fed back to ricecooker during task polling. Ricecooker can than start doing direct file uploads to GCS while the tree is being generated.

Required for learningequality/ricecooker#321

Some suggested stages for tackling this issue:

1. Make a clone of the existing api_add_nodes_to_tree API view that runs the node adding in an asynchronous task and returns the task_id instead - possible that with the existing payload it is feasible to pass the payload solely as task arguments.
2. Investigate size limitations for this approach and add error checking if the API is sent JSON that is too large.
3. If the size of the JSON that can be used in this way is very small, investigate ways to store the JSON in the database from the endpoint, and then reference this saved JSON in the asynchronous task.
4. Optimize the node creation using similar bulk creation methods used in the copying logic in https://github.com/learningequality/studio/blob/unstable/contentcuration/contentcuration/db/models/manager.py#L338

[ozer550]

@rtibbles I would like to work on this issue. I'm new to the Kolibri ecosystem. Can I get some pointers to files where should I look into to implement this?

[rtibbles]

The linked file: https://github.com/learningequality/studio/blob/hotfixes/contentcuration/contentcuration/views/internal.py#L522 here is the best place to start.

In addition, to see how we have efficiently managed bulk node creation in another part of the code base, follow the implementation of the copy_node method of the ContentNode manager class: https://github.com/learningequality/studio/blob/hotfixes/contentcuration/contentcuration/db/models/manager.py#L331

[vkWeb]

@rtibbles assigning this to @ozer550 and myself. We both will pair program this feature together.

[vkWeb]

We both got busy with solving and collobaroting on more important issues. So, this didn't get addressed. Unassigning both of us until we visit this again.

[AllanOXDi]

Let me try to bite it.

[akash5100]

Hi there! I'm interested in this issue. Is anyone currently working on this issue or is it available for open-source contribution? Thank you!

[rtibbles]

Hi @akash5100, thanks for your interest. Unfortunately since I wrote this issue, I am not sure that this is the right path forward, so I may close or rewrite this issue.

[akash5100]

Thank you for your quick reply, @rtibbles! I appreciate your help.
I was wondering if there are any other issues that are available for open-source contributions. I am very interested in contributing to open-source projects and would love to get involved in this project(studio). Thank you for your time and assistance.