Fix new bulk import queuing

[rkorytkowski]

When scheduling some NPM packages the import does never end. The problematic package is hl7.fhir.r4.core:4.0.1. It enters an endless loop at the last group of tasks, which are being repeated in a loop.

[rkorytkowski]

The above fix is following https://stackoverflow.com/a/19018521, but it causes another issue of the initial task hanging (appears to be waiting for results) instead of returning quickly and executing all subtasks asynchronously. The subtasks do run, but extremely slow... In ECS the worker even fails healthchecks by not responding in time. Locally I just see missed heartbeats for workers. It'a very strange behavior.

[rkorytkowski]

Some smaller packages that I hand-crafted do import correctly... I'll attach them shortly.

[rkorytkowski]

hl7.fhir.r4.core-4.0.1-subset.zip
hl7.fhir.r4.core-4.0.1-subset-all-conceptmaps.zip

[rkorytkowski]

Good thing is that I found celery/celery#5958 and apparently it was fixed in Celery. I upgraded Celery and the initial solution when tasks were repeated endlessly seems no longer to be an issue, but still need to fix the issue with initial task not finishing after scheduling all tasks (probably entering some endless loop).

[rkorytkowski]

Finally fixed all issues! The thing was that scheduling more than 1.5k tasks with groups and chains is not handled well by celery. Celery queue gets very slow then. I tweaked the algorithm to create less tasks. I'm dividing the bulk import in a way to always be less than 1k tasks for now. A single task should not be less than 100 resources. So if the package has 500k resources then each task would import 500 resources. I'll cleanup and commit tomorrow.