Retry recreate_index subtasks on worker exit

[mbertrand]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/5615

Description (What does it do?)

• Retry subtasks on SystemExit
• Refactor task options
• In docker, tasks only seem to be retried after a shutdown if there is one celery command issued instead of 2, so adjusted the docker-compose command for celery startup.

How can this be tested?

Testing works best if you have lots of courses and contentfiles ingested, otherwise the indexing task might finish before the celery worker stops. You could try running the folliowing after setting OLL_ settings to the same as RC, and also SKIP_TIKA=True to speed things up a bit:

./manage.py backpopulate_oll_data
./manage.py backpopulate_oll_files

Alternately, you could add some sleep(10) calls within the indexing subtask functions to slow them down.

Open 3 terminals:

• Terminal 1: docker compose up
• Terminal 2: when ready, ./manage.py recreate_index --all
• Terminal 3: Once you start seeing a bunch of recreate_index subtasks running like index_learning_resources, run docker compose stop celery. Wait until you see that celery has exited in terminal 1. Then run docker compose up celery
• Once celery has restarted, the recreate_index tasks should resume, and eventually the task in terminal 2 should complete successfully.

Additional Context

[shanbady]

This appears to work well! I tried breaking it in various ways and couldn't.