Proposal: every celery task should execute mwoffliner directly, instead of running a shell script

[automactic]

Our current worker implementation is to run a shell script (examples), which seems to execute mwoffliner multiple times and generate multiple zim files. In this way, every celery task could generate multiple zim files.

I think, instead of doing this, it would be better to runmwoffliner once, generate one zim file per celery task. Reasons:

1. security: if we allow user to enqueue celery task that execute any command, we expose workers to shell injection attack, accidental file deletion, etc. It's better to have worker set parameters used in mwoffliner in dispatcher and assemble the command programmatically on the worker.
2. distributed system performance: by breaking big tasks into smaller units, more worker potentially could participate at the same time, thereby speeding up the overall process.
3. management:

• the stdout & stderr contains messages regarding one zim file generation
• every celery task need to upload one zim file, easier to figure out the uploading progress and ETA

4. error recovery: In case of error, if every celery task produce multiple zim files, it's

• hard to figure out which generated zim file has error, which doesn't
• impossible for another worker to pick up the process without unnecessarily re-generate zim files that does not have error

[kelson42]

@automactic I agree with this.

[automactic]

further discussion continues on #8