Progress bar for loading from git [$50 awarded]

[bahramhamid]

Make the "save" button of subproject, asynch and show kind of progress bar for loading from the git

[nijel]

Please keep it just one feature request per issue, otherwise it's hard to track.

[nijel]

What I think is needed to implement this:

• Make component creation just save the database model and trigger further update in Celery
• Store flag that the component update is in progress to properly report this in UI
• Once Celery gets to processing, it should log the status and progress
• UI should indicate that that the component is being updated using status icon
• Detailed view showing import progress (link from icon and redirect on component creation)
• Cancel button to terminate the task

Unclear topics:

• Storage for the update flag and progress - cache or database? [probably cache]
• Store celery result ID to be able to terminate the task [in cache]
• How to flag the component update? [cache + celery task state]
• Should this be integrated into Django admin as well [as it will modify the save, it will work out of the box, just the progress won't be shown in the Django admin]
• How to pipe updates from server to client? Websockets or polling? [probably polling for now]

[rhofer]

Potentially to add:

• (Optional) If Email configured, drop email on success / failure to admin role (or similar)

This could be similar as what happening when ci pipeline (gitlab / github) is running with according results (success/failure)

[nijel]

The notification will be already covered by #1956.

[rhofer]

@nijel yes thanks. That should do it.

Another remark:

Error handling / exit strategy: Basically we should remove time-out based processing and really going for asynchronous based on state-event also leading to a progress observation. Additionally, we need to avoid to be caught in an "endless loop". Potentially to provide:

• Cancel / Abort - Button to stop importing (szenario: import started, thereafter, recognized it was the wrong thing, but not to wait for hours until finished...)
• job queueing: what if multiple import with multiple large entities is started --> show queue
  • ... or alternatively for time being, only allow import one after the other (no queue management needed)

/cc @fennibay

[rhofer]

Another scenario which should belong to CRUD with components (import is covered above)

• Deletion of large component to be handled asynchronously, as well

Facing similar problems as with import of large components.

[nijel]

Thank you for your report, the issue you have reported has just been fixed.

• In case you see problem with the fix, please comment on this issue.
• In case you see similar problem, please open separate issue.
• If you are happy with the outcome, consider supporting Weblate by donating.

[nijel]

@rhofer @alesrosina Looking at #2721 I've realized that you're not running Celery (see https://docs.weblate.org/en/latest/admin/install.html#celery) in the OpenShift. Without that this feature will not work, so the OpenShift setup will need some adjustments...

[alesrosina]

@nijel hm, thanks for pointing this out. I will try to fix that.

[alesrosina]

@GrahamDumpleton hey, so I was trying to setup celery also on openshift, but failed - I did it as you recomended here: https://stackoverflow.com/questions/48102638/how-to-run-celery-with-django-on-openshift-3
But for me somehow the process is just respawing all the time (got up to 120 running celery workers). Also if I run it just as ./weblate/examples/celery start before exec mod_wsgi-express start-server $ARGS in .s2i/action_hooks/run.
Any recomendations on how to run celery? As far as I understood, celery itself (https://hub.docker.com/_/celery) recommends running celery inside application container (or pod in openshift).

Thanks for any hints!

[GrahamDumpleton]

@alesrosina I would suggest you create a separate issue/question in some forum and explain from scratch what you are trying to do, what you have done so far, and what errors you get recorded anywhere. It isn't much use to point at StackOverflow questions for someone else's issue which was a completely different setup than you likely have as impossible to guess what you have actually done.