Do a basic check of task results before using the files

[ericholscher]

No description provided.

[ericholscher]

@agjohnson not really happy with this solution, but it at least should solve the 99% case of search getting indexed before the files sync.

[ericholscher]

@agjohnson not really happy with this solution, but it at least should solve the 99% case of search getting indexed before the files sync.

[ericholscher]

Other obvious option is to run it as part of the syncing task, but currently they run on each web, so it would run 4x. Not sure the best middle solution.

[ericholscher]

Other obvious option is to run it as part of the syncing task, but currently they run on each web, so it would run 4x. Not sure the best middle solution.

[agjohnson]

This is another case for celery 3+ and task chaining[1]. This is replicating some of the promise structure that a modern celery provides.

I haven't revisited my branch to upgrade celery in a bit, but could put some priority on it.

1: http://docs.celeryproject.org/en/latest/userguide/canvas.html#chains

[agjohnson]

This is another case for celery 3+ and task chaining[1]. This is replicating some of the promise structure that a modern celery provides.

I haven't revisited my branch to upgrade celery in a bit, but could put some priority on it.

1: http://docs.celeryproject.org/en/latest/userguide/canvas.html#chains

[ericholscher]

Right, but what do we chain?

[ericholscher]

Right, but what do we chain?

[ericholscher]

We're using the proper version of celery that has linking..

[ericholscher]

We're using the proper version of celery that has linking..

[agjohnson]

Ah 3.x, i thought we were 2.x for some reason -- good.

Theoretically we chain on a group, this is what chord() seems to accomplish. I haven't tested this yet though.

[agjohnson]

Ah 3.x, i thought we were 2.x for some reason -- good.

Theoretically we chain on a group, this is what chord() seems to accomplish. I haven't tested this yet though.

[agjohnson]

A quick test shows an example of what we're ultimately looking for, though I'm calling .get here, which still ties up the builder waiting for replies, something we don't want:
https://gist.github.com/agjohnson/19666953a7673cd24630b7091f37b8ca

Pushing this all on to celery makes more sense as if a web queue is offline or backed up, we won't hit the 5sec * 4 timeouts and won't hit deadlocks. We'll want to try to make sure the linked task/chord has a timeout though, as extended backup on the webs could cause a large number of chord unlocks on the celery workers.

I get a loop of unlocking the chord on the example above locally though. I'm not sure if it's related to the version, but upgrading 3.1.23 -> 4.1.x resolved the issue. Things worked on 3.1.23, there just seemed to be unwarranted unlock tasks executed for some reason.

[agjohnson]

A quick test shows an example of what we're ultimately looking for, though I'm calling .get here, which still ties up the builder waiting for replies, something we don't want:
https://gist.github.com/agjohnson/19666953a7673cd24630b7091f37b8ca

Pushing this all on to celery makes more sense as if a web queue is offline or backed up, we won't hit the 5sec * 4 timeouts and won't hit deadlocks. We'll want to try to make sure the linked task/chord has a timeout though, as extended backup on the webs could cause a large number of chord unlocks on the celery workers.

I get a loop of unlocking the chord on the example above locally though. I'm not sure if it's related to the version, but upgrading 3.1.23 -> 4.1.x resolved the issue. Things worked on 3.1.23, there just seemed to be unwarranted unlock tasks executed for some reason.

[ericholscher]

@agjohnson should be ready for another review.

[ericholscher]

@agjohnson should be ready for another review.