resource service triggers background tasks before transaction is committed

[sanderr]

The orchestration service calls the resource service's mark_deployed method in a transaction context. The latter passes it on to resource_action_update, which adds a background task for ConfigurationModel.mark_done_if_done and for agent.resource_event. These background tasks are added before the transaction is committed.

Consequences (hypothesis):

1. mark_done_if_done inspects resource state. I believe it now races (and typically loses) with the transaction commit so that it never sees all resources as deployed. The configuration model is therefore never marked as deployed. This impacts e.g. inmanta-cli version list. An easy reproduction is to simply run the quickstart. Export a new version without changes to the resources, then run inmanta-cli --host 172.30.0.3 version list -e SR_Linux. It will remain in deploying until a repair is triggered. The web console does show all resources as deployed.
2. Does agent.resource_event inspect any database state (through an API call)? Is it safe to be used in this transaction context? We may be sending an event for something that hasn't committed yet. When the agent responds to the event by querying the server for resource state, it may get the wrong state.
3. There may or may not be other flows that pass a connection object into this method.

I did not go more in-depth at this time, because it is not entirely clear to me what is the best way forward. The mark_done_if_done approach might be no longer relevant, in which case it might be better to drop it and use something else for inmanta-cli version list rather than to address the concurrency problem. More input is needed from the team.

[sanderr]

1. Investigate scope of the race condition: which background tasks are affected and which are problematic?
2. If you can fix it in +- 1 day, fix it. Otherwise report back.

[wouterdb]

1. this is the only place where we start a background task in the scope of a transaction