update the watchdog file

[allieychen]

Fix one bug for annotation. The operations could get killed because of the watchdog file is stale even though no cancellation is made. The watchdog file was updated only when waiting for one operation to be done. This fix handles the following two failing cases:

• If there are a lot of PAPI calls, some operations might already start running VEP (after loading cache) while we are still submitting other operations, the stale watchdog file will get this operation to be killed.
• If there are a lot of consecutive done operations, the watchdog file won't be updated in time.

Tested: unit tests & integration tests & manually ran one annotation test.

[coveralls]

Pull Request Test Coverage Report for Build 1562

• 7 of 17 (41.18%) changed or added relevant lines in 2 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage decreased (-0.02%) to 88.958%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
gcp_variant_transforms/libs/annotation/vep/vep_runner.py	2	3	66.67%
gcp_variant_transforms/transforms/annotate_files.py	5	14	35.71%

Files with Coverage Reduction	New Missed Lines	%
gcp_variant_transforms/transforms/annotate_files.py	1	45.16%

Totals
Change from base Build 1548:	-0.02%
Covered Lines:	7041
Relevant Lines:	7915

---

💛 - Coveralls

[kemp-google]

@allieychen Can you put the commentary about why this change is being made in the commit itself? It's much better to have information like this in the commit object rather than the PR since the PR is not a permanent part of the record.

[nmousavi]

Can we separate file update from runners logic, perhaps by running an updater in different thread?
I don't like the fact that update is done in multiple places, it's prone to bug and ugly. Any future change that adds a costly processing step here would require another manual update (similar to what you added in this PR), and it can be easily missed out.

[allieychen]

Thanks for the review! I have moved updating the watchdog file in another thread. PTAL :)

[nmousavi]

Please test it on GCP, and mentioned it in the description.
Once done LGTM.