-
Notifications
You must be signed in to change notification settings - Fork 2
Transient Errors
Transient errors are errors caused by bad luck. Resolving transient errors is almost always very simple: Simply rerun the pipeline exactly the same as when it failed, and it will work, as if by magic.
The real issue here is that the transience of an error message is often not obvious, and if the underlying cause is a serious system outage, you're going to keep failing until the service comes back up.
In my experience, Terra in particular is relatively prone to transient errors. This is because a workflow on Terra requires all of these systems to work perfectly:
- Google Cloud storage (gs://)
- GCP (the VMs your workflow runs on)
- Docker Hub, or whatever other container registry you rely upon for hosting your WDL's Docker images
- Keep in mind that on Terra, every VM pulls the same Docker image every time. It is not cached! You can see solid proof of this in how SRANWRP, a Docker image that to my knowledge only I use in my pipelines, has over 500K pulls.
- Cromwell
If your workflow is imported from Dockstore, launching a workflow additionally queries:
- GitHub
- Dockstore
In my experience, the most common failure is with GCP "fumbling" the VM, followed by Docker Hub rejecting Cromwell's attempts to pull the VM required for the workflow's task section.
The clustering script has an option to call the Microreact API to generate Microreact projects, one per cluster. If Microreact is down, then clusters will of course fail to upload.
myco_sra is somewhat of a unique case due to calling NCBI SRA in order to download FQs, but having built in error handling that might allow it to continue even if NCBI SRA is temporarily down (please note this is theoretical as an NCBI outage has never happened while running myco_sra, at least not to my knowledge). This is because there are dozens of reasons why fastqs may not download from NCBI SRA, almost all of which are problems with the data itself, so myco_sra's download task treats essentially any failure as "we'll just skip that one." So, if you have a temporary outage in NCBI SRA that causes some but not all accessions to fail to download, myco_sra will blithely continue with what it has.
Transient errors shouldn't be confused with preempted VMs, which are clearly advertised as "this can be taken away from you at any time" by Google. When running on Terra, Cromwell attempts to detect preemption and retry automatically.