You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Overview
The TES Task Runner provides an extensibility mechanism to enable just-in-time SAS token resolution on the Batch Nodes. The goal of this story is to implement a new SAS resolution strategy when DRS URLs are provided.
AC:
Must implement the current DRS approach as SAS resolution strategy in the runner.
Should consider removing the dependency with DRS downloader and perform the resolution calls directly.
Must add options to the runner to enable DRS SAS resolution .
Must refactor the scheduler to use this strategy when applicable.
DoD:
Code merge into main, with unit tests that demonstrate the functionality.
Documented options for the runner and the API if applicable,
The text was updated successfully, but these errors were encountered:
ngambani
changed the title
Implement SAS resolution strategy for DRS
[#148 Implement TES Task Runner] Implement SAS resolution strategy for DRS
Oct 3, 2023
The DRS Localizer actually allows all of the DRS downloads to be passed via the command-line in just one call (rather than serially calling the localizer repeatedly for each call). That option is named --manifest-path and requires one argument, a CSV file with two columns (Url, Path) and one row per input file. The file should be without headers and should follow RFC-4180 formatting.
The DRS Localizer uses another option (--identity-client-id) for authorization (pass the ClientId of the azure managed identity) which we should be setting. This means that the container running the localizer needs to not be blocked.
The DRS Localizer uses the following environment variables we have not been using: DRS_RESOLVER_NUM_RETRIES, DRS_RESOLVER_WAIT_INITIAL_SECONDS, DRS_RESOLVER_WAIT_MAXIMUM_SECONDS, DRS_RESOLVER_WAIT_MULTIPLIER, DRS_RESOLVER_WAIT_RANDOMIZATION_FACTOR. We should consider whether we should provide any of these, or simply continue to use the defaults.
The following is provided because it is no longer showing up in the source code:
DRS files are identified via this line of code: task.Inputs.Where(f => f?.Url?.StartsWith("drs://", StringComparison.OrdinalIgnoreCase) == true)
The docker image is available in BatchScheduler as cromwellDrsLocalizerImageName, the resolver URL (found currently in Options.MarthaOptions.Url) is passed double-quoted via the DRS_RESOLVER_URL environment variable, the command-line option --access-token-strategy azure is always provided, and the container should run with the same volume mounts options as the executor. When not using the manifest file, the Url and Path should be passed as command-line arguments in that same order.
Overview
The TES Task Runner provides an extensibility mechanism to enable just-in-time SAS token resolution on the Batch Nodes. The goal of this story is to implement a new SAS resolution strategy when DRS URLs are provided.
AC:
DoD:
The text was updated successfully, but these errors were encountered: