Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#148 Implement TES Task Runner] Optimize calls to WSM when executing tasks via the runner #440

Closed
Tracked by #148
giventocode opened this issue Oct 2, 2023 · 1 comment · Fixed by #452
Closed
Tracked by #148
Assignees

Comments

@giventocode
Copy link
Contributor

giventocode commented Oct 2, 2023

Problem:
Each input and output requires a WSM call to obtain the SAS token. However, the WSM API may fail to respond under stress conditions, such as when running large scatter-gather workflows.

Solution:
To reduce the frequency of WSM calls, by caching the SAS tokens while executing a node task.

AC:

  • Use container scoped JIT tokens in the runner to improve caching performance for both reading and writing operations.
  • Manage possible expiration scenarios, even though they are rare due to the JIT nature of the tokens.
  • Implementation in the URL transformation strategy for Terra.
@giventocode giventocode changed the title Optimize calls to WSM when executing task via the runner Optimize calls to WSM when executing tasks via the runner Oct 2, 2023
@ngambani
Copy link

ngambani commented Oct 3, 2023

@giventocode can we link this issue to the TES runner #148? Or should this be separate from the runner implementation?

@ngambani ngambani changed the title Optimize calls to WSM when executing tasks via the runner [#148 Implement TES Task Runner] Optimize calls to WSM when executing tasks via the runner Oct 4, 2023
@giventocode giventocode self-assigned this Oct 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants