Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Script runners small enhancements #3454

Closed
brian-mulier-p opened this issue Apr 2, 2024 · 3 comments
Closed

Script runners small enhancements #3454

brian-mulier-p opened this issue Apr 2, 2024 · 3 comments
Labels
enhancement New feature or request
Milestone

Comments

@brian-mulier-p
Copy link
Member

Issue description

See comments

@brian-mulier-p
Copy link
Member Author

PEBBLE:
{{workingDir}}
{{outputDir}}
{{bucketPath}}

ENV:
WORKING_DIR
OUTPUT_DIR
BUCKET_PATH

Rename (gcs/s3/azure)Bucket to bucket

Move uploadInputFiles to CommandsWrapper (with additionalVars precomputed before scriptRunner.run())

Open questions:

  • outputDir -> useless ? Since on cloud providers it is set to the same value as workingDir
  • in DOCKER & PROCESS runners everything put in {{outputDir}} are put in the outputFiles. However such operation is costly on cloud providers so we don't want to do this behaviour

To add in doc' -> limitation due to glob patterns hard to handle in remote script runners (we can't list easily files from remote buckets). Currently, remote runners only support hardcoded relative paths to file. We could extend it to prefix but not much further

@loicmathieu
Copy link
Member

For cloud runner, outputDir should be created inside the bucket and all files in this path (via a single prefix query) should be downloaded as outputFiles.

@anna-geller anna-geller added the enhancement New feature or request label Apr 3, 2024
@anna-geller anna-geller added this to the v0.17.0 milestone Apr 3, 2024
@loicmathieu loicmathieu modified the milestones: v0.17.0, v0.16.0 Apr 8, 2024
@loicmathieu
Copy link
Member

These enhancements has been done, I'll close this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants