Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create new BTQ integrator option to force a full copy of input files #1281

Open
prioux opened this issue Nov 3, 2022 · 0 comments · May be fixed by #1299
Open

Create new BTQ integrator option to force a full copy of input files #1281

prioux opened this issue Nov 3, 2022 · 0 comments · May be fixed by #1299
Assignees
Labels
Admin Features or bugs related to administrative features Boutiques Boutiques integration Enhancement Priority: Normal

Comments

@prioux
Copy link
Member

prioux commented Nov 3, 2022

By default, CBRAIN tasks create symbolic links to the CBRAIN DP cache for all the input files that the task need; that way inputs can be shared by multiple task without too much data duplication.

The symbolic links are created by the make_available() utility method in class ClusterTask.

Some stupid tasks though MUTATE their own input data, so creating a symlink doesn't work anymore, the input data of one task would perturb the input data of another one directly in the cache area.

I'd like a new option for make_available() where instead of creating a symlink, the method would make a full copy of the input data directly in the task workdir.

I'd also like this option to be invoked when a Boutiques descriptor for the new integrator contains a statement about making a full copy. This shouldn't be implemented as a module, but as an option in the custom section of the descriptor.

@prioux prioux added Enhancement Priority: Normal Boutiques Boutiques integration Admin Features or bugs related to administrative features labels Nov 3, 2022
@MontrealSergiy MontrealSergiy self-assigned this Jan 19, 2023
@MontrealSergiy MontrealSergiy linked a pull request Jan 27, 2023 that will close this issue
@MontrealSergiy MontrealSergiy linked a pull request Jan 27, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Admin Features or bugs related to administrative features Boutiques Boutiques integration Enhancement Priority: Normal
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants