unused folder systematically created

[RamiALBASHA]

A folder called "output" is systematically created, even if other names are defined by the user for the outputs folder.
This behavior is due to the hard-coded value of "output/" passed to the arg output_directory when calling output_manager.run_startup_sequence in TaskManager.task (here).
There is also at least one other hard-coded path in TaskManager: INPUT_DATA_CSV_WORKING_FOLDER = Path("output/saved_pool_working_folder/") (here), which is likely to result in some problems (have not checked tough).

We can avoid creating this folder by setting the value of output_directory to None when run_startup_sequence is called from task (the outputs folder is created elsewhere in the code) (here):

output_directory=None

Then, by adjusting run_startup_sequence accordingly (here):

if output_directory is not None:
    self.create_directory(output_directory)
    if clear_output_directory:
        self.clear_output_dir(variables_file_path, output_directory)

    if chunkification:
        self.setup_pool_overflow_control(
            output_directory, max_memory_usage_percent, max_memory_usage, save_chunk_threshold_call_count
        )

If this is of interested to the RuFaS team, I can open a PR for these modifications.

[ew3361zh]

Hi Rami, thank you for bringing this up. I think there are some good ideas in your post to discuss further. You're correct though, we shouldn't be hardcoding the output dir path in those spots you listed (or anywhere really). The user should be able to override that location and I think that could be addressed by passing the gnu-arg dir through to those functions where the output folder and its sub folders are created.

As you've probably seen though RuFaS does need that output dir for a number of other purposes - output filters, all the various logs files, outputs from post-processing - so not having an output dir at all doesn't seem like a good option unless you're thinking of handling those things in another way (which might warrant a separate issue).

I think removing the hardcoded output/ and ensuring the user-specified output dir is available for all the necessary uses is worth a PR and I'd be happy to work with you on it if you'd like to get the process started by creating an issue out of this discussion (if that option is not available to you, please let me know). Once we have an issue for it, you can go ahead and create a PR to start resolve it.

And please feel free to tag me @ew3361zh or you can use the new @RuminantFarmSystems/dev-team tag which will ping the whole dev-team when you have the PR ready.

[RamiALBASHA]

Thanks for your reply Niko.
Indeed, the model needs an output folder that includes the filters, otherwise it would not run. However, this is already the user's responsibility to create an output folder somewhere and, to my understanding, running self.create_directory(output_directory) is useless inside the task method.
I have already tested the suggested modifications above and the model runs well (adds all the output files where they should be under the user-defined outputs folder). I'll open an issue and create a PR and let you know once finished.
Best