hard-coded properties path is resulting in errors for custom paths

[RamiALBASHA]

Hi,

Recent developments in the InputManager introduced the constant PRROPERTIES_FILE_PATHS with hard-coded paths for "default", "tasks_properties" and "commodity_properties". This modification results in the model to raise a FileNotFound error when invoking _load_properties() with custom paths to these data files. The old way of doing it, by setting the properties path from the user's input (properties_path = Path(self.__metadata["files"]["properties"]["path"])) did not result in such an error.

The introduction of PRROPERTIES_FILE_PATHS is a bit confusing since it does not agree with the development policy of allowing free-location of data files. In addition, the typo in its name suggests that this variable was not fully reviewed before merging.
Would you consider reading user's paths instead of hard-coding them?

[allisterakun]

Hi @RamiALBASHA,

Thanks for raising this. The properties files (default, tasks_properties, commodity_properties) are not considered user input files — they are part of the model itself and are not intended to be modified or relocated by the user. That's the reason the paths are hard-coded via PROPERTIES_FILE_PATHS. This was a deliberate design decision reached after thorough discussion with the entire dev team.

Regarding the FileNotFoundError: because these files are part of the codebase rather than user-supplied input, we intentionally used relative paths. This means if you move the properties files out of their expected location within the repository, the paths will break — which is by design, since those files should never be relocated.

Also noted on the typo — we'll fix PRROPERTIES_FILE_PATHS → PROPERTIES_FILE_PATHS.

[RamiALBASHA]

Hi @allisterakun ,
Thanks for your response.
I get the point of considering these metadata as part of the model. In principal, this greatly facilitates the input preparation for users.

However:

• The tasks_properties file referes to user-specific data, such as save_animals_directory, filters_directory, csv_output_directory etc. In principal, theses paths are user inputs and the user should be allowed to chose a path of his/her own. I would suggest at least to allow the user to provide the path to tasks_properties as an option, otherwise to read the default path.

• As a third-party user, I run RuFaS as a dependency in a pipeline. The metadata files are not shipped with the RuFaS model when installed. If these files were to be considered as internal, I would also suggest to move them under the RUFAS dir so that they can be read at runtime. I do understand that RuFaS is not initially meant to be used as a dependency, but with small modifications of this type (with some other tricks), the model becomes much easily installed and this will broaden its users community.

[allisterakun]

Hi @RamiALBASHA,

Just to clarify the intent behind this design:

The properties files (default, tasks_properties, commodity_properties) are not user configuration files. They contain metadata about the model’s expected input structure, such as:

• what variables are expected in the input files
• the expected type and restrictions for each variable
• default values to use when a variable is not explicitly provided
• other validation/parsing rules used internally by the InputManager

Because these files define part of the model’s internal data contract, they are intentionally hard-coded and are not meant to be modified or relocated by users.

The user-specific paths and runtime configuration you mentioned (such as save_animals_directory, filters_directory, csv_output_directory, etc.) belong to the task configuration and should be specified in task.json, not in task_properties.

To better understand your use case: what exactly are you trying to set up or achieve by overriding these properties paths? If you can share your setup or intended workflow, we’d be happy to better understand it and help find the right approach.

[RamiALBASHA]

Hi @allisterakun , thanks for clarifying the intent befind these files. Indeed, I uselessly overloaded the tasks_properties file.

Regarding my usage of RuFaS, I have implemented a wrapper for this model in a Docker container so that to be able to run it as a lambda function in a AWS statemachine, along side other functions for pre- and pos-processing data. The code used in my case uses our fork as a dependency. Our fork synched manually with the branch dev which is then modified (cf "adaptation" in the graph below) so that to :

• set setuptools to seek all packages under "RUFAS"
• rm logic for checking python version and dependencies (both rely on pyproject.toml which do not exist in third-party packages)
• verify rufas's version using the native importlib.metadata.version() instead of get_rufas_version() which also relies on pyproject.toml
  and
• bugfix the case when convert_variable_table_path is provided in input_task but is None (default)

Once the tag is produced in our forked repo, I should then copy the metadata properties into our wrapper's package and override their paths, otherwise the model raises FileNotFound error.

flowchart LR
    subgraph rufas["RuFaS"]
        dev_origin["dev"]
    end

    subgraph rufas_msf["RuFaS-MyForageSystem"]
        dev_fork["dev"] --> dev_msf["dev-msf"] 
        dev_msf --> main_fork["main"]
        main_fork --> tag_fork["tag"] 
        dev_msf -->|adaptation| dev_msf 
        
    end

    dev_origin -->|fork| dev_fork
    
    subgraph rufas_wrapper["RuFaS wrapper"]
        wrapper["wrapper funcs"]
    end

    tag_fork --> wrapper

Loading

I hope that this clarifies our use case.

[morrowcj]

The metadata files are not shipped with the RuFaS model when installed. If these files were to be considered as internal, I would also suggest to move them under the RUFAS dir so that they can be read at runtime.

If this is still specifically referring to the "properties" metadata files, I think this makes sense. Default values that the package depends upon should probably be contained within/shipped with the package, especially if users can't provide them externally (which makes sense for properties).

[RamiALBASHA]

Hi @morrowcj , yes, my comment only concerned the metadata properties.

[allisterakun]

Hi @RamiALBASHA, thanks again for raising this. We’re currently discussing this internally with the dev team to make sure we come up with a well-rounded solution.

[RamiALBASHA]

Hi @allisterakun , thanks for the update. As long as this issue is discussed, please also consider the NASEM and NRC tables which I believe are internal due to the associated feed ids.

[ew3361zh]

Hey @RamiALBASHA, update from our end: we have moved main.py and the properties files into RUFAS/ on the dev branch. Would be curious to hear any additional feedback on how that works integrating with your setup and if you have additional thoughts about this process. Thank you again for this discussion - it's been helpful for our thinking and project development on our end!

[RamiALBASHA]

Hi @ew3361zh , thanks for your feedback and work on this topic.
This will highly reduces the maintenance cost on the MSF socitey end. I will forward the information to my (former) colleagues. I will also report this discussion to the current RUFAS maintainers at MSF so that they can share their eventual feedbacks.
However, as a former developer using RUFAS, I believe that shipping the properties file with the package is already a considerable improvement for 3d party users, so thanks again.