WARNING:root:Settings file 'settings.py' does not exist, ignoring it

[albertz]

With our new i6 setups, a RETURNN config file often directly imports model definitions or other things from the i6_experiments repository. That can look like this:

...
import os
import sys

sys.path.insert(0, "/u/zeyer/setups/combined/2021-05-31/recipe")
sys.path.insert(1, "/u/zeyer/setups/combined/2021-05-31/tools/sisyphus")

...

from i6_experiments.users.zeyer.experiments.exp2023_04_25_rf.conformer_import_moh_att_2023_06_30 import (
    from_scratch_model_def as _model_def,
)
from i6_experiments.users.zeyer.experiments.exp2023_04_25_rf.train import (
    _returnn_v2_get_model as get_model,
)
from i6_experiments.users.zeyer.experiments.exp2023_04_25_rf.conformer_import_moh_att_2023_06_30 import (
    from_scratch_training as _train_def,
)
from i6_experiments.users.zeyer.experiments.exp2023_04_25_rf.train import (
    _returnn_v2_train_step as train_step,
)

As the i6_experiments code is mixed with Sisyphus-level logic and with model definitions which are imported here, this will thus end up with an import sisyphus inside of RETURNN. That's why you see both sys.path.insert there.

But sisyphus is just imported here but otherwise not really used. However, it still leads to this warning:

WARNING:root:Settings file 'settings.py' does not exist, ignoring it ([Errno 2] No such file or directory: 'settings.py').

I think we should not print this warning, unless the settings are actually used.
The way this is designed in Sisyphus makes this tricky, though.
However, a simple thing we can do is maybe checking sys.modules['__main__'].__file__ if that is actually some Sisyphus executable or not (like RETURNN) and in case it is not from Sisyphus, do not print the warning.
Or what do you think?

Or maybe you say this is really intended, even at just import sisyphus, wherever you do that?
One workaround on RETURNN side (or wherever else you might have an import sisyphus):
Add sth like this before the import:

os.environ["SIS_GLOBAL_SETTINGS_FILE"] = ""

So we could add this to the RETURNN config. But it's also somewhat ugly.

[JackTemaki]

I wonder how in your case Sisyphus imports end up in your RETURNN config. While there can be model code imported in the manager, I have never seen pipeline code being imported into the config for me... So no opinion from my side here.

[albertz]

I wonder how in your case Sisyphus imports end up in your RETURNN config. While there can be model code imported in the manager, I have never seen pipeline code being imported into the config for me... So no opinion from my side here.

Look at that conformer_import_moh_att_2023_06_30.py file. I have this in there:

from sisyphus import tk

Because in that file itself, I have mixed both the Sisyphus logic and the model + train + recog code.
You might argue that's a bad idea, and I'm myself not really sure about it, but anyway, that's what I have here.

[michelwi]

You might argue that's a bad idea,

That was indeed my first thought as well.

my second thought was that

sys.path.insert(0, "/u/zeyer/setups/combined/2021-05-31/recipe")

from i6_experiments import [...]

is a bad idea for anyone who wants to see what your experiment did in 3 years time after you have updated your recipes. I would only use the i6_experiments from a CloneGitRepositoryJob with a defined commit, so that it is reproducible.

But your recipe design is not the issue here. Going to the question:

I think we should not print this warning, unless the settings are actually used.

Here I agree with you. But are the settings really never used if sisyphus is imported but not the main executable?

[albertz]

Offtopic mumbling: I consider RETURNN, i6_core, RASR, i6_experiments (at least the parts I developed and/or use in my setups), Sisyphus as stable, so that it should never be a problem also in 3 years to run it (or if this does not work, I consider it a bug which should be fixed). I don't want to use CloneGitRepositoryJob for any of those. I find it kind of arbitrary that you use i6_core, Sisyphus as is (not part of the hash), also Linux, Apptainer, the node you run on, whatever other things not part of the hash, but the others (or whatever selection you have) are part of the hash. I also get the feeling we are really overthinking the whole hashing part. In the logs, ideally you should also see a date, and maybe the commit/version of all relevant software, so even if sth breaks, you can still reproduce it. Or even if you don't have that, you can still probably get to a working version back in time. This is anyway a case which is rarely if ever relevant.

Anyway, back to topic:

But are the settings really never used if sisyphus is imported but not the main executable?

In this case, I don't use anything at all from Sisyphus. All Sisyphus related calls are in separate functions which I don't call inside RETURNN.

[critias]

You might argue that's a bad idea,

I would consider it best practices to separate these two, to avoid unnecessary imports. Beside that you are right that it should be possible to include Sisyphus without getting a warning.

How about having a global sisyphus config file e.g. $HOME/.sisyphus which is always loaded first. This would allow to set stuff like the grid engine globally and also allows to define if there should be a warning if no local config is found.
Any opinions about that? I think @curufinwe didn't like that idea, but I not sure what his reasoning was.

[albertz]

What about my initial suggestion?

However, a simple thing we can do is maybe checking sys.modules['__main__'].__file__ if that is actually some Sisyphus executable or not (like RETURNN) and in case it is not from Sisyphus, do not print the warning.

[critias]

I'm a little reserved about that, if you import sisyphus to run a workflow inside a script it would also hide the warnings.
On the other hand I don't know if anyone beside me ever did that and I'm currently not using it. So losing these warnings would probably not hurt anyone 🤔