Relative path set up for params.custom_config_base is not working properly

[maxulysse]

Hi,
We noticed this issue with @aanil
The params custom_config_base is automatically set up with a relative path, and it make Sarek fails while looking for the igenomes.config file, even if said file is at the correct location.

$ nextflow run /path/to/sarek -profile uppmax

N E X T F L O W  ~  version 20.01.0
Launching `/path/to/sarek/main.nf` [prickly_bernard] - revision: e2a5b5f697
WARNING: Could not load nf-core/config/sarek profiles: ../configs//pipeline/sarek.config
No such file: Config file does not exist: /path/to/sarek/../configs/pipeline/conf/igenomes.config

But when setting it this params manually with the absolute /path/to/configs, the issue is no longer here:

(/path/to/configs being /path/to/sarek/../configs but without the ../ relative part of the path).

$ nextflow run /path/to/sarek -profile uppmax --custom_config_base /path/to/configs

N E X T F L O W  ~  version 20.01.0
Launching `/path/to/sarek/main.nf` [marvelous_lichterman] - revision: e2a5b5f697
...

[maxulysse]

And I'm guessing we noticed that only with Sarek, because of the institutional pipeline specific config file that adds an extra level of config files-ception.

[ewels]

After being initially a little suspicious, I have now replicated this locally:

$ nf-core download sarek --compress none

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'


INFO: Saving sarek
 Pipeline release: 2.5.2
 Pull singularity containers: No
 Output directory: nf-core-sarek-2.5.2

INFO: Downloading workflow files from GitHub

INFO: Downloading centralised configs from GitHub

$ nextflow run nf-core-sarek-2.5.2/workflow/ -profile uppmax --help

N E X T F L O W  ~  version 20.02.0-edge
Launching `nf-core-sarek-2.5.2/workflow/main.nf` [serene_hopper] - revision: e2a5b5f697
WARNING: Could not load nf-core/config/sarek profiles: ../configs//pipeline/sarek.config
No such file: Config file does not exist: /Users/philewels/testing/nf-core-sarek-2.5.2/workflow/../configs/pipeline/conf/igenomes.config

[ewels]

WARNING: Could not load nf-core/config/sarek profiles: ../configs//pipeline/sarek.config

This path does seem to exist, if indeed it is properly looking for files relative to the workflow directory and not the running directory.

No such file: Config file does not exist: /Users/philewels/testing/nf-core-sarek-2.5.2/workflow/../configs/pipeline/conf/igenomes.config

This is super odd - configs/pipeline/conf/igenomes.config doesn't exist anywhere and I am confused.

[ewels]

So the nf-core download tool seems to be correctly replacing this line to be custom_config_base = "../configs/" as expected.

What's weird is that the first config file import for nfcore_custom.config doesn't throw an error, but the second for pipeline/sarek.config does. Both files exist after the download.

The iGenomes error still makes no sense to me at all.

[maxulysse]

For the iGenomes error, I guess Nextflow is trying to load conf/igenomes.config which we're asking for, but messes up the path because of the error from the second config file import...

[ewels]

Note that running the nextflow run nf-core-sarek-2.5.2/workflow/ --help command without the -profile uppmax does work, so it must be something in the uppmax config profile...

But what that is, I have no idea.

Full nextflow log when failing with -profile uppmax:

May-08 13:37:24.711 [main] DEBUG nextflow.cli.Launcher - $> nextflow run nf-core-sarek-2.5.2/workflow/ -profile uppmax --help
May-08 13:37:24.995 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 20.02.0-edge
May-08 13:37:26.037 [main] INFO  nextflow.cli.CmdRun - Launching `nf-core-sarek-2.5.2/workflow/main.nf` [stupefied_mandelbrot] - revision: e2a5b5f697
May-08 13:37:26.065 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /Users/philewels/testing/nf-core-sarek-2.5.2/workflow/nextflow.config
May-08 13:37:26.068 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /Users/philewels/testing/nf-core-sarek-2.5.2/workflow/nextflow.config
May-08 13:37:26.077 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `uppmax`
May-08 13:37:26.705 [main] ERROR nextflow.cli.Launcher - Unable to parse config file: '/Users/philewels/testing/nf-core-sarek-2.5.2/workflow/nextflow.config'

  Config file does not exist: /Users/philewels/testing/nf-core-sarek-2.5.2/workflow/../configs/pipeline/conf/igenomes.config

java.nio.file.NoSuchFileException: Config file does not exist: /Users/philewels/testing/nf-core-sarek-2.5.2/workflow/../configs/pipeline/conf/igenomes.config

Confirms that the uppmax profile is being successfully applied..

[ewels]

I think it must be due to this line, but I haven't figure out exactly why yet:

https://github.com/nf-core/configs/blob/47de9b8f767a403e1c0593d6c227427455b2371a/pipeline/sarek.config#L13

This config file is in a different location to the main pipeline nextflow.config, so presumably a relative base could fail here but an absolute path work..

Note that the imports in the configs/nfcore_custom.config file are at the same directory depth as workflow/nextflow.config, so the relative path will still work (kind of by accident).

[ewels]

Is this issue a duplicate of https://github.com/nf-core/tools/issues/528 ?

[ewels]

Closing as I think that this is unfixable.