[BUG]: relative path in `with` block in YAML is interpreted relative from YAML, but copy operation to junifer jobs directory interprets relative to working directory. #224

synchon · 2023-11-07T15:37:03Z

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

I have the following directory structure:

❱ tree                                                                                                                                                   1 !
.
├── 01_queue.sh
├── junifer_extensions
│   ├── juni-farm
│   │   ├── juni_farm
│   │   │   └── datagrabber
│   │   │       ├── hcp_aging.py
│   │   │       ├── hcp_early_psychosis.py
│   │   │       ├── hcp_ya_confounds_cat.py
│   │   │       └── __pycache__
│   │   │           └── hcp_ya_confounds_cat.cpython-310.pyc
│   │   ├── LICENSE
│   │   └── README.md
│   ├── markers
│   └── parcellations
│       ├── data
│       │   ├── README.md
│       │   └── shen_1mm_268_parcellation.nii.gz
│       ├── __pycache__
│       │   └── register_parcellations.cpython-310.pyc
│       └── register_parcellations.py
├── junifer_jobs
│   └── PREPROCESS_HCP1200
└── yamls
    └── hcp_ya.yaml

The top level directory here is my working directory. The code in 01_queue.sh is:

#!/usr/bin/zsh
# 
# Run this code to create the junifer_jobs directory
junifer queue yamls/${1}.yaml

I run 01_queue.sh hcp_ya which performs junifer queue with the hcp_ya.yaml

The content of that yaml file is:

❱ cat yamls/hcp_ya.yaml                                                                                                                                  1 !
workdir: /tmp

with:
  - ../junifer_extensions/parcellations/register_parcellations.py
  - ../junifer_extensions/juni-farm/juni_farm/datagrabber/hcp_ya_confounds_cat.py


datagrabber:
    kind: MultipleHCP
    tasks:
      - REST1
      - REST2

preprocess:
    kind: fMRIPrepConfoundRemover
    detrend: true
    standardize: true
    strategy:
        wm_csf: full
        global_signal: full
    low_pass: 0.08
    high_pass: 0.01
    masks:
      - compute_epi_mask
      - compute_brain_mask
      - threshold: 0

storage:
  kind: HDF5FeatureStorage
  uri: ../../../data/junifer_storage/hcp_ya/hcp_ya.hdf5


markers:
   
  - name: parc-Finn2015Shen268_marker-empiricalFC
    kind: FunctionalConnectivityParcels
    parcellation: Finn2015Shen268
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit
  
  - name: parccortical-Schaefer100x17FSLMNI_marker-empiricalFC
    kind: FunctionalConnectivityParcels
    parcellation: Schaefer100x17
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit
        
  - name: parccortical-Schaefer200x17FSLMNI_marker-empiricalFC
    kind: FunctionalConnectivityParcels
    parcellation: Schaefer200x17
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit
  
  - name: parccortical-Schaefer400x17FSLMNI_marker-empiricalFC
    kind: FunctionalConnectivityParcels
    parcellation: Schaefer400x17
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit

  - name: parccortical-Schaefer100x7FSLMNI_marker-empiricalFC
    kind: FunctionalConnectivityParcels
    parcellation: Schaefer100x7
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit
        
  - name: parccortical-Schaefer200x7FSLMNI_marker-empiricalFC
    kind: FunctionalConnectivityParcels
    parcellation: Schaefer200x7
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit
  
  - name: parccortical-Schaefer400x7FSLMNI_marker-empiricalFC
    kind: FunctionalConnectivityParcels
    parcellation: Schaefer400x7
    cor_method: correlation
    cor_method_params:
      empirical: true
    masks:
      - inherit
  

queue:
  jobname: PREPROCESS_HCP1200
  kind: HTCondor
  env:
    kind: conda
    name: network-stacking-env
  mem: 8G
  verbose: 8

Of course, looking at the with block in the yaml, the file paths are relative from the YAML file:


with:
  - ../junifer_extensions/parcellations/register_parcellations.py
  - ../junifer_extensions/juni-farm/juni_farm/datagrabber/hcp_ya_confounds_cat.py

However when I run this I get:

❱ ./01_queue.sh hcp_ya                  
2023-05-10 08:45:29,730 - JUNIFER - INFO - ===== Lib Versions =====
2023-05-10 08:45:29,730 - JUNIFER - INFO - numpy: 1.23.5
2023-05-10 08:45:29,731 - JUNIFER - INFO - scipy: 1.10.1
2023-05-10 08:45:29,731 - JUNIFER - INFO - pandas: 1.5.3
2023-05-10 08:45:29,731 - JUNIFER - INFO - nilearn: 0.10.0
2023-05-10 08:45:29,731 - JUNIFER - INFO - nibabel: 4.0.2
2023-05-10 08:45:29,731 - JUNIFER - INFO - junifer: 0.0.3.dev71
2023-05-10 08:45:29,731 - JUNIFER - INFO - ========================
2023-05-10 08:45:29,731 - JUNIFER - INFO - Parsing yaml file: /data/project/identification_prediction/projects/network-stacking/code/preprocessing/yamls/hcp_ya.yaml
================================================================================
Registering: Finn2015Shen268
================================================================================
2023-05-10 08:45:29,761 - JUNIFER - INFO - Registering HCPCATConfounds in datagrabber
2023-05-10 08:45:29,761 - JUNIFER - INFO - Registering MultipleHCP in datagrabber
2023-05-10 08:45:29,761 - JUNIFER - INFO - Creating job in /data/project/identification_prediction/projects/network-stacking/code/preprocessing/junifer_jobs/PREPROCESS_HCP1200
Traceback (most recent call last):
  File "/home/lsasse/miniconda3/envs/network-stacking-env/bin/junifer", line 8, in <module>
    sys.exit(cli())
  File "/home/lsasse/miniconda3/envs/network-stacking-env/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/lsasse/miniconda3/envs/network-stacking-env/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/lsasse/miniconda3/envs/network-stacking-env/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/lsasse/miniconda3/envs/network-stacking-env/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/lsasse/miniconda3/envs/network-stacking-env/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/data/project/identification_prediction/projects/network-stacking/envs/junifer/junifer/api/cli.py", line 250, in queue
    api_queue(
  File "/data/project/identification_prediction/projects/network-stacking/envs/junifer/junifer/api/functions.py", line 264, in queue
    shutil.copy(item, jobdir)
  File "/home/lsasse/miniconda3/envs/network-stacking-env/lib/python3.10/shutil.py", line 417, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/lsasse/miniconda3/envs/network-stacking-env/lib/python3.10/shutil.py", line 254, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '../junifer_extensions/parcellations/register_parcellations.py'

Thinking, that maybe files have to be relative from working directory, I then tried to change this to:

with:
  - junifer_extensions/parcellations/register_parcellations.py
  - junifer_extensions/juni-farm/juni_farm/datagrabber/hcp_ya_confounds_cat.py

But using that I get:

❱ ./01_queue.sh hcp_ya                                                                                                                                   1 !
2023-05-10 08:47:09,779 - JUNIFER - INFO - ===== Lib Versions =====
2023-05-10 08:47:09,779 - JUNIFER - INFO - numpy: 1.23.5
2023-05-10 08:47:09,779 - JUNIFER - INFO - scipy: 1.10.1
2023-05-10 08:47:09,779 - JUNIFER - INFO - pandas: 1.5.3
2023-05-10 08:47:09,779 - JUNIFER - INFO - nilearn: 0.10.0
2023-05-10 08:47:09,779 - JUNIFER - INFO - nibabel: 4.0.2
2023-05-10 08:47:09,779 - JUNIFER - INFO - junifer: 0.0.3.dev71
2023-05-10 08:47:09,779 - JUNIFER - INFO - ========================
2023-05-10 08:47:09,779 - JUNIFER - INFO - Parsing yaml file: /data/project/identification_prediction/projects/network-stacking/code/preprocessing/yamls/hcp_ya.yaml
2023-05-10 08:47:09,808 - JUNIFER - ERROR - File in 'with' section does not exist: yamls/junifer_extensions/parcellations/register_parcellations.py
Traceback (most recent call last):
  File "/home/lsasse/miniconda3/envs/network-stacking-env/bin/junifer", line 8, in <module>
    sys.exit(cli())
  File "/home/lsasse/miniconda3/envs/network-stacking-env/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/lsasse/miniconda3/envs/network-stacking-env/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/lsasse/miniconda3/envs/network-stacking-env/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/lsasse/miniconda3/envs/network-stacking-env/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/lsasse/miniconda3/envs/network-stacking-env/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/data/project/identification_prediction/projects/network-stacking/envs/junifer/junifer/api/cli.py", line 244, in queue
    config = parse_yaml(filepath)  # type: ignore
  File "/data/project/identification_prediction/projects/network-stacking/envs/junifer/junifer/api/parser.py", line 59, in parse_yaml
    raise_error(
  File "/data/project/identification_prediction/projects/network-stacking/envs/junifer/junifer/utils/logging.py", line 289, in raise_error
    raise klass(msg)
ValueError: File in 'with' section does not exist: yamls/junifer_extensions/parcellations/register_parcellations.py

Expected Behavior

I think it should just work with relative paths, relative from the YAML directory, even if the working directory is another.

Just to be clear I fixed the issue by adjusting the code in 01_queue.sh to this:

#!/usr/bin/zsh
# 
# Run this code to create the junifer_jobs directory
cd yamls
junifer queue ${1}.yaml
cd ..

With that it works, but I think that is somewhat beside the point. In my understanding the previous version should also work.
Also, of course, this puts the junifer jobs directory under the yamls directory which is not what i want. I could also put absolute paths (and will for now), but I would prefer relative paths.

Steps To Reproduce

I think I gave everything needed above. Junifer wtf is below.

Environment

❱ junifer wtf
junifer:
  version: 0.0.3.dev71
python:
  version: 3.10.8
  implementation: CPython
dependencies:
  click: 8.1.3
  numpy: 1.23.5
  datalad: 0.18.3
  pandas: 1.5.3
  nibabel: 4.0.2
  nilearn: 0.10.0
  sqlalchemy: 1.4.47
  ruamel.yaml: 0.17.26
system:
  platform: Linux-4.19.0-21-amd64-x86_64-with-glibc2.28
environment:
  LC_CTYPE: en_US.UTF-8
  PATH: 
    /home/lsasse/miniconda3/envs/network-stacking-env/bin:/home/lsasse/miniconda3/condabin:/home/lsasse/bin:/home/lsasse/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games



### Relevant log output

_No response_

### Anything else?

_No response_

fraimondo · 2023-11-07T11:04:58Z

TODO: Fix the with value of the config dict with the absolute path here:

junifer/junifer/api/parser.py

Line 67 in f203cd8

spec.loader.exec_module(module) # type: ignore

synchon · 2023-11-07T15:37:59Z

@LeSasse Could you check if this branch works with your project setup as you described?

codecov · 2023-11-07T15:38:15Z

Codecov Report

Merging #224 (62567ad) into main (50d0f97) will increase coverage by 0.00%.
The diff coverage is 100.00%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #224   +/-   ##
=======================================
  Coverage   89.80%   89.81%           
=======================================
  Files          99       99           
  Lines        4367     4371    +4     
  Branches      842      842           
=======================================
+ Hits         3922     3926    +4     
  Misses        305      305           
  Partials      140      140

Flag	Coverage Δ
docs	`100.00% <ø> (ø)`
junifer	`89.81% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
junifer/api/parser.py	`95.34% <100.00%> (+0.47%)`	⬆️

github-actions · 2023-11-07T15:43:01Z

PR Preview Action v1.4.4
Preview removed because the pull request was closed.
2023-11-17 11:48 UTC

LeSasse · 2023-11-07T17:45:20Z

@LeSasse Could you check if this branch works with your project setup as you described?

I can likely do it on friday or early next week.

synchon · 2023-11-08T10:33:45Z

@LeSasse Could you check if this branch works with your project setup as you described?

I can likely do it on friday or early next week.

Sure, take your time.

LeSasse · 2023-11-17T08:05:46Z

@synchon this works in my project folder now.

synchon · 2023-11-17T09:32:02Z

@synchon this works in my project folder now.

Thanks for verifying :)

…arse_yaml()

LeSasse added bug Something isn't working triage New issues waiting to be reviewed labels May 10, 2023

synchon modified the milestones: 0.0.5 (alpha 4), 0.0.4 (alpha 3) Nov 7, 2023

synchon removed the triage New issues waiting to be reviewed label Nov 7, 2023

synchon added 2 commits November 17, 2023 10:52

fix: replace relative paths for custom module with absolute ones in p…

dee1162

…arse_yaml()

chore: add changelog 224.bugfix

f4f45d9

synchon force-pushed the fix/with-section-file-copying branch from 33bd7c4 to f4f45d9 Compare November 17, 2023 09:56

synchon requested a review from fraimondo November 17, 2023 10:01

fraimondo approved these changes Nov 17, 2023

View reviewed changes

fix: update parse_yaml() test

62567ad

synchon merged commit df4cee7 into main Nov 17, 2023
20 checks passed

synchon deleted the fix/with-section-file-copying branch November 17, 2023 11:45

fraimondo mentioned this pull request Feb 28, 2024

[ENH]: Allow for using venv in queue kind #249

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG]: relative path in `with` block in YAML is interpreted relative from YAML, but copy operation to junifer jobs directory interprets relative to working directory. #224

[BUG]: relative path in `with` block in YAML is interpreted relative from YAML, but copy operation to junifer jobs directory interprets relative to working directory. #224

synchon commented Nov 7, 2023 •

edited by LeSasse

Loading

fraimondo commented Nov 7, 2023

synchon commented Nov 7, 2023

codecov bot commented Nov 7, 2023 •

edited

Loading

github-actions bot commented Nov 7, 2023 •

edited

Loading

LeSasse commented Nov 7, 2023

synchon commented Nov 8, 2023

LeSasse commented Nov 17, 2023

synchon commented Nov 17, 2023

[BUG]: relative path in with block in YAML is interpreted relative from YAML, but copy operation to junifer jobs directory interprets relative to working directory. #224

[BUG]: relative path in with block in YAML is interpreted relative from YAML, but copy operation to junifer jobs directory interprets relative to working directory. #224

Conversation

synchon commented Nov 7, 2023 • edited by LeSasse Loading

Is there an existing issue for this?

Current Behavior

Expected Behavior

Steps To Reproduce

Environment

fraimondo commented Nov 7, 2023

synchon commented Nov 7, 2023

codecov bot commented Nov 7, 2023 • edited Loading

Codecov Report

github-actions bot commented Nov 7, 2023 • edited Loading

LeSasse commented Nov 7, 2023

synchon commented Nov 8, 2023

LeSasse commented Nov 17, 2023

synchon commented Nov 17, 2023

[BUG]: relative path in `with` block in YAML is interpreted relative from YAML, but copy operation to junifer jobs directory interprets relative to working directory. #224

[BUG]: relative path in `with` block in YAML is interpreted relative from YAML, but copy operation to junifer jobs directory interprets relative to working directory. #224

synchon commented Nov 7, 2023 •

edited by LeSasse

Loading

codecov bot commented Nov 7, 2023 •

edited

Loading

github-actions bot commented Nov 7, 2023 •

edited

Loading