dvc run : possible bug with deeply nested dependencies

[tdeboissiere]

Please provide information about your setup

ubuntu 18.04, dvc==0.59.2, pip install with miniconda python 3.7

For deeply nested dependencies, it looks like dvc is not tracking them properly in the .dvc files

Following script reproduces the issue:

#!/bin/bash

set -x
set -e

rm -rf dvc_test
mkdir dvc_test && cd dvc_test
mkdir scripts
mkdir -p data/recommended/dataset1/dataset1_proc
echo bar > data/recommended/dataset1/v1.txt
git init
dvc init
echo -e "import sys\
\nwith open(sys.argv[1], 'w') as f: f.write(sys.argv[2])" > scripts/script.py

# Run works
dvc run -y \
    -w $PWD/data/recommended/dataset1/dataset1_proc\
    -f ./data/recommended/dataset1/dataset1_proc/v1.dvc\
    -d ../../../../scripts/script.py \
    -o v1 \
    "mkdir v1 && python ../../../../scripts/script.py v1/v1.txt proc_data"

# Inspecting v1.dvc shows that the script.py dependency is missing on ../
cat data/recommended/dataset1/dataset1_proc/v1.dvc

# Because of the, repro does not work
dvc repro data/recommended/dataset1/dataset1_proc/v1.dvc

Inspecting the v1.dvc file shows:

md5: 4ae72e168a0a6a2f1aaadfb5628640f7
cmd: mkdir v1 && python ../../../../scripts/script.py v1/v1.txt proc_data
deps:
- md5: 791b9c74b1d9308a3226b93a36689dad
  path: ../../../scripts/script.py
outs:
- md5: 188ed6cb603658d01ef7ba8fb7c434fe.dir
  path: v1
  cache: true
  metric: false
  persist: false
+ dvc repro data/recommended/dataset1/dataset1_proc/v1.dvc

Which indicates that the script.py dependency is indeed missing one ../

[efiop]

Hi @tdeboissiere ! Are you sure this is the same command? I see you've specified -w in dvc run ,but there is no wdir: field in the dvc file.

PS: sorry, jumped the gun, trying to reproduce right now.

[tdeboissiere]

@efiop : Sorry, this was run with version 0.57 (which still logged wdir in the .dvc file even if it's the default .).

If you run it with 0.59, wdir is absent indeed

[efiop]

@tdeboissiere Sorry, I just missed the -f option in your command and was surprised about wdir missing. But it makes sense now.

Ok, so running your reproduction script I get:

+ cat data/recommended/dataset1/dataset1_proc/v1.dvc                      
md5: 1f92bcfcf1ef2766da1ea3c373289dca                                     
cmd: mkdir v1 && python ../../../../scripts/script.py v1/v1.txt proc_data 
deps:                                                                     
- md5: 791b9c74b1d9308a3226b93a36689dad                                   
  path: ../../../../scripts/script.py                                     
outs:                                                                     
- md5: 188ed6cb603658d01ef7ba8fb7c434fe.dir                               
  path: v1                                                                
  cache: true                                                             
  metric: false                                                           
  persist: false                                                          
+ dvc repro data/recommended/dataset1/dataset1_proc/v1.dvc                
Stage 'data/recommended/dataset1/dataset1_proc/v1.dvc' didn't change.     
Data and pipelines are up to date.                                        

which has correct path in the dvcfile and dvc repro reports that there is nothing to reproduce, so it sees all the dependencies. So I'm not able to reproduce this issue.

[efiop]

@tdeboissiere Are you still able to reproduce it with your script? Am I missing something?

[tdeboissiere]

@efiop Yes, I still get (version 0.59.2)

ERROR: failed to reproduce 'data/recommended/dataset1/dataset1_proc/v1.dvc': missing dependency: data/scripts/script.py

This is what I get in the dvc_test/data/recommended/dataset1/dataset1_proc/v1.dvc file:

md5: 4ae72e168a0a6a2f1aaadfb5628640f7
cmd: mkdir v1 && python ../../../../scripts/script.py v1/v1.txt proc_data
deps:
- md5: 791b9c74b1d9308a3226b93a36689dad
  path: ../../../scripts/script.py
outs:
- md5: 188ed6cb603658d01ef7ba8fb7c434fe.dir
  path: v1
  cache: true
  metric: false
  persist: false

I am still missing one ../ in the path

[efiop]

@tdeboissiere That is really weird. Could you please try this one once more as is:

(3.7.0-dvc) ➜  dvc git:(master) ✗ cat ./test_2483.sh                         
#!/bin/bash                                                                  
                                                                             
set -x                                                                       
set -e                                                                       
                                                                             
pip uninstall -y dvc; pip install dvc==0.59.2                                
                                                                             
rm -rf dvc_test                                                              
mkdir dvc_test                                                               
cd dvc_test                                                                  
mkdir scripts                                                                
mkdir -p data/recommended/dataset1/dataset1_proc                             
echo bar > data/recommended/dataset1/v1.txt                                  
git init                                                                     
dvc init                                                                     
echo -e "import sys\                                                         
\nwith open(sys.argv[1], 'w') as f: f.write(sys.argv[2])" > scripts/script.py
                                                                             
# Run works                                                                  
dvc run -y \                                                                 
    -w $PWD/data/recommended/dataset1/dataset1_proc\                         
    -f ./data/recommended/dataset1/dataset1_proc/v1.dvc \                    
    -d ../../../../scripts/script.py \                                       
    -o v1 \                                                                  
    "mkdir v1 && python ../../../../scripts/script.py v1/v1.txt proc_data"   
                                                                             
# Inspecting v1.dvc shows that the script.py dependency is missing on ../    
cat data/recommended/dataset1/dataset1_proc/v1.dvc                           
                                                                             
# Because of the, repro does not work                                        
dvc repro data/recommended/dataset1/dataset1_proc/v1.dvc                     

[tdeboissiere]

@efiop : Same results, am still missing a ../

(I also tried it on multiple ubuntu machines)

Edit:

I ran it on a Mac, and this time it worked correctly...

[efiop]

@tdeboissiere That is extremely odd. I've noticed $PWD in your command, which evaluates to nothing on my ubuntu. Maybe you have it defined on yours? How about:

#!/bin/bash                                                                  
                                                                             
set -x                                                                       
set -e                                                                       
                                                                             
pip uninstall -y dvc; pip install dvc==0.59.2                                
                                                                             
rm -rf dvc_test                                                              
mkdir dvc_test                                                               
cd dvc_test                                                                  
mkdir scripts                                                                
mkdir -p data/recommended/dataset1/dataset1_proc                             
echo bar > data/recommended/dataset1/v1.txt                                  
git init                                                                     
dvc init                                                                     
echo -e "import sys\nwith open(sys.argv[1], 'w') as f: f.write(sys.argv[2])" > scripts/script.py
                                                                             
# Run works                                                                  
dvc run \
    -w ./data/recommended/dataset1/dataset1_proc \
    -f ./data/recommended/dataset1/dataset1_proc/v1.dvc \
    -d ../../../../scripts/script.py \
    -o v1 \
    "mkdir v1 && python ../../../../scripts/script.py v1/v1.txt proc_data" 
                                                                             
# Inspecting v1.dvc shows that the script.py dependency is missing on ../    
cat data/recommended/dataset1/dataset1_proc/v1.dvc                           
                                                                             
# Because of the, repro does not work                                        
dvc repro data/recommended/dataset1/dataset1_proc/v1.dvc  

NOTE: removed $PWD and -y.

[tdeboissiere]

@efiop Nope, still got the same error, on multiple machines

[tdeboissiere]

Can you point me to where in the codebase the .dvc file with the deps is filled and I'll have a look ?

[efiop]

@tdeboissiere Asked two more guys to try out, and they are getting the same results as me (one on linux and another one on mac). We are probably missing something here. Are you sure you are running that precise script? Which directory are you running it from? Does it have symlinks as parent dirs?

Can you point me to where in the codebase the .dvc file with the deps is filled and I'll have a look ?

Sure, this is the entry point https://github.com/iterative/dvc/blob/0.59.2/dvc/stage.py#L471 . Then the wdir and path for the dep gets resolved here https://github.com/iterative/dvc/blob/0.59.2/dvc/output/local.py#L20 and dumped here https://github.com/iterative/dvc/blob/0.59.2/dvc/output/local.py#L53 .

[efiop]

@tdeboissiere Another question, are you using default shell that comes with ubuntu? Or something else? bash? zsh? or maybe something as exotic as fish?

[tdeboissiere]

zsh could be an issue.

However, I changed to bash doing:

/bin/bash
# bash shell is activated
bash script_name.sh

but go the same results

I am running that precise script, from the same directory as that same script.
No symlink as parent dirs

[efiop]

@tdeboissiere Thanks for the info! zsh is ok. I'm using it too.

I am running that precise script, from the same directory as that same script.

What do you mean "from the same directory"? The script is location agnostic. Are you running it from your original project dir?

[shcheklein]

@tdeboissiere I've tried to run it from zsh, still the same result for me - Data and pipelines are up to date.. But on the other hand, we have an ongoing issue on Discord with zsh that I can't reproduce and it's clear that it's related to env management - https://discordapp.com/channels/485586884165107732/485596304961962003/620980290978054158