unix relative path issue

[rahulguptajss]

dvc repro failing due to some relative path issue in unix system. Same work fine in windows. Below is the error. File is placed at the correct location and related python works fine if ran independently.

[rahulguptajss]

there seems to be some serious issues in path creation for svc in unix system. It is not creating proper folders and looking for wrong folders as well.

for example: (a\b) is being treated as a file instead of folders.

[efiop]

Ah, this happens when dvc files were created in windows, but then ran in *nix. I can confirm this bug.
Working on the solution right now...

For now, you can work around it by manually converting paths in your dvc files to *nix format, it should do the trick.

[rahulguptajss]

Thanks. Do you mean to say all files extension with .dvc and files inside .dvc folder? I tried it but no luck.

Btw the file, it is unable to find doesn't fall under dvc cache. It is a static file which dvc is tracking only. So in any case, I dont think this workaround can work. dvc repro needs to look into correct file location to find it.

[efiop]

No, only your *.dvc files, no need to go into .dvc directory. I don't think that dos2unix converts paths inside text files, it is made to convert line breaks etc. E.g. you should convert train.p.dvc from:

cmd: python source/scripts/split_train_test.py
deps:                                         
- md5: 5fb9105c342df2de330913ae8e3767a7       
  path: data\CombinedSummary.csv              
- md5: 688eeb0836d7034cf1cfd53c1333ff5f       
  path: source\scripts\split_train_test.py    
outs:                                         
- cache: true                                 
  md5: 184b9aee2f5df48f1ed2cf22d3c33d07       
  path: data\train.p                          

to

cmd: python source/scripts/split_train_test.py
deps:                                         
- md5: 5fb9105c342df2de330913ae8e3767a7       
  path: data/CombinedSummary.csv              
- md5: 688eeb0836d7034cf1cfd53c1333ff5f       
  path: source/scripts/split_train_test.py    
outs:                                         
- cache: true                                 
  md5: 184b9aee2f5df48f1ed2cf22d3c33d07       
  path: data/train.p                          

[efiop]

Sorry, closed by accident.

[rahulguptajss]

Thanks. Can you please provide any ETA on this?

[efiop]

@rahulguptajss Waiting for 0.9.5 to be built right now. Should be ready in half an hour or so. I'll ping you when it is ready.

[efiop]

@rahulguptajss 0.9.5 is ready. Please update dvc.

Btw, with 0.9.5 you won't have to convert those windows paths manually, dvc will do it by itself.

[rahulguptajss]

That's great. Appreciate your lightning fast fixes.

[rahulguptajss]

Since this fix in unix, dvc repro works every time even if there is no change. I am building it in jenkins without any code change. Added jenkins logs in -v mode. There still seems some issue with building in windows and testing in jenkins which is hosted on unix.

dvc pull
dvc checkout
dvc repro
dvc push
python source/scripts/modelValuation.py

dvc pipeline:

dvc add data/CombinedSummary.csv
dvc add data/TestSet.csv

Pipeline:
1:
dvc run -d data/CombinedSummary.csv -d source/scripts/split_train_test.py -o data/train.p python source/scripts/split_train_test.py

2:
dvc run -d data/train.p -d source/scripts/featurization.py -o data/feature.p python source/scripts/featurization.py

3:
dvc run -d data/feature.p -d source/scripts/trainModel.py -o data/vectorizer.pkl -o data/RandomForest_classifier.pkl python source/scripts/trainModel.py

4:
dvc run -d data/TestSet.csv -d data/vectorizer.pkl -d data/RandomForest_classifier.pkl -d source/scripts/modelValuation.py python source/scripts/modelValuation.py

jenkinslog.txt