dvc: use dvcignore when adding data with `dvc add/run`

[efiop]

echo 'dir/foo' > .dvcignore
dvc add dir # should ignore foo and not take it into account when computing mtime and size

[pared]

Note about get_mtime_and_size:
Initial idea and implementation (avialable at https://github.com/pared/dvc/tree/1499_original) was to calculate dict_md5 of map(path : mtime) for each file in directory. That way we could detect new files and files modification, yet still "dvcignore" files.

[PeterFogh]

HI, This issue relates to my question regarding dvcignore from this Discord message.

Can it be used for ignoring .pyc files in a folder which a DVC file is dependent on?

I more details my use case is about having a DVC stage depending on a folder containing python code. When executing the stage the python code in the folder will produce the "pycache" folder with the ".pyc" files. However, when reproducing the DVC stage these ".pyc" files may change even if the python code did not change, this forces DVC stage to be reproduced when it is not needed.

Therefore, I think the dvcignore feature should exclude these ".pyc" files when computing the DVC stage dependency checksum, if the .dvcignore file defines the exclusion of the "pycache" folder or "*.pyc" files.

[pared]

Well then, I guess we should get to spreading dvc ignore to all usages of walk in project. We will need to revisit get_mtime_and_size for directories and come up with a way of calculating dir checksum that will not be updated by ignored files/dirs (mostly mtime will be a problem).

[nik123]

I've been sent here by one of the developers from this conversation on StackOverflow. In general I really need dvc status command to ignore .DS_Store files automatically created by MacOS.

However I'm not sure if my request is a part of this issue or that should be a separate feature request.

[efiop]

Hi @nik123 !

It is indeed part of this issue, so just hang on tight! 🙂

[efiop]

Related to #2010 , maybe worth switching from dulwich to pathspec along the way.