stage: preserve comments and don't add default fields removed by user

[efiop]

md5: adbb969c66c0fe9449b17a57afcd9531
outs:
- cache: true
  md5: c157a79031e1c40f85931829bc5fc552
  metric: false
  path: file
  persist: false
wdir: .

[efiop]

As @shcheklein noted, this should not remove fields that were already set before. This is also relevant to user comments in dvc files, that we currently throw out, which is not nice. Need to consider using something like https://pypi.org/project/ruamel.yaml/ first.

[AlJohri]

It would also be great to maintain the spacing on the ‘cmd’ field. I often change the spacing such a long cmd with several options is split up with new lines in between each argument. DVC currently undoes this spacing

[martinjms]

It would be nice if we could also have a custom filed for users to add some metadata.

For my use case in particular, i would like to add the repository and git commit in which the file was generated as well as the username of the person that generated the data. This is because i intend to use the datasets in more than one repository and i need a way to track the origin of the data

@martinjms , have you seen the issue related to reusing datasets, it might be interesting to you? #1487

[shcheklein]

https://pypi.org/project/ruamel.yaml/ - this parser or something like this can be used to preserve the info

[Suor]

I reviewed the issue and two several strategies to solve it:

1. Superficial. Use ruamel.yaml to parse file, store structure with comments and everything in Stage, then work as usual. On save dump classes to dicts and apply diffs to stored structure. Then serialize structure to file.

2. Fundamental. Use ruamel.yaml, resulting structure will serve as internal state of Stage, deps, outs classes. Some classes that don't have much or any polymorphic behaviour are eliminated and replaced either with their base or even structs + funcs. Since state = ruamel structure then we just dump it when needed.

@efiop what do you think? ;)

P.S. ruamel.yaml do preserve comments and string structure. Custom keys will also be saved in both cases.