command: Show dvc list output in pretty JSON format

[eggqq007]

1. update ls init.py to handle --show-pretty-json
2. introduce pygments into requirements/default.txt

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Closes treeverse/dvc.org#2838
Closes #6646

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

[skshetry]

We could just prettify the JSON by default by checking if we are in isatty mode and then prettify that output.

[eggqq007]

Thanks for the suggestion! The patch was updated

[skshetry]

Btw, why not just do dvc list --show-json . | python -m rich.json - or dvc list --show-json | jq?

[skshetry]

We depend on rich, so you could just do something like following:

from rich.json import JSON

obj = JSON.from_data(data)
ui.write(obj, styled=True)

[skshetry]

Let's move this to dvc/ui/__init__.py as ui.write_json, so that we can reuse it in other parts of dvc.

Also, support for indent= and prettify=True|False that keeps the old behaviour would be nice to have. 🙂

[skshetry]

Also, we need to bump rich requirement, as from_data is only available since 10.9.0.

[eggqq007]

Sure, that makes sense. patch updated.

[skshetry]

The PR looks good. @iterative/dvc, if you have any opinions if dvc should format json. Otherwise, we'll be formatting in other places as well.

[efiop]

We could hide this check under ui.write_json (if rich doesn't do that already for us) CC @skshetry

[efiop]

Though that would get a little confusing with prettify, as it is not clear if it should enforce it or be more like "prettify, but only if it is a tty". Still, it feels like tty check belongs to ui.

[efiop]

Btw, is there any significant performace hit for prettifying? If not, maybe we should indeed consider just showing pretty json by default?

[eggqq007]

Thanks for the comments!. It is hard to tell the performance gap between json.dumps and prettifying. I can do some evaluation on it later. Besides that, I tend to believe the prettifying is not the bottleneck since dvc list should fetch repo info from remote first which may spend more time.

[eggqq007]

I did two simple tests on it.

1. A python array has 100000 element, each element is a dict which contains 3 key-value. json.dump takes about 0.05s, rich.json.from_date takes about 2.6s

2. dvc list takes about 1.8s to fetch https://github.com/iterative/example-get-started while prettifying takes 0.03s

[skshetry]

I don't think performance should be a concern. We could choose a certain threshold like 1000 or so entries where we fallback to json.dumps(indent=4) instead of prettifying it. No one should expect us to prettify the 100000 many entries.

[efiop]

regretfully dvc list(and others) are not yet fully optimized themselves and having 100K entries in the resulting json is not unusual for our use case, so looks like prettifying is kinda expensive to do always. Plus we have other projects depending on this json output, so that migth be slowing them down too. But isatty check kinda solves that, because if someone will be using the json output - they will redirect it somewhere anyway, so we should be safe and efficient, while being pretty. That should make the threshold approach unnecessary.

[eggqq007]

Thanks for the comments! I move the isatty into ui init.py and the patch updated. :)

[skshetry]

We discussed this today in the team, and everyone is okay with this getting merged. For consistency, we'd like this to be used in all of the other commands that has --json flags (though in a separate PR).

[skshetry]

@eggqq007, would you like to create a PR (or, issue) for #6647 (review)? Thanks for this PR.

[eggqq007]

@eggqq007, would you like to create a PR (or, issue) for #6647 (review)? Thanks for this PR.

Sure, new issue in #6712