Refactor list rendering in commands #12704

turbaszek · 2020-11-29T15:28:53Z

This commit unifies the mechanism of rendering output of tabular
data. This gives users a possibility to either display a tabular
representation of data or render it as valid json or yaml payload.

Closes: #12699

Instead of using tabulate to render commands output we use rich. Due to this change the --output argument will no longer accept formats of tabulate tables. Instead it accepts:

table - will render the output in predefined table
json - will render the output as a json
yaml - will render the output as yaml

By doing this we increased consistency and gave users possibility to manipulate the output programmatically (when using json or yaml).

Affected commands:

airflow dags list
airflow dags report
airflow dags list-runs
airflow dags list-jobs
airflow connections list
airflow connections get
airflow pools list
airflow pools get
airflow pools set
airflow pools delete
airflow pools export
airflow role list
airflow providers list
airflow providers get
airflow tasks states-for-dag-run
airflow users list
airflow variables list

Example:

Table:

root@e794bcc2d698:/opt/airflow# airflow tasks states-for-dag-run tasks_are_awesome 2020-11-13T00:00:00+00:00
dag_id            | execution_date            | task_id | state   | start_date                       | end_date
==================+===========================+=========+=========+==================================+=================================
tasks_are_awesome | 2020-11-13T00:00:00+00:00 | numbers | success | 2020-11-29T14:53:46.811030+00:00 | 2020-11-29T14:53:46.974545+00:00
tasks_are_awesome | 2020-11-13T00:00:00+00:00 | show__2 | success | 2020-11-29T14:53:56.926441+00:00 | 2020-11-29T14:53:57.118781+00:00
tasks_are_awesome | 2020-11-13T00:00:00+00:00 | show    | success | 2020-11-29T14:53:56.915802+00:00 | 2020-11-29T14:53:57.125230+00:00
tasks_are_awesome | 2020-11-13T00:00:00+00:00 | show__1 | success | 2020-11-29T14:53:56.922131+00:00 | 2020-11-29T14:53:57.129091+00:00
tasks_are_awesome | 2020-11-13T00:00:00+00:00 | show__3 | success | 2020-11-29T14:53:56.931243+00:00 | 2020-11-29T14:53:57.126306+00:00

JSON:

root@e794bcc2d698:/opt/airflow# airflow tasks states-for-dag-run tasks_are_awesome 2020-11-13T00:00:00+00:00 --outpu json
[{"dag_id": "tasks_are_awesome", "execution_date": "2020-11-13T00:00:00+00:00", "task_id": "numbers", "state": "success", "start_date": "2020-11-29T14:53:46.811030+00:00", "end_date": "2020-11-29T14:53:46.974545+00:00"}, {"dag_id": "tasks_are_awesome", "execution_date": "2020-11-13T00:00:00+00:00", "task_id": "show__2", "state": "success", "start_date": "2020-11-29T14:53:56.926441+00:00", "end_date": "2020-11-29T14:53:57.118781+00:00"}, {"dag_id": "tasks_are_awesome", "execution_date": "2020-11-13T00:00:00+00:00", "task_id": "show", "state": "success", "start_date": "2020-11-29T14:53:56.915802+00:00", "end_date": "2020-11-29T14:53:57.125230+00:00"}, {"dag_id": "tasks_are_awesome", "execution_date": "2020-11-13T00:00:00+00:00", "task_id": "show__1", "state": "success", "start_date": "2020-11-29T14:53:56.922131+00:00", "end_date": "2020-11-29T14:53:57.129091+00:00"}, {"dag_id": "tasks_are_awesome", "execution_date": "2020-11-13T00:00:00+00:00", "task_id": "show__3", "state": "success", "start_date": "2020-11-29T14:53:56.931243+00:00", "end_date": "2020-11-29T14:53:57.126306+00:00"}]

YAML:

root@e794bcc2d698:/opt/airflow# airflow tasks states-for-dag-run tasks_are_awesome 2020-11-13T00:00:00+00:00 --output yaml
- dag_id: tasks_are_awesome
  end_date: '2020-11-29T14:53:46.974545+00:00'
  execution_date: '2020-11-13T00:00:00+00:00'
  start_date: '2020-11-29T14:53:46.811030+00:00'
  state: success
  task_id: numbers
- dag_id: tasks_are_awesome
  end_date: '2020-11-29T14:53:57.118781+00:00'
  execution_date: '2020-11-13T00:00:00+00:00'
  start_date: '2020-11-29T14:53:56.926441+00:00'
  state: success
  task_id: show__2
- dag_id: tasks_are_awesome
  end_date: '2020-11-29T14:53:57.125230+00:00'
  execution_date: '2020-11-13T00:00:00+00:00'
  start_date: '2020-11-29T14:53:56.915802+00:00'
  state: success
  task_id: show
- dag_id: tasks_are_awesome
  end_date: '2020-11-29T14:53:57.129091+00:00'
  execution_date: '2020-11-13T00:00:00+00:00'
  start_date: '2020-11-29T14:53:56.922131+00:00'
  state: success
  task_id: show__1
- dag_id: tasks_are_awesome
  end_date: '2020-11-29T14:53:57.126306+00:00'
  execution_date: '2020-11-13T00:00:00+00:00'
  start_date: '2020-11-29T14:53:56.931243+00:00'
  state: success
  task_id: show__3

Using `jq`

root@e794bcc2d698:/opt/airflow# airflow tasks states-for-dag-run tasks_are_awesome 2020-11-13T00:00:00+00:00 --output json | jq ".[] | .task_id"
"numbers"
"show__2"
"show"
"show__1"
"show__3"

Using `yq`

root@e794bcc2d698:/opt/airflow# airflow tasks states-for-dag-run tasks_are_awesome 2020-11-13T00:00:00+00:00 --output yaml | yq ".[] | {sd: .start_date, ed: .end_date}"
{
  "sd": "2020-11-29T14:53:46.811030+00:00",
  "ed": "2020-11-29T14:53:46.974545+00:00"
}
{
  "sd": "2020-11-29T14:53:56.926441+00:00",
  "ed": "2020-11-29T14:53:57.118781+00:00"
}
{
  "sd": "2020-11-29T14:53:56.915802+00:00",
  "ed": "2020-11-29T14:53:57.125230+00:00"
}
{
  "sd": "2020-11-29T14:53:56.922131+00:00",
  "ed": "2020-11-29T14:53:57.129091+00:00"
}
{
  "sd": "2020-11-29T14:53:56.931243+00:00",
  "ed": "2020-11-29T14:53:57.126306+00:00"
}

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

turbaszek · 2020-11-29T15:42:47Z

The only thing that I'm wondering about is suppressing / removing log records when using json/yaml. Because otherwise the output is not valid. For example:

root@e794bcc2d698:/opt/airflow# airflow dags list --output yaml | yq "."
/opt/airflow/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1Volume`.
/opt/airflow/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1VolumeMount`.
yq: Error running jq: ParserError: expected '<document start>', but found '{'
  in "<stdin>", line 1, column 27.
[
  "2020-11-29T15:41:59",
  435
]

or

^[[Aroot@e794bcc2d698:/opt/airflow# airflow providers hooks --output yaml | yq ".[]"
/usr/local/lib/python3.6/site-packages/snowflake/connector/options.py:39 UserWarning: You have an incompatible version of 'pyarrow' installed, please install a version that adheres to: 'pyarrow<0.18.0,>=0.17.0; extra == "pandas"'
yq: Error running jq: ParserError: while parsing a block mapping
  in "<stdin>", line 1, column 1
expected <block end>, but found '{'
  in "<stdin>", line 1, column 27.

It can also appear inline with the output:

Config info
executor             | LocalExecutor
task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler
sql_alchemy_conn     | postgresql+psycopg2://postgres:airflow@postgres/airflow
dags_folder          | /files/dags
plugins_folder       | /root/airflow/plugins
base_log_folder      | /root/airflow/logs

[2020-11-29 15:46:57,661] {providers_manager.py:207} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.compute_ssh.ComputeEngineSSHHook' from 'apache-airflow-providers-google' package: No module named 'google.cloud.oslogin_v1'
/usr/local/lib/python3.6/site-packages/snowflake/connector/options.py:39 UserWarning: You have an incompatible version of 'pyarrow' installed, please install a version that adheres to: 'pyarrow<0.18.0,>=0.17.0; extra == "pandas"'
Providers info
apache-airflow-providers-amazon           | 1.0.0b2
apache-airflow-providers-apache-cassandra | 1.0.0b2
apache-airflow-providers-apache-druid     | 1.0.0b2

I've never seen this in any other application, so I lean toward suppressing warnings and logs.

XD-DENG · 2020-11-29T17:12:28Z

The only thing that I'm wondering about is suppressing / removing log records when using json/yaml. Because otherwise the output is not valid. For example:

... ...
I've never seen this in any other application, so I lean toward suppressing warnings and logs.

The same preference for me.

For table output I think we should suppress log as well. not only for json/yaml.

XD-DENG

nit

airflow/cli/simple_table.py

docs/usage-cli.rst

turbaszek · 2020-11-30T18:17:58Z

@potiuk @mik-laj @kaxil @ashb thoughts?

airflow/cli/simple_table.py

docs/usage-cli.rst

XD-DENG · 2020-11-30T18:50:27Z

UPDATING.md

airflow pools import and airflow providers hooks should be in this list as well?

Done in caf81ff

@turbaszek after your latest change (some commands don't print table anymore), maybe we need to further update this doc as well.

airflow/cli/commands/connection_command.py

airflow/cli/commands/task_command.py

turbaszek · 2020-11-30T20:07:27Z

airflow/cli/commands/pool_command.py

Should we just print "Pool deleted" instead of printing info about it?

turbaszek · 2020-11-30T20:08:18Z

airflow/cli/commands/pool_command.py

Same here, most of clis I know would show "Pool created" instead of showing info. We can consider a verbose flag.

turbaszek · 2020-11-30T20:09:17Z

airflow/cli/commands/pool_command.py

We are exporting pools to file, I think we shouldn't be printing them after that.

Personally I agree with the three points you raised (including this one).

XD-DENG · 2020-12-01T06:50:04Z

airflow/cli/commands/pool_command.py

Maybe add try-except similar to what you added in pool_delete?

I would like to address this in followup PR. Probably create new decorator that will handle known errors and show only message instead of lines of traceback to make it more user friendly.

github-actions · 2020-12-01T11:29:53Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2020-12-01T11:30:30Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

This commit unifies the mechanism of rendering output of tabular data. This gives users a possibility to eiter display a tabular representation of data or render it as valid json or yaml payload. Closes: apache#12699

XD-DENG

Maybe other folks will have further suggestions. From my point of view, all my questions/concerns have been addressed well.

Nice feature indeed. Thanks @turbaszek

github-actions · 2020-12-01T11:48:38Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

UPDATING.md

tests/cli/commands/test_task_command.py

tests/cli/commands/test_dag_command.py

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

This PR is a followup after apache#12375 and apache#12704 it improves handling of some errors in cli commands to avoid show users to much traceback and uses SystemExit consitently.

This PR is a followup after #12375 and #12704 it improves handling of some errors in cli commands to avoid show users to much traceback and uses SystemExit consistently.

boring-cyborg bot added the area:CLI label Nov 29, 2020

turbaszek requested review from XD-DENG, ashb, dimberman, kaxil, mik-laj and potiuk November 29, 2020 15:35

XD-DENG reviewed Nov 29, 2020

View reviewed changes

airflow/cli/simple_table.py Outdated Show resolved Hide resolved

airflow/cli/simple_table.py Outdated Show resolved Hide resolved

docs/usage-cli.rst Outdated Show resolved Hide resolved

turbaszek force-pushed the refactor-list-commands branch 2 times, most recently from 780d3da to 02a5ff2 Compare November 30, 2020 08:13

This comment has been minimized.

Sign in to view

turbaszek force-pushed the refactor-list-commands branch from 02a5ff2 to 48416ed Compare November 30, 2020 14:51

turbaszek requested a review from XD-DENG November 30, 2020 18:10

XD-DENG reviewed Nov 30, 2020

View reviewed changes

airflow/cli/commands/connection_command.py Outdated Show resolved Hide resolved

XD-DENG reviewed Nov 30, 2020

View reviewed changes

airflow/cli/commands/task_command.py Outdated Show resolved Hide resolved

turbaszek commented Nov 30, 2020

View reviewed changes

XD-DENG reviewed Dec 1, 2020

View reviewed changes

turbaszek force-pushed the refactor-list-commands branch from 922abe5 to 87533da Compare December 1, 2020 09:49

turbaszek added 3 commits December 1, 2020 12:38

Refactor list rendering in commands

904f415

This commit unifies the mechanism of rendering output of tabular data. This gives users a possibility to eiter display a tabular representation of data or render it as valid json or yaml payload. Closes: apache#12699

fixup! Refactor list rendering in commands

2d5aeba

Add note in docs

58dc49b

turbaszek added 4 commits December 1, 2020 12:38

fixup! Add note in docs

e546e52

fixup! fixup! Add note in docs

a408961

fixup! fixup! fixup! Add note in docs

5738002

fixup! fixup! fixup! fixup! Add note in docs

03e4867

turbaszek force-pushed the refactor-list-commands branch from 87533da to 0006363 Compare December 1, 2020 11:38

turbaszek requested a review from XD-DENG December 1, 2020 11:38

XD-DENG approved these changes Dec 1, 2020

View reviewed changes

github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Dec 1, 2020

fixup! fixup! fixup! fixup! fixup! Add note in docs

d12f31a

turbaszek force-pushed the refactor-list-commands branch from 0006363 to d12f31a Compare December 1, 2020 14:19

potiuk approved these changes Dec 1, 2020

View reviewed changes

kaxil reviewed Dec 1, 2020

View reviewed changes

UPDATING.md Outdated Show resolved Hide resolved

kaxil reviewed Dec 1, 2020

View reviewed changes

tests/cli/commands/test_task_command.py Show resolved Hide resolved

kaxil reviewed Dec 1, 2020

View reviewed changes

tests/cli/commands/test_dag_command.py Show resolved Hide resolved

Update UPDATING.md

261ab74

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

turbaszek requested a review from kaxil December 1, 2020 23:26

kaxil approved these changes Dec 1, 2020

View reviewed changes

turbaszek merged commit cba8d62 into apache:master Dec 2, 2020

turbaszek deleted the refactor-list-commands branch December 2, 2020 09:20

turbaszek mentioned this pull request Dec 2, 2020

Remove side effects from TestCliDags and TestCliTasks #12753

Closed

turbaszek mentioned this pull request Dec 2, 2020

Improve error handling in cli and introduce consistency #12764

Merged

XD-DENG mentioned this pull request Dec 11, 2020

Handle None values properly when CLI output is YAML/JSON format #13024

Merged

mik-laj mentioned this pull request Feb 27, 2021

The tables are not parsable by standard linux utilities. #14517

Closed

Refactor list rendering in commands #12704

Refactor list rendering in commands #12704

Uh oh!

Conversation

turbaszek commented Nov 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example:

Using jq

Using yq

Uh oh!

turbaszek commented Nov 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XD-DENG commented Nov 29, 2020

Uh oh!

XD-DENG left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

turbaszek commented Nov 30, 2020

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 1, 2020

Uh oh!

github-actions bot commented Dec 1, 2020

Uh oh!

XD-DENG left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 1, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

turbaszek commented Nov 29, 2020 •

edited

Loading

Using `jq`

Using `yq`

turbaszek commented Nov 29, 2020 •

edited

Loading