feat(api/endpoint): allow filtering eventlogs by attributes #34417

hainenber · 2023-09-16T08:30:12Z

feat(api/endpoint): Allow filtering for Event Logs at /eventlogs endpoint using the entity's attributes.

Tests added and API schema updated

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

boring-cyborg · 2023-09-16T08:30:17Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

hussein-awala · 2023-09-16T15:30:45Z

airflow/api_connexion/endpoints/event_log_endpoint.py

+    if event:
+        query = query.where(Log.event == event)
+    if before:
+        query = query.where(Log.dttm <= timezone.parse(before))


Usually we use strictly less then before in this type of queries.

Fixed in latest commit. Thanks for the review!

hussein-awala · 2023-09-16T15:39:26Z

airflow/api_connexion/openapi/v1.yaml

+    EventLogDAGID:
+      in: query
+      name: dag_id
+      schema:
+        type: string
+      required: false
+      description: DAG ID of an event log.


We have already a dag id param:

airflow/airflow/api_connexion/openapi/v1.yaml

Lines 4797 to 4803 in 6c649ae

DAGID:

in: path

name: dag_id

schema:

type: string

required: true

description: The DAG ID.

so I don't know if we should use it or create a new one, @pierrejeambrun wdyt?

I think we should re-use it if this is exatly the same thing. No need to duplicate I believe

Actually this dag_id parameter works for path and not as query parameter. In addition, it's set as required while eventlogs's filtering parameters aren't strictly required.

In this case, maybe we can prefix that with Filter and put them in the Filter section where other filters reside in the spec. Also maybe a generic name such as FilterDAGID so we can easily reuse it. (shouldn't be specific to EventLog I believe)

Parameters are now converted to filters in latest commit. Thanks!

hussein-awala · 2023-09-16T15:39:43Z

airflow/api_connexion/openapi/v1.yaml

+    EventLogTaskID:
+      in: query
+      name: task_id
+      schema:
+        type: string
+      required: false
+      description: Task ID of an event log.


same for task id

Thanks for the review! I've replied in above review.

hussein-awala · 2023-09-16T15:42:22Z

tests/api_connexion/endpoints/test_event_log_endpoint.py

+        eventlog1 = create_log_model(event="1", dag_id="1", task_id="1", owner="1", when=self.default_time)
+        eventlog2 = create_log_model(event="2", dag_id="2", task_id="2", owner="2", when=self.default_time_2)


Could you try to use different values? (ex event_1, dag_1, task_1, ...) instead of having the same value for all the variables, without that we cannot ensure that the filter is really working.

Fixed in latest commit. Thanks for the review!

hainenber · 2023-09-24T15:33:34Z

hello there, can someone help reviewing and merge if deemed sufficient? Thanks!

bbovenzi

lgtm

airflow/api_connexion/openapi/v1.yaml

hainenber requested review from ryanahamilton, ashb, bbovenzi, pierrejeambrun and ephraimbuddy as code owners September 16, 2023 08:30

boring-cyborg bot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues labels Sep 16, 2023

hainenber force-pushed the allow-filtering-eventlogs-by-attrs branch from 6bd57a7 to b6a6c67 Compare September 16, 2023 08:44

hussein-awala reviewed Sep 16, 2023

View reviewed changes

eladkal added this to the Airflow 2.8.0 milestone Sep 16, 2023

eladkal added the type:improvement Changelog: Improvements label Sep 16, 2023

hainenber force-pushed the allow-filtering-eventlogs-by-attrs branch 3 times, most recently from 5aa520e to 4af14ac Compare September 21, 2023 13:57

hainenber added 3 commits September 21, 2023 22:17

feat(api/endpoint): allow filter eventlogs by attrs

0426228

fix(api/endpoint): filter event logsin time without equality

d4258e6

chore(api/endpoint): reduce line length

b3d1b8a

hainenber force-pushed the allow-filtering-eventlogs-by-attrs branch from 4af14ac to b3d1b8a Compare September 21, 2023 15:17

feat(api/eventlog): convert param to generic filter

ef42b72

bbovenzi approved these changes Sep 26, 2023

View reviewed changes

hussein-awala approved these changes Sep 26, 2023

View reviewed changes

airflow/api_connexion/openapi/v1.yaml Outdated Show resolved Hide resolved

Update v1.yaml

02bcddf

uranusjr merged commit 3189ebe into apache:main Sep 27, 2023
43 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(api/endpoint): allow filtering eventlogs by attributes #34417

feat(api/endpoint): allow filtering eventlogs by attributes #34417

hainenber commented Sep 16, 2023

boring-cyborg bot commented Sep 16, 2023

hussein-awala Sep 16, 2023

hainenber Sep 17, 2023

hussein-awala Sep 16, 2023

pierrejeambrun Sep 16, 2023

hainenber Sep 17, 2023

pierrejeambrun Sep 22, 2023

hainenber Sep 23, 2023

hussein-awala Sep 16, 2023

pierrejeambrun Sep 16, 2023

hainenber Sep 17, 2023

hussein-awala Sep 16, 2023

hainenber Sep 17, 2023

hainenber commented Sep 24, 2023

bbovenzi left a comment

	DAGID:
	in: path
	name: dag_id
	schema:
	type: string
	required: true
	description: The DAG ID.

		eventlog1 = create_log_model(event="1", dag_id="1", task_id="1", owner="1", when=self.default_time)
		eventlog2 = create_log_model(event="2", dag_id="2", task_id="2", owner="2", when=self.default_time_2)

feat(api/endpoint): allow filtering eventlogs by attributes #34417

feat(api/endpoint): allow filtering eventlogs by attributes #34417

Conversation

hainenber commented Sep 16, 2023

boring-cyborg bot commented Sep 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hainenber commented Sep 24, 2023

bbovenzi left a comment

Choose a reason for hiding this comment