Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fetch intermediate log async GKEStartPod #39348

Merged
merged 5 commits into from
May 29, 2024

Conversation

pankajastro
Copy link
Member

@pankajastro pankajastro commented May 1, 2024

This PR introduces a parameter that enables the retrieval of intermediate logs for the GKEStartPod asynchronous operator.

  • Add param last_log_time and logging_interval in GKEStartPodTrigger serialize
  • Add optional param last_log_time in method invoke_defer_method

Example DAG:

start_pod = GKEStartPodOperator(
        task_id="start_pod",
        project_id=PROJECT_ID,
        location=LOCATION,
        cluster_name=GKE_CLUSTER_NAME,
        do_xcom_push=True,
        namespace=GKE_NAMESPACE,
        image="ubuntu:jammy",
        cmds=["sh", "-c", "timeout 300 bash -c 'while true; do echo \"meow\"; sleep 30; done'"],
        name="test-sleep",
        in_cluster=False,
        on_finish_action="delete_pod",
        deferrable=True,
        get_logs=True,
        logging_interval=5,
        gcp_conn_id=GCP_CONN_ID
    )
Screenshot 2024-05-01 at 6 53 50 PM

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:cncf-kubernetes Kubernetes provider related issues provider:google Google (including GCP) related issues labels May 1, 2024
@pankajastro pankajastro marked this pull request as ready for review May 1, 2024 13:32
Copy link
Collaborator

@dirrao dirrao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Can you add the test case with last_log_time and last_log_time?

@jonathonbattista
Copy link

Any updates on this?

@pankajastro
Copy link
Member Author

Any updates on this?

I'll try address the pending review by end of this week

@jonathonbattista
Copy link

Bump on this 🙏

@jonathonbattista
Copy link

@pankajastro Any ETA on this? Thanks

@pankajastro
Copy link
Member Author

@pankajastro Any ETA on this? Thanks

CI is green now. Hopefully, we will merge soon. In the meantime, could you please test it once?

@pankajastro pankajastro merged commit a2c09d2 into apache:main May 29, 2024
41 checks passed
@pankajastro pankajastro deleted the logging_interval_gke branch May 29, 2024 12:39
fdemiane pushed a commit to fdemiane/airflow that referenced this pull request Jun 6, 2024
* Fetch intermediate log in async GKEStartPod

This PR introduces a parameter that enables the retrieval of intermediate logs for the GKEStartPod asynchronous operator.

Add param last_log_time and logging_interval in GKEStartPodTrigger serialize
Add optional param last_log_time in method invoke_defer_method
Example DAG:

start_pod = GKEStartPodOperator(
        task_id="start_pod",
        project_id=PROJECT_ID,
        location=LOCATION,
        cluster_name=GKE_CLUSTER_NAME,
        do_xcom_push=True,
        namespace=GKE_NAMESPACE,
        image="ubuntu:jammy",
        cmds=["sh", "-c", "timeout 300 bash -c 'while true; do echo \"meow\"; sleep 30; done'"],
        name="test-sleep",
        in_cluster=False,
        on_finish_action="delete_pod",
        deferrable=True,
        get_logs=True,
        logging_interval=5,
        gcp_conn_id=GCP_CONN_ID
    )
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:cncf-kubernetes Kubernetes provider related issues provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants