Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[frontend] ui can't retrieve pod logs when using argo-workflows 3.4+ (POD_NAMES change) #8935

Closed
thesuperzapper opened this issue Mar 7, 2023 · 10 comments · Fixed by #10568

Comments

@thesuperzapper
Copy link
Member

Currently, if you upgrade to argo-workflows 3.4+, the log-retrieval function of the ml-pipeline-ui will not work, and the following error will be thrown: Could not get main container logs: Error: Unable to retrieve workflow status: [object Object]

This is because argo-workflows 3.4 changed the default value of their POD_NAMES feature flag to v2, which has changed the way pods in workflows get named.

This means that the frontend will not know the correct name of the pod when it tries to retrieve logs because of our workflowNameFromPodName() function.

/**
* Infers workflow name from pod name.
* @param podName name of the pod.
*/
function workflowNameFromPodName(podName: string) {
const chunks = podName.split('-');
chunks.pop();
return chunks.join('-');
}

Some docs about POD_NAMES environment variable meaning:


Impacted by this bug? Give it a 👍.

@v-raja
Copy link

v-raja commented Mar 7, 2023

Might be a duplicate of #8449

@connor-mccarthy connor-mccarthy added this to Needs triage in KFP Runtime Triage via automation Mar 9, 2023
@connor-mccarthy connor-mccarthy moved this from Needs triage to Backlog in KFP Runtime Triage Mar 9, 2023
@connor-mccarthy
Copy link
Member

/assign @jlyaoyuli, FYI

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Aug 26, 2023
@thesuperzapper
Copy link
Member Author

This is still a problem, bumping for bot.

@stale stale bot removed the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Aug 27, 2023
@agilgur5
Copy link

agilgur5 commented Oct 2, 2023

FWIW, I recently had to make a small change to this in Argo's own UI (see argoproj/argo-workflows#11766), and the code for pod names is limited to this single file.

You can also set POD_NAMES=v1 for compatibility.

Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Dec 31, 2023
@thesuperzapper
Copy link
Member Author

Bumping for bot.

@stale stale bot removed the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Feb 13, 2024
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Apr 13, 2024
@thesuperzapper
Copy link
Member Author

thesuperzapper commented Apr 13, 2024

@zijianjoy @connor-mccarthy @chensun this is still very important, as while we can tell users to force the old pod naming with the POD_NAMES=v1 env-var, it does make it a lot harder for people to use their Argo Workflows that they already have.

There is a similar issue about how we can't retrieve task results in Argo Workflows 3.4+

@stale stale bot removed the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Apr 13, 2024
@HumairAK
Copy link
Contributor

@thesuperzapper this is resolved in #10568 via a workaround where we generate the name ourselves, and in future argo releases once this is merged, we can switch to retrieving it from here once again instead

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

Successfully merging a pull request may close this issue.

6 participants