[Log Collector] Change log file names to old format #3647
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Tankilevitch
suggested changes
May 25, 2023
| return nil | ||
| }); err != nil { | ||
| // get run log file path | ||
| runLogFilePath := s.resolvePodLogFilePath(projectName, runUID) |
There was a problem hiding this comment.
Suggested change
| runLogFilePath := s.resolvePodLogFilePath(projectName, runUID) | |
| runLogFilePath := s.resolveRunLogFilePath(projectName, runUID) |
Tankilevitch
approved these changes
May 25, 2023
TomerShor
added a commit
to TomerShor/mlrun
that referenced
this pull request
May 25, 2023
TomerShor
added a commit
to TomerShor/mlrun
that referenced
this pull request
May 25, 2023
TomerShor
added a commit
to TomerShor/mlrun
that referenced
this pull request
May 25, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue:
We have seen in the field that the current implementation, of storing the log files in the
<project>/<run_uid>_<pod_name>format, proves to be problematic when getting the log files in systems with an extensive number of runs.The issue happens when listing the logs directory to find the latest run log. Our flex fuse that mlrun is using for storing the logs cannot handle listing a directory with >2000 runs in it, and the log collector sidecar gets stuck.
Fix:
To fix it, we revert back to the old log file path format -
<project>/<run_uid>.We have done some research and found that currently each run has only one pod, and thus it will have only one file.
This enables us to remove the listDir operation to find the most recent log file, and just use the constant filename and getting it directly.
The change is relevant both in the log collector and in the API's legacy method.