Get logs for all pipelines, even if they're not running #9749
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jrockway
force-pushed
the
jonathan/core-2144-get-logs-for-all-pipelines
branch
from
298d6b5
to
d70c737
Compare
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #9749 +/- ##
==========================================
+ Coverage 59.10% 59.16% +0.05%
==========================================
Files 582 582
Lines 69880 69904 +24
==========================================
+ Hits 41304 41358 +54
+ Misses 27983 27964 -19
+ Partials 593 582 -11 ☔ View full report in Codecov by Sentry. |
robert-uhl
reviewed
Feb 20, 2024
| func (s *debugServer) listApps(ctx context.Context) (_ []*debug.App, retErr error) { | ||
| // list apps returns a list of running apps, and a list of apps which may possibly exist. The | ||
| // intent is to use the first result for things like "kubectl describe" and the second for getting | ||
| // loki logs. |
There was a problem hiding this comment.
Could this be refactored? Might it be clearer to merge podApps (which would be the old listApps and pipelineApps (which would generate all apps for all known pipelines), rather than in GetDumpV2Template listing pipelines, passing that into listApps, then in listApps iterating over running pods, deleting a pipeline pod’s associated pod from the list of pipelines, then creating an app for each remaining pipeline?
There was a problem hiding this comment.
This should really be implemented in some completely different way; running pods should only affect logs, loki should be queried to enumerate all logs that loki can get, etc. But this is a release blocker because everyone is using autoscaling now and all the debug dumps that people are generating are useless, so I'm trying to keep it as minimal as possible for the backport.
jrockway
force-pushed
the
jonathan/core-2144-get-logs-for-all-pipelines
branch
from
d70c737
to
c78845c
Compare
jrockway
added a commit
that referenced
this pull request
Feb 20, 2024
The current algorithm for determining which pods to get logs for
involves looking at running pods. This is not correct for pipelines; a
pipeline could be autoscaled to 0. This PR adjusts the algorithm to add
in any pipelines that don't currently have pods to the lokiLogs request.
For example:
```
$ pachctl debug template
...
describes:
- name: console
- name: default/edges
pipeline:
name: edges
project: default
- name: etcd
- name: pachd
- name: pachyderm-kube-event-tail
- name: pachyderm-proxy
- name: pg-bouncer
logs:
- name: console
- name: default/edges
pipeline:
name: edges
project: default
- name: etcd
- name: pachd
- name: pachyderm-kube-event-tail
- name: pachyderm-proxy
- name: pg-bouncer
lokiLogs:
- name: console
- name: default/edges
pipeline:
name: edges
project: default
- name: default/montage
pipeline:
name: montage
project: default
- name: etcd
- name: pachd
- name: pachyderm-kube-event-tail
- name: pachyderm-proxy
- name: pg-bouncer
...
```
Montage has autoscaling and isn't currently up; it now appears in
lokiLogs, but not logs or describes.
acohen4
pushed a commit
that referenced
this pull request
Feb 21, 2024
… (#9750) The current algorithm for determining which pods to get logs for involves looking at running pods. This is not correct for pipelines; a pipeline could be autoscaled to 0. This PR adjusts the algorithm to add in any pipelines that don't currently have pods to the lokiLogs request. For example: ``` $ pachctl debug template ... describes: - name: console - name: default/edges pipeline: name: edges project: default - name: etcd - name: pachd - name: pachyderm-kube-event-tail - name: pachyderm-proxy - name: pg-bouncer logs: - name: console - name: default/edges pipeline: name: edges project: default - name: etcd - name: pachd - name: pachyderm-kube-event-tail - name: pachyderm-proxy - name: pg-bouncer lokiLogs: - name: console - name: default/edges pipeline: name: edges project: default - name: default/montage pipeline: name: montage project: default - name: etcd - name: pachd - name: pachyderm-kube-event-tail - name: pachyderm-proxy - name: pg-bouncer ... ``` Montage has autoscaling and isn't currently up; it now appears in lokiLogs, but not logs or describes.
zmajeed
pushed a commit
that referenced
this pull request
Feb 21, 2024
The current algorithm for determining which pods to get logs for
involves looking at running pods. This is not correct for pipelines; a
pipeline could be autoscaled to 0. This PR adjusts the algorithm to add
in any pipelines that don't currently have pods to the lokiLogs request.
For example:
```
$ pachctl debug template
...
describes:
- name: console
- name: default/edges
pipeline:
name: edges
project: default
- name: etcd
- name: pachd
- name: pachyderm-kube-event-tail
- name: pachyderm-proxy
- name: pg-bouncer
logs:
- name: console
- name: default/edges
pipeline:
name: edges
project: default
- name: etcd
- name: pachd
- name: pachyderm-kube-event-tail
- name: pachyderm-proxy
- name: pg-bouncer
lokiLogs:
- name: console
- name: default/edges
pipeline:
name: edges
project: default
- name: default/montage
pipeline:
name: montage
project: default
- name: etcd
- name: pachd
- name: pachyderm-kube-event-tail
- name: pachyderm-proxy
- name: pg-bouncer
...
```
Montage has autoscaling and isn't currently up; it now appears in
lokiLogs, but not logs or describes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current algorithm for determining which pods to get logs for involves looking at running pods. This is not correct for pipelines; a pipeline could be autoscaled to 0. This PR adjusts the algorithm to add in any pipelines that don't currently have pods to the lokiLogs request. For example:
Montage has autoscaling and isn't currently up; it now appears in lokiLogs, but not logs or describes.