Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metadata API in destination service not configured for k8s Jobs #11531

Closed
siggy opened this issue Oct 24, 2023 · 0 comments · Fixed by #11541 or #11543
Closed

Metadata API in destination service not configured for k8s Jobs #11531

siggy opened this issue Oct 24, 2023 · 0 comments · Fixed by #11541 or #11543
Labels

Comments

@siggy
Copy link
Member

siggy commented Oct 24, 2023

What is the issue?

I'm observing lots of failed to retrieve job from indexer warnings in the destination container logs, for pods belonging to Jobs:

case "Job":
parentObj, err = api.getByNamespace(Job, pod.Namespace, parent.Name)
if err != nil {
log.Warnf("failed to retrieve job from indexer %s/%s: %s", pod.Namespace, parent.Name, err)
if retry {
parentObj, err = api.client.
Resource(batchv1.SchemeGroupVersion.WithResource("jobs")).
Namespace(pod.Namespace).
Get(ctx, parent.Name, metav1.GetOptions{})
if err != nil {
log.Warnf("failed to retrieve job from direct API call %s/%s: %s", pod.Namespace, parent.Name, err)
}
}
}

It appears the metadata API client is only configured for Nodes and ReplicaSets:

metadataAPI, err := k8s.InitializeMetadataAPI(*kubeConfigPath, "local", k8s.Node, k8s.RS)

How can it be reproduced?

Start some k8s Jobs (I don't have a repro)?

Logs, error output, etc

seeing these warnings in the destination container:

2023-10-24T02:58:13.881735669Z time="2023-10-24T02:58:13Z" level=warning msg="failed to retrieve job from indexer [namespace]/[pod]: metadata informer (6) not configured"

output of linkerd check -o short

N/A

Environment

stable-2.14.1

Possible solution

Modify the call to InitializeMetadataAPI to include Job.

Additional context

No response

Would you like to work on fixing this bug?

maybe

@siggy siggy added the bug label Oct 24, 2023
mateiidavid added a commit that referenced this issue Oct 27, 2023
This edge release includes a fix for the `ServiceProfile` CRD resource schema.
The schema incorrectly required `not` response matches to be arrays, while the
in-cluster validator parsed `not` response matches as objects. In addition, an
issues has been fixed in `linkerd profile`. When used with the `--open-api`
flag, it would not strip trailing slashes when generating a resource from
swagger specifications.

* Fixed an issue where trailing slashes wouldn't be stripped when generating
  `ServiceProfile` resources through `linkerd profile --open-api` ([#11519])
* Fixed an issue in the `ServiceProfile` CRD schema. The schema incorrectly
  required that a `not` response match should be an array, which the service
  profile validator rejected since it expected an object. The schema has been
  updated to properly indicate that `not` values should be an object ([#11510];
  fixes [#11483])
* Improved logging in the destination controller by adding the client pod's
  name to the logging context. This will improve visibility into the messages
  sent and received by the control plane from a specific proxy ([#11532])
* Fixed an issue in the destination controller where the metadata API would not
  initialize a `Job` informer. The destination controller uses the metadata API
  to retrieve `Job` metadata, and relies mostly on informers. Without an
  initialized informer, an error message would be logged, and the controller
  relied on direct API calls ([#11541]; fixes [#11531])

[#11541]: #11532
[#11532]: #11532
[#11531]: #11531
[#11519]: #11519
[#11510]: #11510
[#11483]: #11483

Signed-off-by: Matei David <matei@buoyant.io>
mateiidavid added a commit that referenced this issue Oct 27, 2023
This edge release includes a fix for the `ServiceProfile` CRD resource schema.
The schema incorrectly required `not` response matches to be arrays, while the
in-cluster validator parsed `not` response matches as objects. In addition, an
issues has been fixed in `linkerd profile`. When used with the `--open-api`
flag, it would not strip trailing slashes when generating a resource from
swagger specifications.

* Fixed an issue where trailing slashes wouldn't be stripped when generating
  `ServiceProfile` resources through `linkerd profile --open-api` ([#11519])
* Fixed an issue in the `ServiceProfile` CRD schema. The schema incorrectly
  required that a `not` response match should be an array, which the service
  profile validator rejected since it expected an object. The schema has been
  updated to properly indicate that `not` values should be an object ([#11510];
  fixes [#11483])
* Improved logging in the destination controller by adding the client pod's
  name to the logging context. This will improve visibility into the messages
  sent and received by the control plane from a specific proxy ([#11532])
* Fixed an issue in the destination controller where the metadata API would not
  initialize a `Job` informer. The destination controller uses the metadata API
  to retrieve `Job` metadata, and relies mostly on informers. Without an
  initialized informer, an error message would be logged, and the controller
  relied on direct API calls ([#11541]; fixes [#11531])

[#11541]: #11532
[#11532]: #11532
[#11531]: #11531
[#11519]: #11519
[#11510]: #11510
[#11483]: #11483

Signed-off-by: Matei David <matei@buoyant.io>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
1 participant