Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LOG-4181: fix loki test flake caused by rate limiting #2127

Merged
merged 1 commit into from
Aug 16, 2023

Conversation

jcantrill
Copy link
Contributor

@jcantrill jcantrill commented Aug 15, 2023

Description

This PR:

  • Updates the loki test image to 2.8.4
  • Modifies the ingestion rates and limits to address e2e loki test flake

Links

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 15, 2023
@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 15, 2023

@jcantrill: This pull request references LOG-4181 which is a valid jira issue.

In response to this:

Description

This PR:

  • Updates the loki test image to 2.8.0
  • Modifies the ingestion rates and limits to address e2e loki test flake

Links

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 15, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcantrill

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 15, 2023
Copy link
Contributor

@xperimental xperimental left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The limits seem a bit excessive. This works, but might hide an issue down the road that we then only see when something really goes wrong and even hits those limits.

@@ -26,7 +26,7 @@ import (
)

const (
Image = "grafana/loki:2.5.0"
Image = "grafana/loki:2.8.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd just go with the latest patch version (or the one we use in Loki Operator, 2.8.3):

Suggested change
Image = "grafana/loki:2.8.0"
Image = "grafana/loki:2.8.4"

@jcantrill
Copy link
Contributor Author

bq. The limits seem a bit excessive. This works, but might hide an issue down the road that we then only see when something really goes wrong and even hits those limits.

These limits were added based upon what I saw in the error logs of the test so they are real numbers. My concern more is to resolve the test flake and this test verifies we can write to Loki. We make no changes to Loki config outside of using the defaults which are likely not ideal. The test is intended to be an integration test that verifies connectivity between the collector and Loki. While I agree it could be tweaked better, we would need to define additional goals of the test and bullet proof it accordingly

@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 16, 2023

@jcantrill: This pull request references LOG-4181 which is a valid jira issue.

In response to this:

Description

This PR:

  • Updates the loki test image to 2.8.4
  • Modifies the ingestion rates and limits to address e2e loki test flake

Links

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 16, 2023

@jcantrill: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@xperimental
Copy link
Contributor

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 16, 2023
@openshift-merge-robot openshift-merge-robot merged commit 1b9f9aa into openshift:master Aug 16, 2023
10 checks passed
@jcantrill jcantrill deleted the log4181_on_master branch August 16, 2023 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. release/5.8
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants