Skip to content

ITHttpInputSourceTest instability blocking the development pipeline#11749

Merged
jihoonson merged 1 commit intoapache:masterfrom
loquisgon:ignore-is-it
Sep 28, 2021
Merged

ITHttpInputSourceTest instability blocking the development pipeline#11749
jihoonson merged 1 commit intoapache:masterfrom
loquisgon:ignore-is-it

Conversation

@loquisgon
Copy link

@loquisgon loquisgon commented Sep 28, 2021

The test has become very unstable recently and it is blocking the development pipeline. Offline debugging indicates that this is potentially a Travis memory limitation issue for docker where due to some reason (Travis change? Software change?) the runtime memory requirement increased to make this test fail. The docker memory issue described above is just a hypothesis based on log interpretation and local tests. Debugging and finding a solution may take some time so I am proposing to ignore this test to unblock the pipeline while I debug this and find a solution. The final solution may include some memory parameter changes and/or some structural changes to the test.

@suneet-s
Copy link
Contributor

Partial update for #11744 so non-ingestion commits can continue to be merged till a complete fix is implemented

Copy link
Contributor

@suneet-s suneet-s left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for this approach of disabling tests first while the investigation continues on how to get a complete fix for this test.

LGTM after CI

@jihoonson
Copy link
Contributor

@loquisgon please update the title and the description. The title will be included in the git commit history, so it should be intuitive enough to understand the overview of change. What test will be ignored? Who's "we" in the title? Also, I feel a bit uncomfortable about this change because the PR description currently lacks the plan for the real fix. Who will continue working on the real fix? What is the plan for the follow-up?

@loquisgon loquisgon changed the title Ignore for now to unblock pipeline, we'll keep debugging offline ITHttpInputSourceTest instability blocking the development pipeline Sep 28, 2021
@loquisgon
Copy link
Author

loquisgon commented Sep 28, 2021

@jihoonson I have updated the title and description but this PR is exactly what the previous title and description indicated. It is a pragmatic, tactical, action to just unblock the pipeline while I continue debugging the issue offline. I have identified docker related memory pressure issues in my local environment that make this test fail with some similarity to the Travis failures. But getting to the bottom may take some more time due to the slow debug cycle. The final solution may be in the form of having parameters change and/or structural test changes (maybe don't do memory intensive ingestion if all we are looking for is to test an input source). But I don't know yet.

@jihoonson
Copy link
Contributor

@loquisgon thanks for updating the title and description. I asked to update them not because I don't agree with the motivation of this PR, but it lacks details. It didn't say what test you are disabling and why it is a good temporary workaround to unblock development. They are clear now. Thanks for looking into the issue. I'm going to approve this PR, but we should wait for Travis to finish and see whether it is less flaky than before.

@jihoonson
Copy link
Contributor

Cool, Travis seems to pass all tests without restart! At least I didn't restart any job. I'm going to merge this. Thanks again @loquisgon for working on this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants