Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IdeaPluginIntegrationTest is flaky with Python 3 #7150

Closed
stuhood opened this issue Jan 24, 2019 · 3 comments

Comments

Projects
3 participants
@stuhood
Copy link
Member

commented Jan 24, 2019

Various test methods (one of which is test_idea_plugin_single_target) can fail with:

 IdeaPluginIntegrationTest.test_idea_plugin_single_target 

self = <pants_test.backend.project_info.tasks.test_idea_plugin_integration.IdeaPluginIntegrationTest testMethod=test_idea_plugin_single_target>

    def test_idea_plugin_single_target(self):
      target = 'examples/src/scala/org/pantsbuild/example/hello:hello'
    
>     self._run_and_check([target])

.pants.d/pyprep/sources/67eac28d5ba103629c5ed2e3e4f43fc169cf7286/pants_test/backend/project_info/tasks/test_idea_plugin_integration.py:102: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.pants.d/pyprep/sources/67eac28d5ba103629c5ed2e3e4f43fc169cf7286/pants_test/backend/project_info/tasks/test_idea_plugin_integration.py:95: in _run_and_check
    project_dir = self._get_project_dir(output_file.name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pants_test.backend.project_info.tasks.test_idea_plugin_integration.IdeaPluginIntegrationTest testMethod=test_idea_plugin_single_target>
output_file = '/home/travis/build/pantsbuild/pants/.pants.d/tmp/tmpr7jeryy2.pants.d/tmpwh44n4vp'

    def _get_project_dir(self, output_file):
      with open(output_file, 'r') as result:
>       return result.readlines()[0]
E       IndexError: list index out of range

.pants.d/pyprep/sources/67eac28d5ba103629c5ed2e3e4f43fc169cf7286/pants_test/backend/project_info/tasks/test_idea_plugin_integration.py:67: IndexError

This is likely caused by some other underlying flakiness (unfortunately). But nonetheless.

@stuhood stuhood added the flaky-test label Jan 24, 2019

@wisechengyi

This comment has been minimized.

Copy link
Contributor

commented Jan 25, 2019

Interesting, these tests have there for a while and were not flaky. Wonder what's changed.

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor

commented Jan 25, 2019

Hm, I saw this while working on #7141 and thought it was due to unicode issues that I solved in the PR. I'll dig more into this right now and try to stabilize.

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor

commented Jan 25, 2019

@stuhood I'm able to reproduce locally with ./pants3 clean-all test tests/python/pants_test/backend/project_info/tasks:idea_plugin_integration with about 20+ runs, with test_idea_plugin_single_target() failing.

I could not reproduce with ./pants clean-all test tests/python/pants_test/backend/project_info/tasks:idea_plugin_integration (py2) after running below script for about 30 minutes.

#!/bin/bash

while $@; do :; done

So, this started flaking as of #7141.

My hypothesis is the command has always been flakey with Py3 for some reason and we simply weren't testing it as it was on the blacklist. I do not think the changes I made in #7141 caused it, as I distinctly remember the error before I made the fixes, so it's more likely they were always there.

--

I'm trying to find the cause.. With how Python works, we know the output file exists. The issue is it's not being written to.

Let me know if we should revert #7141 until we solve this.

@Eric-Arellano Eric-Arellano changed the title IdeaPluginIntegrationTest has multiple flaky tests IdeaPluginIntegrationTest is flaky with Python 3 Jan 25, 2019

cosmicexplorer added a commit that referenced this issue Jan 25, 2019

blacklist idea plugin gen from faulty earlier edit (#7157)
### Problem

See #7150. The idea plugin gen integration test became flaky on py3 as of #7141.

### Solution

Blacklist test on py3 as per [slack discussion](https://pantsbuild.slack.com/archives/CBNMV1LRH/p1548436157036300).

@Eric-Arellano Eric-Arellano added this to In progress in Adding Python 3 support Jan 25, 2019

@Eric-Arellano Eric-Arellano self-assigned this Jan 26, 2019

@Eric-Arellano Eric-Arellano moved this from In progress to To do in Adding Python 3 support Mar 8, 2019

@Eric-Arellano Eric-Arellano moved this from To do to In progress in Adding Python 3 support Mar 30, 2019

Eric-Arellano added a commit that referenced this issue Mar 30, 2019

Fix flaky `./pants idea-plugin` when using Python 3 by properly using…
… ConsoleTask interface (#7460)

### Problem
The Idea plugin integration tests would frequently flake with Python 3.

This appears to be because the task is a `ConsoleTask`, but it overrides `execute()` when it is instead expected to override `console_output()`. With this override, we would no longer call `self._outstream.flush()`, so the tests would sometimes complain that there was no entry in the `--output-file`, as the change would not always be persisted. It makes sense that this did not start flaking until Python 3, because IO streams changed their behavior in Python 3.

Refer to `ConsoleTask` for everything we need to do to properly output to the console and files: https://github.com/pantsbuild/pants/blob/e620a9e862a245088bc5b477aed2b87e2bc6a307/src/python/pants/task/console_task.py#L52-L62

Will close #7150.

### Solution
No longer override `execute()` and instead move the logic into `console_output()`.

### Result
The tests no longer flake.

This was confirmed locally by creating the script `untilfail`:

```bash
#!/bin/bash

while $@; do :; done
```

Then running `./untilfail ./pants clean-all test.pytest tests/python/pants_test/backend/project_info/tasks:idea_plugin_integration -- -k test_idea_plugin_single_target` for 10 minutes on both macOS and Ubuntu without fail.

@Eric-Arellano Eric-Arellano moved this from In progress to Done in Adding Python 3 support Mar 30, 2019

stuhood added a commit that referenced this issue Apr 1, 2019

Fix flaky `./pants idea-plugin` when using Python 3 by properly using…
… ConsoleTask interface (#7460)

### Problem
The Idea plugin integration tests would frequently flake with Python 3.

This appears to be because the task is a `ConsoleTask`, but it overrides `execute()` when it is instead expected to override `console_output()`. With this override, we would no longer call `self._outstream.flush()`, so the tests would sometimes complain that there was no entry in the `--output-file`, as the change would not always be persisted. It makes sense that this did not start flaking until Python 3, because IO streams changed their behavior in Python 3.

Refer to `ConsoleTask` for everything we need to do to properly output to the console and files: https://github.com/pantsbuild/pants/blob/e620a9e862a245088bc5b477aed2b87e2bc6a307/src/python/pants/task/console_task.py#L52-L62

Will close #7150.

### Solution
No longer override `execute()` and instead move the logic into `console_output()`.

### Result
The tests no longer flake.

This was confirmed locally by creating the script `untilfail`:

```bash
#!/bin/bash

while $@; do :; done
```

Then running `./untilfail ./pants clean-all test.pytest tests/python/pants_test/backend/project_info/tasks:idea_plugin_integration -- -k test_idea_plugin_single_target` for 10 minutes on both macOS and Ubuntu without fail.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.