Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run each test suite with a pristine ATOM_HOME directory #19457

Merged
merged 2 commits into from Jun 7, 2019

Conversation

Projects
None yet
2 participants
@as-cii
Copy link
Member

commented Jun 6, 2019

I had originally opened this pull request to address flakiness in autocomplete-plus tests, but in investigating that issue I discovered a couple of problems:

  1. compile-cache was being symlinked in AtomWindow tests. This was causing some issues when the ATOM_HOME folder did not exist, but it has been fixed as part of 6a88fa4. Furthermore, it allowed us to move main process tests out of the macOS Build step, which resulted in removing quite a bit of duplication between macOS_build and macOS_test.
  2. When running script/test, there was the potential for previous tests to affect the behavior of subsequent ones, because ATOM_HOME would be reused across test suites. I addressed this in ee0ddaa, which will now create a new temporary directory for each suite.

I've run tests multiple times with the two improvements illustrated above, and those seem to mitigate some of the flakiness we were observing for autocomplete-plus. I am still not entirely sure of what the root cause of the flakiness was (I couldn't reproduce the problem locally, and debugging tests on CI has an extremely bad feedback loop), but these seem to be good enhancements to our test suite nonetheless.

Don't symlink compile-cache folder in AtomWindow tests
In these tests, we create a temporary `ATOM_HOME` to avoid cluttering 
the user's real `~/.atom` folder.

Adding a symlink to the real `compile-cache` was introduced to speed up 
main process tests, so that the transpilation cache could be reused. 

Unfortunately, when the real `~/.atom` folder did not exist (such as on 
a pristine environment on CI), it would confuse Atom, which would think 
that it didn't need to re-create a `compile-cache` folder again, but 
wouldn't be able to write to it because the symlink pointed to a 
non-existant directory.

Main process tests were overhauled and made faster recently, so we can 
safely remove this performance optimization.

@as-cii as-cii force-pushed the as/debug-autocomplete-failures branch 2 times, most recently from 66b21c5 to 9522bb3 Jun 6, 2019

Run each test suite with a pristine ATOM_HOME directory
This ensures that every test suite does not clutter subsequent ones. It 
will also prevent altering the user's `~/.atom` directory when running 
tests locally.

@as-cii as-cii force-pushed the as/debug-autocomplete-failures branch from b6f9bec to ee0ddaa Jun 7, 2019

@as-cii as-cii changed the title Debug autocomplete-plus failures on Azure Pipelines Run each test suite with a pristine ATOM_HOME directory Jun 7, 2019

@as-cii as-cii marked this pull request as ready for review Jun 7, 2019

@as-cii as-cii merged commit 2e6a0ac into master Jun 7, 2019

1 of 2 checks passed

continuous-integration/appveyor/pr AppVeyor build failed
Details
Atom Pull Requests #20190607.2 succeeded
Details

@as-cii as-cii deleted the as/debug-autocomplete-failures branch Jun 7, 2019

@rafeca

This comment has been minimized.

Copy link
Contributor

commented Jun 10, 2019

It seems like we're still getting build failures on autocomplete-plus (link). I'm gonna investigate them

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.