Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pytest-xdist prevents disturbing stdout, revert to travis_wait #1566

Merged
merged 2 commits into from
Nov 26, 2019

Conversation

reaperhulk
Copy link
Contributor

This prevents us from seeing realtime output but in exchange I've made pytest fast-fail so jobs won't run past the first failing test. Not ideal but we'll keep improving

This prevents us from seeing realtime output but in exchange I've made
pytest fast-fail so jobs won't run past the first failing test. Not
ideal but we'll keep improving
@ehennenfent
Copy link
Contributor

While I'm glad you've gotten the tests passing, I'm unsure about killing pytest on the first failing test. Usually we want to at least know about all of the failures so that we can effectively debug them (especially if there are multiple bugs in one PR). If we kill the tests after the first failure, then the first step for debugging from now on will be to re-run the full test suite somewhere other than Travis. On a smaller project, I'd say that's fine, but I don't think it's a constraint we should impose on Manticore because not only can our tests take hours, but they also require one to properly set up keystone and solc, which is unfortunately nontrivial.

@reaperhulk
Copy link
Contributor Author

I'm happy to do whatever the actual people doing the real work on this project want! I'll go ahead and remove the fast fail behavior.

@ehennenfent
Copy link
Contributor

Definitely appreciate that you're messing around with potential improvements to the CI, so feel free to keep filing PR's. Depending on how things with pytest go, we may want to revert #1561 and work on consolidating these changes into one big PR (assuming we don't find another way to get it working in the next couple of days).

Copy link
Contributor

@ehennenfent ehennenfent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go ahead and merge this so we can rebase the other PR's and get the tests passing again.

@ehennenfent ehennenfent merged commit ea112ee into trailofbits:master Nov 26, 2019
ekilmer added a commit that referenced this pull request Dec 17, 2019
* master: (33 commits)
  removes travis entirely (#1572)
  Run mypy on manticore directory by default (#1573)
  Fix deprecation warnings related to setting verbosity (#1574)
  Bump version to fix pip install on Python 3.6 (#1571)
  Direct CI badge directly to CI action (#1570)
  switch to github actions (take two) (#1568)
  Add --quick-mode to configure Manticore for fast exploration (#1555)
  pytest-xdist prevents disturbing stdout, revert to travis_wait (#1566)
  Show black diff on travis builds (#1565)
  various changes to resolve test timeout/invalid dep issues (#1563)
  try pytest xdist (#1561)
  switch to pytest (#1560)
  Fixes different problems found by lgtm (#1558)
  reorg some deps a bit, usee env markers for dataclasses (#1559)
  fix some deprecation warnings (#1556)
  Add flag to only generate alive states when finalizing Manticore (#1554)
  Arrayvarname (#1552)
  Fix Z3 version parsing (#1551)
  Update README.md (#1553)
  Manticore 0.3.2 (#1547)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants