New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for reliable incremental test execution #61

Open
Duhemm opened this Issue Nov 24, 2017 · 6 comments

Comments

Projects
None yet
5 participants
@Duhemm
Member

Duhemm commented Nov 24, 2017

I never use it so I completely forgot about it.

Do people actually use it often? If so, we should probably support it since the idea is nice.

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Nov 24, 2017

Member

I've used it in scalameta where we have one project with dozens of test suites. It works great in some cases (for example while editing on test suite) but sometimes not so much.

One somewhat related feature which I find seriously missing in sbt (it works in bazel) is the ability to infer which tests to run from a list of files that have changed. For large projects sbt test can take a long time to run and a lot of the time you only change a couple files in one or two leaf modules. Of course, if your tests rely on escaping the zinc analysis graph then you need to handle this manually, but for most projects I think something fairly basic can get you far.

Member

olafurpg commented Nov 24, 2017

I've used it in scalameta where we have one project with dozens of test suites. It works great in some cases (for example while editing on test suite) but sometimes not so much.

One somewhat related feature which I find seriously missing in sbt (it works in bazel) is the ability to infer which tests to run from a list of files that have changed. For large projects sbt test can take a long time to run and a lot of the time you only change a couple files in one or two leaf modules. Of course, if your tests rely on escaping the zinc analysis graph then you need to handle this manually, but for most projects I think something fairly basic can get you far.

@jvican

This comment has been minimized.

Show comment
Hide comment
@jvican

jvican Nov 24, 2017

Member

Yeah, that's something I want so badly. I'd like to implement it, I know how to get it working. I'm assigning this to me, when Mr. @Duhemm is done with the test support I'll get it done.

Member

jvican commented Nov 24, 2017

Yeah, that's something I want so badly. I'd like to implement it, I know how to get it working. I'm assigning this to me, when Mr. @Duhemm is done with the test support I'll get it done.

@jvican

This comment has been minimized.

Show comment
Hide comment
@jvican

jvican Nov 24, 2017

Member

Upstream ticket I opened some months ago: #56!

Member

jvican commented Nov 24, 2017

Upstream ticket I opened some months ago: #56!

@ShaneDelmore

This comment has been minimized.

Show comment
Hide comment
@ShaneDelmore

ShaneDelmore Nov 30, 2017

I use test-only more often but I want to use test-quick, it just doesn't have great change detection.
In my perfect world I want test-quick to run tests in roughly this order:

  1. Tests that failed last run
  2. New tests
  3. Changed tests
  4. Tests impacted by changes since last run

Additionally in my beautiful perfect world I then want test to run with the above order, followed by
5. Test suite last time taken to run (in the case parallel testing is enabled, start with slow suites)

ShaneDelmore commented Nov 30, 2017

I use test-only more often but I want to use test-quick, it just doesn't have great change detection.
In my perfect world I want test-quick to run tests in roughly this order:

  1. Tests that failed last run
  2. New tests
  3. Changed tests
  4. Tests impacted by changes since last run

Additionally in my beautiful perfect world I then want test to run with the above order, followed by
5. Test suite last time taken to run (in the case parallel testing is enabled, start with slow suites)

@jvican

This comment has been minimized.

Show comment
Hide comment
@jvican

jvican Mar 8, 2018

Member

I use test-only more often but I want to use test-quick, it just doesn't have great change detection.

I think this is an artifact of sbt's implementation. I believe we could make it work better, following a similar strategy to the one you propose, but that goes against one of my short-term goals in Zinc: remove the API representation altogether and not serialize it in the analysis file.

Member

jvican commented Mar 8, 2018

I use test-only more often but I want to use test-quick, it just doesn't have great change detection.

I think this is an artifact of sbt's implementation. I believe we could make it work better, following a similar strategy to the one you propose, but that goes against one of my short-term goals in Zinc: remove the API representation altogether and not serialize it in the analysis file.

@jvican jvican changed the title from Support for `testQuick` to Add support for incremental test execution Mar 12, 2018

@jvican jvican changed the title from Add support for incremental test execution to Add support for reliable incremental test execution Mar 12, 2018

@dzamlo

This comment has been minimized.

Show comment
Hide comment
@dzamlo

dzamlo Aug 1, 2018

I recently stated using testQuick with sbt. For me, it seems to works correctly.

This is really a time saver. It would be really nice to have that supported in bloop. Especially in combinations with the -w flags. So you can do some modification and have a very quick feedback on whether the tests pass. This would improve the "red-green-refactor" cycle.

dzamlo commented Aug 1, 2018

I recently stated using testQuick with sbt. For me, it seems to works correctly.

This is really a time saver. It would be really nice to have that supported in bloop. Especially in combinations with the -w flags. So you can do some modification and have a very quick feedback on whether the tests pass. This would improve the "red-green-refactor" cycle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment