Implement test steps API

[dsherret]

We're going to start work on a prototype of the test steps API.

• Implement proposal without metadata as unstable RFC: subtests, hooks, and parallel tests via `Deno.Tester` API #10771
• Implement proposal with metadata
• Write polyfill for declarative in CLI RFC: Test Collections—Declarative approach to subtests #11235
• Decide on declarative living in CLI, std, or /x
• Test implementation with mocha polyfill https://gist.github.com/lucacasonato/54c03bb267074aaa9b32415dbfb25522

[sebastienfilion]

The RFC you link (#10771) is a long conversation with a lot of contracting concepts. Which one exactly do you propose to implement?
Also, can you define what "metadata" are you referring to?

[dsherret]

What's outlined in the main post. See stepMetadata property.

[sebastienfilion]

Are you going to implement the Tester class? Otherwise, I don't understand to what end.
Also this post is confusing; are you just saying that we're going to implement everything and see what sticks?

[dsherret]

We're implementing what's outlined in the RFC at #10771 (so yes, because the Tester class is outlined there). Then we're going to implement a pollyfill for #11235 that uses the implementation of #10771. After we'll make further decisions on where the #11235 approach will live (it might end up being avialble in the CLI, std, or even in deno.land/x. In the case of it being in the CLI then we'll drop the stepMetadata property, but all of this isn't decided yet). We're going to make these decisions before merging the end result.

[caspervonb]

After we'll make further decisions on where the #11235 approach will live (it might end up being avialble in the CLI, std, or even in deno.land/x. In the case of it being in the CLI then we'll drop the stepMetadata property, but all of this isn't decided yet)

This is fairly confusing.
So we're doing declarative test registration that runs imperative tests that then can be used to registers declarative tests again?

[dsherret]

So we're doing declarative test registration that runs imperative tests that then can be used to registers declarative tests again?

This is a temporary implementation detail. No decision has been made in this regard (as stated above, we will make a decision on this later).

[caspervonb]

This is a temporary implementation detail

Okay sure but it's a big one.

The style of the API doesn't really matter at all, but when the tests are registered and executed does since we have already established a test runner with deferred execution.

e.g, in e.g in https://gist.github.com/caspervonb/62db35a0cc9532dc618b96fcda205576 the convenience function provides you with an imperative style but it is still deferred execution with the same semantics as top level tests.
It's quite literally just a nested test harness.

Immediately executing user controlled tests still fundamentally clash with how regular top level tests actually work with deferred runner controlled and throttled execution so I'm very confused as to how we are planning on making deferred test execution live in deno.land/std or deno.land/x.

Are we throwing out the test runner completely now and going immediate execution all the way? This seems to be really at odds with what I've been building out for the past few months.

We can do immediate steps with reporting and that can live anywhere but we can't build TestDefinitions on immediate execution without fundamentally changing how the test runner works. If the test runner isn't in charge of scheduling and executing the tests why is it in the test runner? Without that we're just calling functions.

Both proposals leave many questions open, including but not limited to:

• How does concurrent_jobs tie into all of this and how do you disable concurrency for debugging a failed test?
• How do you run setup and teardown for top level tests?
• How do you run top level tests concurrently?

[caspervonb]

Just to simplify and clarify my main concern which is the ability to control execution in the test runner.
Are the execution of subtests in this metadata thing deferred and controlled by the test runner or are they immediate and executed by user?

[lucacasonato]

• The step metadata only affects filtering (and maybe display). It can not be used to provide things like permissions.
• Metadata can only be present on top level Deno.test registrations.
• Subtests are invoked when t.step is called. The returned future resolves when the test is complete. We can delay the actual invocation after calling t.step if necessary for whatever reason, because of the returned Promise.

[caspervonb]

Thanks @lucacasonato, so this is still just test steps that's slightly less confusing.

Altho still see no practical use for metadata and don't understand it's purpose as we can filter just fine without it.
As I see no use for it nor any strong arguments for having it, I am against including it.

---

Write polyfill for declarative in CLI RFC: Test Collections—Declarative approach to subtests #11235
Decide on declarative living in CLI, std, or /x

This bit is still confusing tho as one just cannot have tightly integrated deferred execution in user land.
Test steps and say #11789 can co-exist just fine but #11789 can't be built on test steps in deno.land/std or deno.land/x.

[bartlomieju]

@dsherret can this issue be closed now?

[dsherret]

@bartlomieju yeah, we can decide on next steps later.