Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Odd default ordering of tests when using contexts #1036

Closed
jgebal opened this issue Dec 29, 2019 · 0 comments · Fixed by #1037
Closed

Odd default ordering of tests when using contexts #1036

jgebal opened this issue Dec 29, 2019 · 0 comments · Fixed by #1037
Assignees
Labels
Milestone

Comments

@jgebal
Copy link
Member

jgebal commented Dec 29, 2019

Describe the bug
When a test suite doesn't contain contexts, the default ordering of tests is determined by test procedure position in package specification.

When test suite contains a context, the ordering is no longer directly related to test or context position in test suite.

Provide version info
Affects all versions that support contexts.

To Reproduce
Create two test packages. One with contexts one without.

create or replace package demo_1 as
  --%suite

  --%test(First)
  procedure t1;
  --%test(Second)
  procedure t2;
  --%test(Third)
  procedure t3;
  --%test(Forth)
  procedure t4;
end;
/
create or replace package body demo_1 as
  procedure t1 is begin null; end;
  procedure t2 is begin null; end;
  procedure t3 is begin null; end;
  procedure t4 is begin null; end;
end;
/
create or replace package demo_2 as
  --%suite

  --%test(First)
  procedure t1;

  --%context(Context 1)

    --%test(Second)
    procedure t2;

  --%endcontext
  --%context(Context 2)

    --%test(Third)
    procedure t3;

  --%endcontext

  --%test(Forth)
  procedure t4;
end;
/
create or replace package body demo_2 as
  procedure t1 is begin null; end;
  procedure t2 is begin null; end;
  procedure t3 is begin null; end;
  procedure t4 is begin null; end;
end;
/

Run tests:
call ut.run('demo_1,demo_2');

Results are:

demo_2
  Context 2
    Third [.006 sec]
  Context 1
    Second [.003 sec]
  First [.003 sec]
  Forth [.003 sec]

demo_1
  First [.001 sec]
  Second [.001 sec]
  Third [.001 sec]
  Forth [.001 sec]

Finished in .028732 seconds
8 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

Expected behavior
Run tests:
call ut.run('demo_1,demo_2');

Results are:

demo_2
  First [.003 sec]
  Context 1
    Second [.003 sec]
  Context 2
    Third [.006 sec]
  Forth [.003 sec]

demo_1
  First [.001 sec]
  Second [.001 sec]
  Third [.001 sec]
  Forth [.001 sec]

Finished in .028732 seconds
8 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

Additional context
The ordering of tests is mostly relevant when trying to write tests as documentation for your code, so that the tests tell a story.

@jgebal jgebal added the bug label Dec 29, 2019
@jgebal jgebal added this to the 3.1.10 milestone Dec 29, 2019
jgebal added a commit that referenced this issue Dec 29, 2019
@jgebal jgebal self-assigned this Dec 29, 2019
jgebal added a commit that referenced this issue Jan 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant