Update tests.csv to override priority in assertions column

[howard-e]

Preview Tests

See #1039.

This PR does the following:

• Update tests.csv#assertions column regex
• Supports 0 priority assertions when tests.csv is being parsed

[mcking65]

@howard-e thank you for the fast work on this!! Overall, it is working beautifully with one exception: the sequence of assertions in the output is not correct. I added a comment below.

To test, I merged this branch into the branch for PR 1040 and ran the build.

Compare the sequence of tests in:
preview of radio test plan in PR 1040

With the sequence of tests in:
Preview of radio test plan in PR 1015

The sequence in PR 1015 is correct, and it should be the same in PR1040.

[mcking65]

This is not accurate. The requirement is still that assertions are sorted in the order they appear in the assertions cell of tests.csv. I noticed that the order is being reversed by this code and then the assertions derived from commands.csv are appended. The reason we would put a 0 priority assertion in tests.csv at all is to ensure it is placed in the correct sequence. Example: if the assertions column of tests.csv includes "assertion1 0:assertion2 assertion3" should yield the order assertion1, assertion2, assertion3 for commands that include assertion2 and assertion1 assertion3 for commands that do not.

[howard-e]

Okay! I misinterpreted how "append" was being used in the original issue text. Removing that block of code should maintain that behavior, thanks for sharing that example.

Done in 096cc14.

[mcking65]

@howard-e Thinking some more about the feedback I provided yesterday, I think I understand the confusion around assertion order. In #1039, I wrote:

One might ask why is it necessary to specify the assertion in tests.csv at all. Why not simply add it where desired in the command.csv files. The answer is that the order of assertions in the presentation of assertions for a command is determined by the order of the assertionIDS in the assertions column of tests.csv. We could avoid the need to include the assertion in tests.csv in scenarios where it is desired for the exceptional additional assertion to be the last assertion in the presentation of the assertions for a command. That is, we could specify that if an assertion is not included in tests.csv but is included in commands.csv for a given test, then that assertion is appended to the list generated from tests.csv. If such logic is not too opaque, that would be a reasonable way to further simplify the csv files in some scenarios.

By that, I did not mean that the assertions listed in the assertionExceptions column in commands.csv would always be presented last.

Here is a more clear way of stating the requirement as described in that paragraph:

1. The presentation order of assertions in a test is determined by the order of assertion IDs in the assertions column of the row of tests.csv that specifies the test.
2. If a row in commands.csv includes one or more assertionId values in the assertionExceptions column, any assertionIds with a non-0 priority that are not specified in the corresponding assertions cell in tests.csv are appended to the end of the list of assertions specified in tests.csv for the corresponding test of the corresponding AT.
   Example 1:

• for test1, the assertions column of tests.csv includes "a1 a2 a3"
• For AT1, a row for test1 and command1 in at1-commands.csv specifies assertionExceptions of "0:a2 2:a4"
• The test plan for AT1/test1/command1 presents assertions in the order: a1, a3, a4

In example 1:

• a2 is omitted from the test of command1 because its priority is set to 0 in at1-commands.csv
• a4 is the last assertion in the test of command1 because it is not listed in tests.csv.

Example 2:

• for test1, the assertions column of tests.csv includes "a1 a2 0:a3 a4"
• For AT1, a row for test1 and command2 in at1-commands.csv specifies assertionExceptions of "1:a5 1:a6"
• The test plan for AT1/test1/command2 presents assertions in the order: a1, a2, a4, a5, a6
• For AT1, a row for test1 and command3 in at1-commands.csv specifies assertionExceptions of "0:a2 1:a3 1:a5"
• The test plan for AT1/test1/command3 presents assertions in the order: a1, a3, a4, a5"

In example2:

• a3 is omitted from the test of command2 because its priority is set to 0 in tests.csv and is not overridden in the command2 row in at1-commands.csv.
• a5 and a6 are appended to the end of the test of command2 because they are not specified in tests.csv.
• a2 is not included in the test of command3 because it is set to 0 priority in at1-commands.csv.
• a3 is included as the 2nd assertion for command3 because its priority is overridden in at1-commands.csv and because of its position in the string of assertions in tests.csv.

Another useful simplification would be to be able to leave the priorities off of assertionExceptions that are non-0 , e.g., a5 and a6, when specified in at1-commands.csv. In that case, their priority could be derived from assertions.csv.

[howard-e]

Example 1:

• for test1, the assertions column of tests.csv includes "a1 a2 a3"
• For AT1, a row for test1 and command1 in at1-commands.csv specifies assertionExceptions of "0:a2 2:a4"
• The test plan for AT1/test1/command1 presents assertions in the order: a1, a3, a4

In example 1:

• a2 is omitted from the test of command1 because its priority is set to 0 in at1-commands.csv
• a4 is the last assertion in the test of command1 because it is not listed in tests.csv.

Following up from yesterday's CG meeting that in the examples, there is a feature being requested here that doesn't exist and could be discussed outside the context of this PR. That feature being to include a new assertion for a test through commands.csv > assertionExceptions that didn't originally exist in tests.csv > assertions

[alflennik]

I was able to verify that this is working. I used 0 priority assertions via the tests.csv to hide an assertion for a particular test, as well as using the at-commands.csv files to hide the assertion from one of the ATs.

The code looks good, nothing much to add there.

Side note, thanks Howard for taking the time to walk me through this! It's helped me grok how the test generation works.

[mcking65]

working beautifully!!! Love it!!!