Running a preconfigured test suite with ALOpsAppTest seems not to work right when ran after a previous ALOpsAppTest task #520
Comments
|
Hope this all makes sense. Was testing a lot and might have forgotten to add some details or have added too much. |
|
We'll have a look - but since we don't have a repro, we'll only be able to look at the code, and hope we find anything that indicates this behavior.. |
I reckon you have tried to repro it. Could giving you access to this ADO project help you? |
|
The testfilter is only there for the old way of working.. it doesn't work in the +v15 testability. For the rest - I'd really like you to provide a (simple) test-project so I can see what is going on. Do know we are simply using default BC functionality. We assume the DisabledTests.json are uploaded to the test-framework, which makes it re-used the next time as well. How do you conclude that the second test-run is re-populate the suite? Where do you get that from the log? |
|
Will try to get you a "(simple) test-project". In what "format" do you want it, @waldo1001? Could it be a link to a GitHub repo (branch)? |
Clear. |
Here you go: https://github.com/fluxxus-nl/ALOpsIssue520
The project has a App and Test app (folder) and has two pipeline yaml files: Both have two test runs in reverse order (compared to each other). azure-pipelines1.yml gives the expected result, while azure-pipelines2.yml gives the wrong results which this issue is reporting (above).
Simple: the test result tab shows for the first pipeline 9 successful tests (as expected) and the second shows 12 successful ones (as it has reloaded all tests in the preconfigured API test suite as described above). Next that letting the pipeline generate a database backup, downloading and installing it in my docker container shows that also:
Result for azure-pipeline1.yml
Result for azure-pipeline2.yml
|
|
@waldo1001, hope the above gives enough details for you. |
|
Hi @lvanvugt , I did some research. And I can only conclude: everything works as intended - just not as expected ;-). Now - we're obviously using the same (and only) entry point into running tests, which is basically the same as BCCH does: running a page in a client session. So I simply reproduced the steps manually: published the apps to a bc environment, and started the test-tool the same way as you did. What happens, using this tool: THIS is the reason why your suite contains all tests. Next, all the "disabledtests" does is simply again change the records, to skip the tests. THIS is the reason why your second test-run will skip the tests again - because it has changed the data. Now - as this interface to run tests is all we have, and since we're bound to the way it works - I would strongly suggest that you change the way you're running tests, simply by using multiple test-suites. Suggestion: don't use "disabledtests.json", but simply run UT in one step and upload results You'll be amazed how the results will look like. Example: codeunit 80097 "Install API TestSuite"
{
Subtype = Install;
trigger OnInstallAppPerCompany()
begin
SetupTestSuiteAPI();
SetupTestSuiteUT();
end;
local procedure SetupTestSuiteAPI()
var
ALTestSuite: Record "AL Test Suite";
TestSuiteMgt: Codeunit "Test Suite Mgt.";
SuiteName: Code[10];
begin
SuiteName := 'API';
if ALTestSuite.Get(SuiteName) then
ALTestSuite.Delete(true);
TestSuiteMgt.CreateTestSuite(SuiteName);
ALTestSuite.Get(SuiteName);
TestSuiteMgt.SelectTestMethodsByRange(ALTestSuite, StrSubstNo('%1', Codeunit::"LookupValue APIV1"));
TestSuiteMgt.ChangeTestRunner(ALTestSuite, Codeunit::"Test Runner - Isol. Disabled");
end;
local procedure SetupTestSuiteUT()
var
ALTestSuite: Record "AL Test Suite";
TestSuiteMgt: Codeunit "Test Suite Mgt.";
SuiteName: Code[10];
begin
SuiteName := 'UT';
if ALTestSuite.Get(SuiteName) then
ALTestSuite.Delete(true);
TestSuiteMgt.CreateTestSuite(SuiteName);
ALTestSuite.Get(SuiteName);
TestSuiteMgt.SelectTestMethodsByRange(ALTestSuite, StrSubstNo('%1', Codeunit::"LookupValue UT Customer"));
end;
}...
- task: ALOpsAppTest@1
displayName: "ALOPS - Run TestSuite for UT"
inputs:
usedocker: true
import_testtoolkit: false
import_action: "Skip"
testpage: "130455"
testsuite: "UT"
failed_test_action: "Ignore"
show_available_tests: true
continueOnError: true
- task: PublishTestResults@2
displayName: "Publish Test Results **/TestResults.xml"
inputs:
testResultsFormat: XUnit
testResultsFiles: "**/TestResults.xml"
failTaskOnFailedTests: true
testRunTitle: "BC Test Results: $(Build.BuildId)"
- task: ALOpsAppTest@1
displayName: "ALOPS - Run TestSuite for API"
inputs:
usedocker: true
import_testtoolkit: false
import_action: "Skip"
testpage: "130455"
testsuite: "API"
failed_test_action: "Ignore"
show_available_tests: true
continueOnError: true
- task: PublishTestResults@2
displayName: "Publish Test Results **/TestResults.xml"
inputs:
testResultsFormat: XUnit
testResultsFiles: "**/TestResults.xml"
failTaskOnFailedTests: true
testRunTitle: "BC Test Results: $(Build.BuildId)"
...And you'll have a beautiful result in DevOps: HOpe this helps.. . |
waldo1001
added
question
|
I'm closing this issue. |
Issue
Wanting to run my API tests I have added an Install codeunit to my app to setup the test suite API to contain a specific set of test codeunits (actually only one right now, see below codeunit
LookupValue APIV1(81090 )) and have another test runner codeunit assigned to it, i.e.Test Runner - Isol. Disabled(130451).Install Codeunit
ALOpsAppTest tasks
I have added the following tasks to my .yml expecting the 2nd will run that specific test suite.
ALOpsAppTest log
The 2nd does run the test suite API, but before it does it starts adding all test codeunits available in BC as the log shows. It seems that this is related to the 1st test task having
disabledtestsparameter set, because when I remove the 1st test task the 2nd (and now only test task) behaves as expected (see below Expected behavior).Note
Codeunit
LookupValue APIV1(81090 ) is marked as Skipped in the log which is very strange. It is the codeunit that was already added to the API test suite by means of the install codenuit. This led me to think that this is related to thedisabledtestsparameter of the previous test task as this codeunit is in the DisabledTests.json.Expected behavior
When a test suite is referenced in
ALOpsAppTestand it has already been populated (in this case by an install codeunit) it should not re-populated, when a previous tests task has been created (withdisablestestsparameter set).Additional context
I tried an alternative way using the testfilter parameter for
ALOpsAppTest, but then an error is thrown in the log:ALOpsAppTest task
ALOpsAppTest log (only relevant part)
The text was updated successfully, but these errors were encountered: