xUnitXmlPublisher #91

Aliazzzz · 2020-01-25T19:48:59Z

PR for xUnitXmlPublisher with most of the remarks fixed.

TcUnit Verifier

Disabled .prepend method => needs further debugging Disabled unittesting for .prepend and filewriter => needs further debugging

…ls'} and //Test_Prepend();) Recommented headers like (* *) Removed all disabled code Renamed all instances of jUnit to xUnit. Added an explanatory note about "MaxFileSize : UDINT := 65535;" Set the EnablePublish parameter to false default (in 1.1 it will be TcUnit-Runner used default) Used same naming conventions as the rest of TcUnit/TcUnit-Verifier as much as possible Removed pragma {attribute 'hide_all_locals'} as the code is open source Renamed FB_JUnitXmlPublisher into FB_XUnitXmlLogger (implements I_TestResultLogger) Renamed the default file name into "c:/tcunit_xunit_testresults.xml" Started all variables/comments with uppercase Refactored all variables from e.g. iLoop to LoopCount (which is an asset to the code I must say)

Refactored code, Fixed several issues, TcUnit-Verifier: Fixed FileControl Tests Fixed StreamBuffer Tests

sagatowski · 2020-01-25T19:55:07Z

This is the same features as in PR as #90. Not necessary to create another PR for this. I'm copying my comments here for keeping track and "checking off" everything:

First of all, sorry for the long delay of the review of the PR.
Here are my initial comments:

  <ItemGroup>
    <SubObject Include="FB_TestStreamBuffer.Test_Prepend">
      <ExcludeFromBuild>true</ExcludeFromBuild>
    </SubObject>
  </ItemGroup>

Whats this?

And some minor:

Start all variables/comments with uppercase
End comments with (* blabla *) instead of (* blabla*)
Is it possible to "Beautify" the XML-output?

Aliazzzz · 2020-03-24T19:26:55Z

I have a new PR...

sagatowski · 2020-03-24T19:39:27Z

Please, I kindly (again) ask you not to open a new PR everytime a change is made in the code. It is absolutely not necessary. It makes it really hard to follow the flow and changes. Right now I'm required to look into three separate PRs to see all changes. If you merge the changes in, this PR is then automatically updated as they refer to the same branch of code. I'll re-open this one and close the newly created. Please don't create a fourth PR for this change.

Are all the comments above fixed now?
These are the ones that were not fixed the last time I looked:

CfUnitTestResults?
Rename xUnit-params so that it's clear that they relate to the xUnit reporting and not to the TcUnit framework in general.
Settings for the "Real-Time" have been changed in the TcUnit-Verifier (see TcUnit-Verifier_TwinCAT.tsproj), this inhibits the tests on all systems. Leave them at default.
Same as above for TcUnit (see TcUnit/TcUnit.tsproj). Leave this file unchanged.
There are no tests added in the C#/.net verifier. By running the .NET-verifier, how can I know that the xUnit publishing works? I see there are two test FBs (FB_TestStreamBuffer and FB_TestFileControl) added in the test-suites for TwinCAT-verifier, but I don't see any C#-classes added for these two. Maybe you forgot to commit some C# code? With the code today I can make changes in the FB_StreamBuffer or file control, and the .NET-verifier will still say everything is OK.
In TcUnitVerifier.plcproj I noticed the addition:

  <ItemGroup>
    <SubObject Include="FB_TestStreamBuffer.Test_Prepend">
      <ExcludeFromBuild>true</ExcludeFromBuild>
    </SubObject>
  </ItemGroup>

Whats this?

The test Test_BufferSize doesn't seem to test anything. It only does an assertion.
The tests inside FB_TestStreamBuffer seem to depend on each other, which is not a good habit (https://medium.com/better-programming/13-tips-for-writing-useful-unit-tests-ca20706b5368, see point 8). Is it possible to make the tests be independent of each other?
It's not necessary to initiate szFilename to STRING := 'testresults.xml' as this is set in the constructor anyway and is just confusing
I tested the resulting testresults.xml from running the TcUnit-Verifier against the following XSD: https://raw.githubusercontent.com/junit-team/junit5/master/platform-tests/src/test/resources/jenkins-junit.xsd in the following validator https://www.freeformatter.com/xml-validator-xsd.html and it failed.
Noticed that SysDir has been added as reference. Is this available in Tc 4020 as well?

And some minor:

Is it possible to "Beautify" the XML-output?

If everything above is checked & good to go, I'll merge this in the weekend!
Great work @Aliazzzz & @dfreiberger !

sagatowski · 2020-03-24T19:51:22Z

I see that everything has been really well commented in the commits. I'll go thru all the comments against the above during the week, and notify when I'm done if there is anything left to do in the above list.

Aliazzzz · 2020-03-24T20:14:52Z

Edited

you wrote:

Please, I kindly (again) ask you not to open a new PR everytime a change is made in the code. It is absolutely not necessary. It makes it really hard to follow the flow and changes. Right now I'm required to look into three separate PRs to see all changes. If you merge the changes in, this PR is then automatically updated as they refer to the same branch of code. I'll re-open this one and close the newly created. Please don't create a fourth PR for this change.

I have deleted the previous PR's on purpose, It's not that I am doing it to bother you on purpose. Please have patience with me and 'git', as I have trouble understanding the PR mechanisme in regards to the commits I made. I was not aware that new changes could be merged in the current PR. That feature is new for me...

The last request: ** Is it possible to "Beautify" the XML-output?**
=> yes, it is possbile but also fairly useless. I'll explain why;
When the xml file is consumed by a third party like Jenkins, Drone or some other CI/CD software with xUnit capabilities, the contents will be parsed and presented to the user in a human readable manner anyway. Therefore, beautifying the intermediate file will only cost extra time, but doesn't really add anything extra. If the user wants it beautified, there are thousands of online resources to quickly beautify it.

dfreiberger

I have added some comments which were on the other PR (https://github.com/Aliazzzz/TcUnit/pull/1).

I tested the output file against https://raw.githubusercontent.com/junit-team/junit5/master/platform-tests/src/test/resources/jenkins-junit.xsd with https://www.freeformatter.com/xml-validator-xsd.html, and it passed for me.

dfreiberger · 2020-03-25T01:49:42Z

...-Verifier/TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/PRG_TEST.TcPOU

@@ -1,5 +1,5 @@
 <?xml version="1.0" encoding="utf-8"?>
-<TcPlcObject Version="1.1.0.1" ProductVersion="3.1.4022.18">
+<TcPlcObject Version="1.1.0.1" ProductVersion="3.1.4024.3">


I might need to fix this - I am not sure if TwinCAT will allow it to be opened in an older version once it has been opened in a newer version. Please let me know if it causes any issues for you.

Hi! Yes, I've had some issues when trying to open projects created in newer versions in TwinCAT, and that's the reason I've described this in the contributing.md-file (please use 4022). This is possible by installing a 4022 on top of existing 4024 by using the remote manager versions, and selecting them in the remote manager prior to opening the project. I noticed that all new files are created using 4024, which can cause issues.

https://www.beckhoff.com/english.asp?download/tc3-download-remote-manager.htm
(4022.32 is fine). I've made a small description of how to use different versions of TwinCAT here:
https://alltwincat.com/2017/06/29/developing-code-for-different-runtimes/

dfreiberger · 2020-03-25T01:52:32Z

.../TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_TestXmlControl.TcPOU

+                   Actual :=  ActualTag,
+                   Message := 'NextTag does not contain expected content');
+ActualTag := XmlControl.NextTag();
+AssertEquals_String(Expected := Tag1Name, 


@Aliazzzz I am not sure if you saw my question in https://github.com/Aliazzzz/TcUnit/pull/1 -- what is the expected behavior here? This test is currently failing as I wrote it.

This was my question before:

It looks like NextTag is not used anywhere, so maybe we should just remove it. But if it is to be used, then it is returning different tag names depending on whether they are a starting or ending tag, or nested. E.g. in NextTagNested, for xml content of <MyTag1><MyTag2/><MyTag3>Data</MyTag3><MyTag4/></MyTag1> it returns, "MyTag1", "MyTag2/", "MyTag3/", "/MyTag3", "MyTag4/".

Hi,

NextTag etc, were originally conceived as part of a xml parsing capability. You may phase it out if you wish.

I dropped xml parsing because we will only create xml and not consume it locally (it makes no sense). Besides, if someone would like to parse Xml, there is a full functioning opensource alternative available via OSCAT XML Reader in their network library.

Just wanted to chime in that I don't think the OSCAT network library works with TwinCAT3 (only vanilla codesys). I did a conversion of the OSCAT BASIC library in TwinCAT (as described here: https://alltwincat.com/2018/01/01/oscat-basic-library-in-twincat/) but according to the comments in this post and some tries myself I've never managed to get it to work.

Hm, that is strange indeed, but sadly I have no experience with the TwinCAT version.

I will look into the issue to see if it's easily fixable. Of course it would be better to move these out into their own library if we intend them for use beyond TcUnit, so it may be better to just remove the unused methods for now.

I have it in a separate project ;-)
https://forge.codesys.com/lib/xml-pac/home/Home/
On the other hand, you can just let it go...

dfreiberger · 2020-03-25T01:56:41Z

...cUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_TestStreamBuffer.TcPOU

@@ -43,7 +42,9 @@ END_VAR
      <Implementation>
        <ST><![CDATA[TcUnit.TEST('Test_Append');

-Stream.Clear();
+Setup();


To avoid re-using the existing StreamBuffer instance, I create a new one in each test. However I had to keep the Buffer in the main method otherwise I was getting a stack overflow. So I have added a "Setup" routine which MEMSETs the buffer to zero before every test. @sagatowski, do you think this is a good way to do it? Have you thought about adding Setup/Teardown method hooks as part of the framework?

It would be possible also to just use a smaller buffer for the tests since we don't need to test the full buffer, except possibly if we were to add error handling that needs to be tested for a buffer overflow scenario.

I think your solution with MEMSET'ing to zero is fine!
What happens with the system if we get a buffer overflow?

I will check into the buffer overflow scenario this weekend, and try to finish up the other issues we identified.

@dfreiberger Great, thanks!

dfreiberger · 2020-03-25T01:57:51Z

...-Verifier/TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/PRG_TEST.TcPOU

    TestNumberOfAssertionsCalculation : FB_TestNumberOfAssertionsCalculation;
    TestFileControl : FB_TestFileControl;
+    TestXmlControl : FB_TestXmlControl;


I didn't add tests for xUnitXmlPublisher itself. It might make sense to have some tests that confirm that it outputs a correctly formatted file, for example checks the file against the xUnit XSD schema.

Aliazzzz · 2020-03-25T17:34:08Z

It looks like I accidentally introduced some tabs here.

@Aliazzzz can you please make me a contributor to your repository so that I can push changes to fix this? Otherwise we will need another PR into yours. Or let me know if you would like me to create a PR.

Done!

Aliazzzz · 2020-03-25T17:34:46Z

Ooops

sagatowski · 2020-03-25T17:50:17Z

Edited

you wrote:

Please, I kindly (again) ask you not to open a new PR everytime a change is made in the code. It is absolutely not necessary. It makes it really hard to follow the flow and changes. Right now I'm required to look into three separate PRs to see all changes. If you merge the changes in, this PR is then automatically updated as they refer to the same branch of code. I'll re-open this one and close the newly created. Please don't create a fourth PR for this change.

I have deleted the previous PR's on purpose, It's not that I am doing it to bother you on purpose. Please have patience with me and 'git', as I have trouble understanding the PR mechanisme in regards to the commits I made. I was not aware that new changes could be merged in the current PR. That feature is new for me...

The last request: ** Is it possible to "Beautify" the XML-output?**
=> yes, it is possbile but also fairly useless. I'll explain why;
When the xml file is consumed by a third party like Jenkins, Drone or some other CI/CD software with xUnit capabilities, the contents will be parsed and presented to the user in a human readable manner anyway. Therefore, beautifying the intermediate file will only cost extra time, but doesn't really add anything extra. If the user wants it beautified, there are thousands of online resources to quickly beautify it.

The workflow of PRs is fairly well described here:
https://www.atlassian.com/git/tutorials/making-a-pull-request

Basically the PR is a "Container" for the requested change. You are making changes to your branch and the PR holds all the comments/commits together. So basically you do a PR, show your changes, we have a discussion, you (or other people) do more changes, and once everything is fine, the PR is merged into the main and it's closed. This means the PR is not something that is created for every commit, but for a series of commits that are related to the description of the PR. I wouldn't describe it as a "feature" that changes in your code "automatically" go into the PR, it's rather just the nature of it.

I agree on skipping the "Beautifying" of the XML-output, let's skip that one! The C# code for doing this is one line of code, but I guess you seldom get these free stuff with PLCs! :-)

Aliazzzz · 2020-03-25T18:09:08Z

Hi!

Thank your the explanation of "PR".

I agree on skipping the "Beautifying" of the XML-output, let's skip that one! The C# code for doing this is one line of code, but I guess you seldom get these free stuff with PLCs! :-)

You are talking about the method to enable/disable "Beautifying" of the XML-output?
Actually it will be several (but not too many) lines of code spread over various methods or you have to write asmart function with a single point of concern. It involves adding CRLF and TAB into the textbuffer at specific tag closing's.

If you wish you can crack this "Beautifying" nut. To be honest I tried to implement it but gave me loads of headaches and dropped it after several unsuccesfull implementations. I simply gave up because wrong "beautification" will break well formed and valid xml so it won't be parsed :-( That is a big penalty for something "fairly" useless...

sagatowski · 2020-03-26T09:57:13Z

Hi!

Thank your the explanation of "PR".

I agree on skipping the "Beautifying" of the XML-output, let's skip that one! The C# code for doing this is one line of code, but I guess you seldom get these free stuff with PLCs! :-)

You are talking about the method to enable/disable "Beautifying" of the XML-output?
Actually it will be several (but not too many) lines of code spread over various methods or you have to write asmart function with a single point of concern. It involves adding CRLF and TAB into the textbuffer at specific tag closing's.

If you wish you can crack this "Beautifying" nut. To be honest I tried to implement it but gave me loads of headaches and dropped it after several unsuccesfull implementations. I simply gave up because wrong "beautification" will break well formed and valid xml so it won't be parsed :-( That is a big penalty for something "fairly" useless...

@Aliazzzz Fully agree with you, let's skip the beautifying and not add any more complexity. Thanks for your great work!

Aliazzzz · 2020-03-26T10:42:30Z

@Aliazzzz Fully agree with you, let's skip the beautifying and not add any more complexity. Thanks for your great work!

Don't forget to thank @dfreiberger also ;-)
Without him I'd never manage to rectify or solve the Beckhoff specific issues of which I have no or insufficient knowledge :-(

* Fix tabs and comment * Remove "Cf" prefix associated with Codesys library * Align variable naming with library conventions * Rename xUnit-params so that it's clear that they relate to the xUnit reporting and not to the TcUnit framework in general. * Fix param not updated in refactor * Revert change to tsproj file. * Revert change to tsproj realtime settings * Add TestStreamBuffer tests to Verifier * Create independent tests * Add FileControl tests to Verifier * Add FileControl test to project * Added tests for XmlControl to Verifier * Fixed issue in test * Change XML structure to test another scenario * Remove line ids Co-authored-by: David Freiberger <david.freiberger@cellares.com>

…e machine

dfreiberger

I think I have addressed most of the items we discussed so far.

I am still running into an issue - a random subset of tests (usually 10-15) are failing for me when I run this. So far I have tried to increase AdsLogMessageFifoRingBufferSize and play with timing, to no avail. I am investigating where the failure is at.

Example:

2020-03-28 18:24:11 - Asserting results...
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 3444, ACT: 3445, MSG: Not equal UINT 0 times
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 789234475, ACT: 34523327234, MSG: Not equal ULINT 0 times
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 34, ACT: 36, MSG: Not equal USINT 0 times
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports the messages [FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: T#754m15s10ms, ACT: T#694m13s244ms, MSG: Not equal TIME,FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: T#12h34m15s10ms, ACT: T#11h34m13s244ms, MSG: Not equal TIME,] 0 times
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: TOD#15:36:30.123, ACT: TOD#06:21:11.492, MSG: Not equal TIME_OF_DAY 0 times
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_UDINT_Array_DifferInContent', EXP: ARRAY[-4] = 5, ACT: ARRAY[1] = 4, MSG: Arrays differ
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_UINT_Array_DifferInSize', EXP: SIZE = 3, ACT: SIZE = 4, MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_UINT_Array_DifferInContent', EXP: ARRAY[3] = 99, ACT: ARRAY[3] = 12, MSG: Arrays differ
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_ULINT_Array_DifferInSize', EXP: SIZE = 2, ACT: SIZE = 1, MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_ULINT_Array_DifferInContent', EXP: ARRAY[1] = 9400000000000, ACT: ARRAY[1] = 18446744073709551615, MSG: Arrays differ
2020-03-28 18:24:12 - Test suite AnyToUnionValue does not report: FAILED TEST 'PRG_TEST.AnyToUnionValue@Test_STRING', EXP: (Data size = 81), ACT: (Data size = 256), MSG: Values differ
2020-03-28 18:24:12 - Test suite MultipleAssertWithSameParametersInSameCycleWithSameTest reports message FAILED TEST 'PRG_TEST.MultipleAssertWithSameParametersInSameCycleWithSameTest@Assert_SeveralTimes', EXP: 0x12345678, ACT: 0x90ABCDEF, MSG: Values differ 0 times
2020-03-28 18:24:12 - Test suite MultipleAssertWithSameParametersInDifferentCyclesButWithDifferentTests reports message FAILED TEST 'PRG_TEST.MultipleAssertWithSameParametersInDifferentCyclesButWithDifferentTests@Assert_SeveralTimes', EXP: 0x12345678, ACT: 0x90ABCDEF, MSG: Values differ 2 times

A different run:

2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: D#1996-05-06, ACT: D#2019-01-20, MSG: Not equal DATE 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: DT#1996-05-06-15:36:30, ACT: DT#1972-03-29-00:00:00, MSG: Not equal DATE_AND_TIME 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 4444, ACT: 3333, MSG: Not equal DINT 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 0xAAAAAAAA, ACT: 0xBBBBBBBB, MSG: Not equal DWORD 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 30000, ACT: 32000, MSG: Not equal INT 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 50000, ACT: 50001, MSG: Not equal LINT 0 times
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array2d_DifferInSize_D2', EXP: SIZE = [-4..-3,-1..0] (2x2), ACT: SIZE = [1..2,0..2] (2x3), MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array2d_DifferInContent_LBOUND', EXP: ARRAY[-5,-1] = 0.0, ACT: ARRAY[1,0] = 1.0, MSG: Arrays differ
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array2d_DifferInContent_Middle', EXP: ARRAY[-4,0] = 0.0, ACT: ARRAY[2,1] = 1.0, MSG: Arrays differ
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array2d_DifferInContent_UBOUND', EXP: ARRAY[-3,1] = 0.0, ACT: ARRAY[3,2] = 1.0, MSG: Arrays differ
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array3d_DifferInSize_D1', EXP: SIZE = [-5..-4,1..3,-2..-1] (2x3x2), ACT: SIZE = [1..1,4..6,6..7] (1x3x2), MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array3d_DifferInSize_D2', EXP: SIZE = [-5..-4,1..3,-2..-1] (2x3x2), ACT: SIZE = [1..2,4..5,6..7] (2x2x2), MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[-1] = 7.88, ACT: ARRAY[0] = 7.99, MSG: Not equal LREAL 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[1,0] = 6.96, ACT: ARRAY[1,0] = 6.68, MSG: Not equal LREAL2D 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[1,0,1] = 6.0, ACT: ARRAY[1,0,1] = 6.4, MSG: Not equal LREAL3D 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[1] = 0xEDCBA09876543210, ACT: ARRAY[1] = 0x01234567890ABCDE, MSG: Not equal LWORD 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[2] = 2.44001, ACT: ARRAY[2] = 2.44003, MSG: Not equal REAL 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[0] = -128, ACT: ARRAY[0] = 127, MSG: Not equal SINT 0 times

dfreiberger · 2020-03-28T18:41:14Z

TcUnit-Verifier/TcUnit-Verifier_DotNet/TcUnit-Verifier/Program.cs

@@ -31,7 +31,7 @@ static int Main(string[] args)

            OptionSet options = new OptionSet()
                .Add("v=|TcUnitVerifierPath=", "Path to TcUnit-Verifier TwinCAT solution", v => tcUnitVerifierPath = v)
-                .Add("t=|TcUnitTargetAdsNetId=", "Ads Target Net Id", t => tcUnitTargetAdsNetId = t)
+                .Add("t=|TcUnitTargetNetId=", "(Optional, default 127.0.0.1.1.1) Target NetId of TwinCAT runtime to deploy TcUnit-Verifier PLC to", t => tcUnitTargetNetId = t)


I have added this additional parameter to allow downloading of verifier project to alternate target. This was needed for me to develop locally since my laptop doesn't work with the TwinCAT runtime (it has some Kernel security features that cause blue screen if I attempt to run the XAR directly on it). So I deploy to a virtual machine instead with a different Ams NetId.

Good addition, I like it!

dfreiberger · 2020-03-28T18:41:49Z

...cUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_TestStreamBuffer.TcPOU

+
+Cpy := Stream.Copy(StartPos := 1, EndPos := Stream.Length, CopyLen => CopyLen); 
+
+// expected behavior: the buffer receives up to Buffer Size - 1 of string, the last byte is NULL character (termination)


I have added a test to verify the behavior of buffer overflow.

dfreiberger · 2020-03-28T18:42:24Z

TcUnit-Verifier/TcUnit-Verifier_DotNet/TcUnit-Verifier/FB_TestXmlControl.cs

@@ -18,8 +18,6 @@ public FB_TestXmlControl(IEnumerable<ErrorList.Error> errors, string testFunctio
            Test_CloseOpenTag();
            Test_CloseTag();
            Test_NewTagData();
-            Test_NextTagFlat();


I went ahead and removed these methods and associated tests

dfreiberger · 2020-03-28T20:00:51Z

I think I have addressed most of the items we discussed so far.

I am still running into an issue - a random subset of tests (usually 10-15) are failing for me when I run this. So far I have tried to increase AdsLogMessageFifoRingBufferSize and play with timing, to no avail. I am investigating where the failure is at.

Example:

I found that if I increase Thread.Sleep() to a much larger value, such as 10 seconds, then it works successfully with no failures. This seems to point to an issue with accessing the ErrorItems from Visual Studio through the DTE interface.

https://github.com/tcunit/TcUnit/blob/master/TcUnit-Verifier/TcUnit-Verifier_DotNet/TcUnit-Verifier/Program.cs#L111-L116

            while (true)
            {
                Thread.Sleep(10000);


                ErrorItems errorItems = vsInstance.GetErrorItems();
                log.Info("... got " + errorItems.Count + " report lines so far.");

dfreiberger · 2020-04-03T15:14:30Z

@sagatowski have you had a chance to review this further? I am wondering if it is good to merge.

sagatowski · 2020-04-05T07:42:47Z

...rifier/TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/TcUnitVerifier.plcproj

@@ -122,6 +131,12 @@
    <PlaceholderReference Include="TcUnit">
      <DefaultResolution>TcUnit, * (www.tcunit.org)</DefaultResolution>
      <Namespace>TcUnit</Namespace>
+      <Parameters>
+        <Parameter xmlns="">
+          <Key>XUNITENABLEPUBLISH</Key>


I'm not entirely sure what this is? I tried to look for it in the TcUnit-verifier project but couldn't find anything. Is it the usage of the xUnitEnablePublish-parameter of TcUnit in the verifier, but the default value set to true?

Yes - This is overriding the xUnitEnablePublish setting in the TcUnit library. We could delete this since we aren't using the output file in the verifier for anything.

sagatowski · 2020-04-05T11:54:53Z

@sagatowski have you had a chance to review this further? I am wondering if it is good to merge.

@dfreiberger I'm doing the merging now. Will be finished today!

sagatowski · 2020-04-05T17:24:24Z

TcUnit-Verifier/TcUnit-Verifier_DotNet/TcUnit-Verifier/Program.cs

            new FB_TestFinishedNamed(errors);
-            new FB_TestNumberOfAssertionsCalculation(errors);


Why was this removed?

This was accidentally removed - I have added it back and verified that the Verifier tests are passing afterwards. Sorry about this.

sagatowski · 2020-04-05T17:33:48Z

I've left some comments in the code. Also, the TcUnit-verifier (C#/.NET) doesn't work anymore. I only get this message now:

sagatowski · 2020-04-05T19:13:05Z

I think I have addressed most of the items we discussed so far.

I am still running into an issue - a random subset of tests (usually 10-15) are failing for me when I run this. So far I have tried to increase AdsLogMessageFifoRingBufferSize and play with timing, to no avail. I am investigating where the failure is at.

Example:

2020-03-28 18:24:11 - Asserting results...
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 3444, ACT: 3445, MSG: Not equal UINT 0 times
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 789234475, ACT: 34523327234, MSG: Not equal ULINT 0 times
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 34, ACT: 36, MSG: Not equal USINT 0 times
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports the messages [FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: T#754m15s10ms, ACT: T#694m13s244ms, MSG: Not equal TIME,FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: T#12h34m15s10ms, ACT: T#11h34m13s244ms, MSG: Not equal TIME,] 0 times
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: TOD#15:36:30.123, ACT: TOD#06:21:11.492, MSG: Not equal TIME_OF_DAY 0 times
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_UDINT_Array_DifferInContent', EXP: ARRAY[-4] = 5, ACT: ARRAY[1] = 4, MSG: Arrays differ
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_UINT_Array_DifferInSize', EXP: SIZE = 3, ACT: SIZE = 4, MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_UINT_Array_DifferInContent', EXP: ARRAY[3] = 99, ACT: ARRAY[3] = 12, MSG: Arrays differ
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_ULINT_Array_DifferInSize', EXP: SIZE = 2, ACT: SIZE = 1, MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_ULINT_Array_DifferInContent', EXP: ARRAY[1] = 9400000000000, ACT: ARRAY[1] = 18446744073709551615, MSG: Arrays differ
2020-03-28 18:24:12 - Test suite AnyToUnionValue does not report: FAILED TEST 'PRG_TEST.AnyToUnionValue@Test_STRING', EXP: (Data size = 81), ACT: (Data size = 256), MSG: Values differ
2020-03-28 18:24:12 - Test suite MultipleAssertWithSameParametersInSameCycleWithSameTest reports message FAILED TEST 'PRG_TEST.MultipleAssertWithSameParametersInSameCycleWithSameTest@Assert_SeveralTimes', EXP: 0x12345678, ACT: 0x90ABCDEF, MSG: Values differ 0 times
2020-03-28 18:24:12 - Test suite MultipleAssertWithSameParametersInDifferentCyclesButWithDifferentTests reports message FAILED TEST 'PRG_TEST.MultipleAssertWithSameParametersInDifferentCyclesButWithDifferentTests@Assert_SeveralTimes', EXP: 0x12345678, ACT: 0x90ABCDEF, MSG: Values differ 2 times

A different run:

2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: D#1996-05-06, ACT: D#2019-01-20, MSG: Not equal DATE 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: DT#1996-05-06-15:36:30, ACT: DT#1972-03-29-00:00:00, MSG: Not equal DATE_AND_TIME 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 4444, ACT: 3333, MSG: Not equal DINT 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 0xAAAAAAAA, ACT: 0xBBBBBBBB, MSG: Not equal DWORD 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 30000, ACT: 32000, MSG: Not equal INT 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 50000, ACT: 50001, MSG: Not equal LINT 0 times
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array2d_DifferInSize_D2', EXP: SIZE = [-4..-3,-1..0] (2x2), ACT: SIZE = [1..2,0..2] (2x3), MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array2d_DifferInContent_LBOUND', EXP: ARRAY[-5,-1] = 0.0, ACT: ARRAY[1,0] = 1.0, MSG: Arrays differ
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array2d_DifferInContent_Middle', EXP: ARRAY[-4,0] = 0.0, ACT: ARRAY[2,1] = 1.0, MSG: Arrays differ
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array2d_DifferInContent_UBOUND', EXP: ARRAY[-3,1] = 0.0, ACT: ARRAY[3,2] = 1.0, MSG: Arrays differ
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array3d_DifferInSize_D1', EXP: SIZE = [-5..-4,1..3,-2..-1] (2x3x2), ACT: SIZE = [1..1,4..6,6..7] (1x3x2), MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array3d_DifferInSize_D2', EXP: SIZE = [-5..-4,1..3,-2..-1] (2x3x2), ACT: SIZE = [1..2,4..5,6..7] (2x2x2), MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[-1] = 7.88, ACT: ARRAY[0] = 7.99, MSG: Not equal LREAL 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[1,0] = 6.96, ACT: ARRAY[1,0] = 6.68, MSG: Not equal LREAL2D 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[1,0,1] = 6.0, ACT: ARRAY[1,0,1] = 6.4, MSG: Not equal LREAL3D 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[1] = 0xEDCBA09876543210, ACT: ARRAY[1] = 0x01234567890ABCDE, MSG: Not equal LWORD 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[2] = 2.44001, ACT: ARRAY[2] = 2.44003, MSG: Not equal REAL 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[0] = -128, ACT: ARRAY[0] = 127, MSG: Not equal SINT 0 times

Did you manage to figure out what was the problem with this?

…moved

dfreiberger · 2020-04-06T00:44:50Z

I've left some comments in the code. Also, the TcUnit-verifier (C#/.NET) doesn't work anymore. I only get this message now:

@sagatowski I am not sure what is happening with this. I think I will have to create the same environment you are running to try to reproduce (as you specified in the contributor guidelines).

dfreiberger · 2020-04-06T00:47:07Z

I think I have addressed most of the items we discussed so far.
I am still running into an issue - a random subset of tests (usually 10-15) are failing for me when I run this. So far I have tried to increase AdsLogMessageFifoRingBufferSize and play with timing, to no avail. I am investigating where the failure is at.
Example:

2020-03-28 18:24:11 - Asserting results...
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 3444, ACT: 3445, MSG: Not equal UINT 0 times
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 789234475, ACT: 34523327234, MSG: Not equal ULINT 0 times
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 34, ACT: 36, MSG: Not equal USINT 0 times
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports the messages [FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: T#754m15s10ms, ACT: T#694m13s244ms, MSG: Not equal TIME,FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: T#12h34m15s10ms, ACT: T#11h34m13s244ms, MSG: Not equal TIME,] 0 times
2020-03-28 18:24:12 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: TOD#15:36:30.123, ACT: TOD#06:21:11.492, MSG: Not equal TIME_OF_DAY 0 times
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_UDINT_Array_DifferInContent', EXP: ARRAY[-4] = 5, ACT: ARRAY[1] = 4, MSG: Arrays differ
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_UINT_Array_DifferInSize', EXP: SIZE = 3, ACT: SIZE = 4, MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_UINT_Array_DifferInContent', EXP: ARRAY[3] = 99, ACT: ARRAY[3] = 12, MSG: Arrays differ
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_ULINT_Array_DifferInSize', EXP: SIZE = 2, ACT: SIZE = 1, MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:24:12 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_ULINT_Array_DifferInContent', EXP: ARRAY[1] = 9400000000000, ACT: ARRAY[1] = 18446744073709551615, MSG: Arrays differ
2020-03-28 18:24:12 - Test suite AnyToUnionValue does not report: FAILED TEST 'PRG_TEST.AnyToUnionValue@Test_STRING', EXP: (Data size = 81), ACT: (Data size = 256), MSG: Values differ
2020-03-28 18:24:12 - Test suite MultipleAssertWithSameParametersInSameCycleWithSameTest reports message FAILED TEST 'PRG_TEST.MultipleAssertWithSameParametersInSameCycleWithSameTest@Assert_SeveralTimes', EXP: 0x12345678, ACT: 0x90ABCDEF, MSG: Values differ 0 times
2020-03-28 18:24:12 - Test suite MultipleAssertWithSameParametersInDifferentCyclesButWithDifferentTests reports message FAILED TEST 'PRG_TEST.MultipleAssertWithSameParametersInDifferentCyclesButWithDifferentTests@Assert_SeveralTimes', EXP: 0x12345678, ACT: 0x90ABCDEF, MSG: Values differ 2 times

A different run:

2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: D#1996-05-06, ACT: D#2019-01-20, MSG: Not equal DATE 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: DT#1996-05-06-15:36:30, ACT: DT#1972-03-29-00:00:00, MSG: Not equal DATE_AND_TIME 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 4444, ACT: 3333, MSG: Not equal DINT 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 0xAAAAAAAA, ACT: 0xBBBBBBBB, MSG: Not equal DWORD 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 30000, ACT: 32000, MSG: Not equal INT 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwice reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwice@TwiceAssertCall', EXP: 50000, ACT: 50001, MSG: Not equal LINT 0 times
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array2d_DifferInSize_D2', EXP: SIZE = [-4..-3,-1..0] (2x2), ACT: SIZE = [1..2,0..2] (2x3), MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array2d_DifferInContent_LBOUND', EXP: ARRAY[-5,-1] = 0.0, ACT: ARRAY[1,0] = 1.0, MSG: Arrays differ
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array2d_DifferInContent_Middle', EXP: ARRAY[-4,0] = 0.0, ACT: ARRAY[2,1] = 1.0, MSG: Arrays differ
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array2d_DifferInContent_UBOUND', EXP: ARRAY[-3,1] = 0.0, ACT: ARRAY[3,2] = 1.0, MSG: Arrays differ
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array3d_DifferInSize_D1', EXP: SIZE = [-5..-4,1..3,-2..-1] (2x3x2), ACT: SIZE = [1..1,4..6,6..7] (1x3x2), MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:19:04 - Test suite ArrayPrimitiveTypes does not report: FAILED TEST 'PRG_TEST.ArrayPrimitiveTypes@Test_REAL_Array3d_DifferInSize_D2', EXP: SIZE = [-5..-4,1..3,-2..-1] (2x3x2), ACT: SIZE = [1..2,4..5,6..7] (2x2x2), MSG: Arrays differ, size of arrays not matching.
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[-1] = 7.88, ACT: ARRAY[0] = 7.99, MSG: Not equal LREAL 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[1,0] = 6.96, ACT: ARRAY[1,0] = 6.68, MSG: Not equal LREAL2D 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[1,0,1] = 6.0, ACT: ARRAY[1,0,1] = 6.4, MSG: Not equal LREAL3D 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[1] = 0xEDCBA09876543210, ACT: ARRAY[1] = 0x01234567890ABCDE, MSG: Not equal LWORD 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[2] = 2.44001, ACT: ARRAY[2] = 2.44003, MSG: Not equal REAL 0 times
2020-03-28 18:19:04 - Test suite AssertEveryFailedTestTwiceArrayVersion reports message FAILED TEST 'PRG_TEST.AssertEveryFailedTestTwiceArrayVersion@TwiceAssertCall_Arrays', EXP: ARRAY[0] = -128, ACT: ARRAY[0] = 127, MSG: Not equal SINT 0 times

Did you manage to figure out what was the problem with this?

@sagatowski I was only able to get this to work by increasing the Thread.Sleep time to a large value (15 seconds). I didn't check this change in as I assume it is due to me running the code in a VM or somehow related to my environment.

https://github.com/tcunit/TcUnit/blob/master/TcUnit-Verifier/TcUnit-Verifier_DotNet/TcUnit-Verifier/Program.cs#L111-L116

            while (true)
            {
                Thread.Sleep(15000);

                ErrorItems errorItems = vsInstance.GetErrorItems();
                log.Info("... got " + errorItems.Count + " report lines so far.");

                var newErrors = errorList.AddNew(errorItems);

dfreiberger · 2020-04-06T02:36:22Z

I've left some comments in the code. Also, the TcUnit-verifier (C#/.NET) doesn't work anymore. I only get this message now:

@sagatowski I am not sure what is happening with this. I think I will have to create the same environment you are running to try to reproduce (as you specified in the contributor guidelines).

@sagatowski I reproduced the same error as you did on a fresh VS2013/TC3.1.4022 install. After clicking "Ignore" when this error pops up, the Verifier program functions normally and I don't see the error again on subsequent builds even after cloning the repo again. I am not sure what changed, no tracked files changed after I clicked "Ignore".

Detailed steps to reproduce:

Created new Windows 10 VM
Installed VS 2013 Professional
Cloned this repo
Opened TcUnit.sln, Save as Library and Install
Opened TcUnit-Verifier_TwinCAT, set the REALTIME settings to support my VM
Opened TcUnit-Verifier_DotNet, built the project
Ran TcUnit-Verifier.exe
C:\Users\twincat\Documents\TcUnit\TcUnit-Verifier\TcUnit-Verifier_DotNet\TcUnit-Verifier\bin\Debug>TcUnit-Verifier.exe -v "C:\Users\twincat\Documents\TcUnit\TcUnit-Verifier\TcUnit-Verifier_TwinCAT\TcUnit-Verifier_TwinCAT.sln"
Received the same error that you did just after Generating TcUnit-Verifier_TwinCAT boot project...
I tried a couple of times clicking Abort, got the same error. I was able to activate and run just fine using Visual Studio.
Then I tried clicking Ignore. Subsequently I did not get the error again, and everything worked normally.

The git diff before and after the error went away did not change, so no tracked files changed.

$ git diff
diff --git a/TcUnit-Verifier/TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT.tsproj b/TcUnit-Verifier/TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT.tsproj
index 53a1e53..3f37070 100644
--- a/TcUnit-Verifier/TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT.tsproj
+++ b/TcUnit-Verifier/TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT.tsproj
@@ -1,7 +1,10 @@
 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
-<TcSmProject xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.beckhoff.com/schemas/2012/07/TcSmProject" TcSmVersion="1.0" TcVersion="3.1.4022.30">
+<TcSmProject xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.beckhoff.com/schemas/2012/07/TcSmProject" TcSmVersion="1.0" TcVersion="3.1.4022.22">
        <Project ProjectGUID="{3B151CEE-1DB6-4543-9B44-7AFACBDFB147}" TargetNetId="127.0.0.1.1.1" Target64Bit="true" ShowHideConfigurations="#x3c7">
                <System>
+                       <Settings MaxCpus="4" NonWinCpus="1">
+                               <Cpu CpuId="3"/>
+                       </Settings>
                        <Tasks>
                                <Task Id="3" Priority="20" CycleTime="100000" AmsPort="350" AdtTasks="true">
                                        <Name>PlcTask</Name>

- Increased Thread.Sleep time between checking the visual studio error list from 1s to 10s (would however be better to investigate why 1s fails). See PR #91 for discussion. - Updated verifier readme with new example of how it looks like when running the .NET-verifier - Updated formatting of text to be in par with the rest of the code - Removed FB header description of FB_XmlControl as this was no longer correct - Changed the parameter XUNITENABLEPUBLISH (From TRUE -> FALSE) in the TcUnit-Verifier_TwinCAT project

sagatowski · 2020-04-06T07:08:11Z

I've just merged the code into the main. See commit-comments of what additional changes I did.

@dfreiberger Thanks for the excellent descriptions of everything you've done! As you say, once I clicked ignore I never saw the error message again.

Out of curiosity, what was your development environment when you did NOT receive this message?

Again @dfreiberger and @Aliazzzz , thanks for this contribution!

Aliazzzz · 2020-04-06T07:26:12Z

No problem, plus many thanks to @dfreiberger

Aliazzzz · 2020-04-07T08:15:37Z

@sagatowski,
Are you going to close PR #91? I guess that most of the merges you made are the latest version allready.

dfreiberger · 2020-04-07T15:38:36Z

I've just merged the code into the main. See commit-comments of what additional changes I did.

@dfreiberger Thanks for the excellent descriptions of everything you've done! As you say, once I clicked ignore I never saw the error message again.

Out of curiosity, what was your development environment when you did NOT receive this message?

Again @dfreiberger and @Aliazzzz , thanks for this contribution!

@sagatowski awesome! Thanks for looking at this so quickly. Also thank you @Aliazzzz for creating this feature, I am excited to use it!

Regarding the development environment where I did not receive the message: As far as I remember I haven't received it in VS 2017 with TwinCAT 3.1.4024.x, targeting to a local VM or to a remote VM. I have a few different VMs I have been using as the target, and I don't think I have seen this error for any of them with 4024.

Aliazzzz · 2020-04-07T16:20:16Z

No problem! I hope many people will find this feature as useful as you do 👍

sagatowski · 2020-04-10T19:33:25Z

For whatever it's worth, the annoying

I've left some comments in the code. Also, the TcUnit-verifier (C#/.NET) doesn't work anymore. I only get this message now:

@sagatowski I am not sure what is happening with this. I think I will have to create the same environment you are running to try to reproduce (as you specified in the contributor guidelines).

For whatever it's worth, I created the issue #98 for this, and with a simple Thread.Sleep() solved this issue. Never get this window again.

Aliazzzz added 7 commits January 25, 2020 20:26

TcUnit JUnitXmlPublisher

fb41e76

TcUnit Verifier

Quick fixes to enable the xml writer output => now works

c3776fd

Disabled .prepend method => needs further debugging Disabled unittesting for .prepend and filewriter => needs further debugging

updated filepath

6f7814f

TcUnit:

83c34fd

Refactored code, Fixed several issues, TcUnit-Verifier: Fixed FileControl Tests Fixed StreamBuffer Tests

LineIDs.dbg had to sacrificed...

b17aa69

restored FB_xUnitXmlPublisher.TcPOU

9f5e885

sagatowski mentioned this pull request Mar 22, 2020

junitxmlpublisher fix #90

Closed

Aliazzzz closed this Mar 24, 2020

sagatowski mentioned this pull request Mar 24, 2020

Here you go! #96

Closed

sagatowski reopened this Mar 24, 2020

dfreiberger reviewed Mar 25, 2020

View reviewed changes

Aliazzzz closed this Mar 25, 2020

Aliazzzz reopened this Mar 25, 2020

dfreiberger and others added 5 commits March 28, 2020 11:05

Fix tabs

f7ca5d6

Fix reference to FB_TestXmlControl

bf11471

Add AdsTargetNetID as a parameter so that tests can be ran on a remot…

f2b5070

…e machine

Remove NextTag and NextParameter methods as they are not used in TcUnit

8db6e59

dfreiberger added 4 commits March 28, 2020 11:05

Replace tabs with spaces

1bcfe74

Renamed XUnit to xUnit to match the name of the standard

6fcba60

Add test to verify buffer overflow behavior

8152734

Better describe Target NetId parameter

52a7bc1

dfreiberger reviewed Mar 28, 2020

View reviewed changes

sagatowski reviewed Apr 5, 2020

View reviewed changes

Re-add FB_TestNumberOfAssertionsCalculation which was accidentally re…

a95fe8a

…moved

This was referenced Apr 6, 2020

Integration to Jenkins/build automation server #7

Closed

Export results in Xunit XML format #11

Closed

sagatowski closed this Apr 8, 2020


		Cpy := Stream.Copy(StartPos := 1, EndPos := Stream.Length, CopyLen => CopyLen);

		// expected behavior: the buffer receives up to Buffer Size - 1 of string, the last byte is NULL character (termination)

		new FB_TestFinishedNamed(errors);
		new FB_TestNumberOfAssertionsCalculation(errors);

xUnitXmlPublisher #91

xUnitXmlPublisher #91

Conversation

Aliazzzz commented Jan 25, 2020

sagatowski commented Jan 25, 2020 • edited

Aliazzzz commented Mar 24, 2020

sagatowski commented Mar 24, 2020

sagatowski commented Mar 24, 2020

Aliazzzz commented Mar 24, 2020 • edited

dfreiberger left a comment

Choose a reason for hiding this comment

dfreiberger Mar 25, 2020 • edited

Choose a reason for hiding this comment

sagatowski Mar 25, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Aliazzzz Mar 27, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Aliazzzz commented Mar 25, 2020

Aliazzzz commented Mar 25, 2020

sagatowski commented Mar 25, 2020

Aliazzzz commented Mar 25, 2020 • edited

sagatowski commented Mar 26, 2020

Aliazzzz commented Mar 26, 2020

dfreiberger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dfreiberger commented Mar 28, 2020

dfreiberger commented Apr 3, 2020

sagatowski Apr 5, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sagatowski commented Apr 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sagatowski commented Apr 5, 2020

sagatowski commented Apr 5, 2020

dfreiberger commented Apr 6, 2020

dfreiberger commented Apr 6, 2020

dfreiberger commented Apr 6, 2020

sagatowski commented Apr 6, 2020

Aliazzzz commented Apr 6, 2020

Aliazzzz commented Apr 7, 2020

dfreiberger commented Apr 7, 2020

Aliazzzz commented Apr 7, 2020

sagatowski commented Apr 10, 2020

sagatowski commented Jan 25, 2020 •

edited

Aliazzzz commented Mar 24, 2020 •

edited

dfreiberger Mar 25, 2020 •

edited

sagatowski Mar 25, 2020 •

edited

Aliazzzz Mar 27, 2020 •

edited

Aliazzzz commented Mar 25, 2020 •

edited

sagatowski Apr 5, 2020 •

edited