Produce a single "tests were skipped" for multiple collections not having a timeStamp property

[jerstlouis]

Test the presence of a timeStamp property on all collections that do have it, but report only once that tests were skipped on the n collections that don't have it, so that you don't end up with 960 "test skipped" if you test 1000 collections and only 40 have a timeStamp.

[dstenger]

We discussed this issue in the team and came to the conclusion that it is easier to understand the test results when the skipped tests are present.
Reporting the skipped tests only once will not work because each collection is handled individually. Also, details might get lost if reporting just occurs once.
The only improvement might be to skip the execution of the tests if no timestamp is available (they will not appear in the report anymore) but, as said before, we consider the current solution as more informative for users.

[jerstlouis]

@dstenger at the moment, testing 3 collections produces 27 skipped tests about missing time stamp, 9 for each collections with identical messages (org.testng.SkipException: Property timeStamp is not set in collection items {collectionId}).

Out of these 9:

• 2 for validate Features With Limit Response Time Stamp
• 6 for validate Features With Bounding Box Response Time Stamp
• 1 for validate Features Response Time Stamp

That seems quite excessive -- a single message for each collection would provide the same informative value.

[jerstlouis]

@dstenger I realized today that I was quite confused about this timeStamp warning. The Abstract Test 24 / Requirement 29 is actually not about the GeoJSON "properties" containing one called timeStamp at all, and completely unrelated to the temporal extent or datetime filtering capabilities, but it is the timestamp of when the response was generated by the server.

With that in mind:

• it makes even less sense to generate this warning so many times (the server either decides to return a timestamp response or not, and that aplies to all requests),
• it would be good to clarify the warning to avoid this kind of confusion:

Property timeStamp is not set in collection items 'NaturalEarth:cultural:ne_10m_admin_0_antarctic_claims'

could instead read something like:

No server response timeStamp set for collection items request ('NaturalEarth:cultural:ne_10m_admin_0_antarctic_claims')

[bpross-52n]

@dstenger at the moment, testing 3 collections produces 27 skipped tests about missing time stamp, 9 for each collections with identical messages (org.testng.SkipException: Property timeStamp is not set in collection items {collectionId}).

Out of these 9:

• 2 for validate Features With Limit Response Time Stamp
• 6 for validate Features With Bounding Box Response Time Stamp
• 1 for validate Features Response Time Stamp

That seems quite excessive -- a single message for each collection would provide the same informative value.

@jerstlouis I checked why there are so many tests executed. For example, the validate Features With Bounding Box Response Time Stamp test is executed six times for each collection (see here). We will enhance the text of the skip exceptions with information about the requested bounding box.