Test whether a recordset is a subset of an asserted recordset

[rvdpluijm]

We're currently testing our financial data warehouse using NBI. An essential part of this is testing whether the ETL handles incrementals in a correct manner. Therefore, I have to check whether all records from the staging tables have been transferred to the data warehouse fact tables

Is it therefore possible to use an "contains" or "is contained by" assertion to check whether the checked recordset is a proper subset / superset of the asserted recordset (or could that feature be implemented)?

[FuegoArtificial]

Hi rvdpluijm,

just as a possible point of start, perhaps the assertion subsetof could work. I'd give it a try. although it's just documented for a list of items, it might work for a resultset?
another approach would be if you take a query result comparison where one query returns the DWH-fact table values and the assertion returns a result after an inner join between the fact table and the staging values. it might not be enough for your scenario, yet. However, building two queries will at some point achieve your goal.

my two cents :)

[rvdpluijm]

@FuegoArtificial Sadly, that didn't work. It conflicted with the XSD:

Validation of current file using XML schema:

ERROR: Element '{http://NBi/TestSuite}query': This element is not expected. Expected is one of ( {http://NBi/TestSuite}item, {http://NBi/TestSuite}predefined, {http://NBi/TestSuite}range-integer, {http://NBi/TestSuite}range-date, {http://NBi/TestSuite}range-integer-pattern, {http://NBi/TestSuite}one-column-query, {http://NBi/TestSuite}members ).
ERROR: Element '{http://NBi/TestSuite}query': This element is not expected. Expected is one of ( {http://NBi/TestSuite}item, {http://NBi/TestSuite}predefined, {http://NBi/TestSuite}range-integer, {http://NBi/TestSuite}range-date, {http://NBi/TestSuite}range-integer-pattern, {http://NBi/TestSuite}one-column-query, {http://NBi/TestSuite}members ).

I'll check whether I can build an approach using your guidance. Thanks!

[Seddryck]

@rvdpluijm It's not supported at the moment but shouldn't be too difficult to implement. Will check.

[Seddryck]

It will be too complex to have the same names for constraints on collection (equivalentTo, subsetOf and contains) and constraints on result-sets (queries - currently only equalTo). If I was attempting to follow the initial idea (having the same names for both), the XML definition would be a nightmare to parse and code.

I'm planning to

• continue to support subsetOf in versions 1.X for collection but start to recommend contained-in.
• for result-sets (queries), I'm planning to use subset-of and superset-of.

@FuegoArtificial @zakwillis @BenCurvers @rvdpluijm
Your comments and other suggestions are welcome!

If validated I should update #288

[FuegoArtificial]

I like the idea to distinguish the naming if it would be too complex to implement otherwise.

Though, while thinking in "sets" when using semantic models like elements of a dimension and writing MDX-queries (SET x AS {x,y}), I would stick with the wording "subset" when being in this semantic model domain like OLAP Cubes.
Synonyms could be "containedIn" or "includedIn" as wikipedia suggests similarily in the definition of subsets: https://en.wikipedia.org/wiki/Subset
This would have the advantage of backward compatibility as well while you wouldn't need to change predicates for the current semantic.
What do you think?

[zakwillis]

Hi there, I think subset/superset sounds good and it is always good to use single responsibilities. You obviously have the exists clause both mdx and sql. I don't think this matters that much as long as we can take advantage of this functionality as it feels like a natural inclusion to a great product. Best, Zak

…

Sent from my Windows Phone

________________________________ From: Tilo Sommerwerk<mailto:notifications@github.com> Sent: ‎28/‎08/‎2017 20:03 To: Seddryck/NBi<mailto:NBi@noreply.github.com> Cc: zakwillis<mailto:zak_willis@hotmail.com>; Mention<mailto:mention@noreply.github.com> Subject: Re: [Seddryck/NBi] Test whether a recordset is a subset of an asserted recordset (#196) I like the idea to distinguish the naming if it would be too complex to implement otherwise. Though, while thinking in "sets" when using semantic models like elements of a dimension and writing MDX-queries (SET x AS {x,y}), I would stick with the wording "subset" when being in this semantic model domain like OLAP Cubes. Synonyms could be "containedIn" or "includedIn" as wikipedia suggests similarily in the definition of subsets: https://en.wikipedia.org/wiki/Subset This would have the advantage of backward compatibility as well while you wouldn't need to change predicates for the current semantic. What do you think? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#196 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALol8GMAQYQxKdS42c6cbGCukp7QhGyTks5scw8WgaJpZM4I2PyW>.

[Seddryck]

It's available in a preview of release 1.16: https://ci.appveyor.com/project/Seddryck/nbi/build/1.16.0-ci.1

[Seddryck]

documentation is available at http://www.nbi.io/docs/compare-superset-subset/ but the title of the page is incorrect.