Tablefy Output and Add Demo CI Workflow

[wenovus]

The demo workflow is not required to pass -- its purpose is to show what the test results would be if it were run on the current public models.

Example failing (table) output: https://gist.github.com/OpenConfigBot/a4af6cb5cab50719b08017e93faa577b
Example passing output: https://gist.github.com/OpenConfigBot/75654c2ce460f11169acc06f37f1b1aa

[robshakir]

One minor change, but this LGTM.

I assume that the 'fail' output is just a demo that will be reverted before submission?

[wenovus]

One minor change, but this LGTM.

I assume that the 'fail' output is just a demo that will be reverted before submission?

The demo is allowed to fail. This is because patterns must be added prior to being used in models-ci, and that means a breaking change CAN cause an expected failure. If there is a failure, however, the current policy (see updated README), is to update the minor version.

[robshakir]

I'm confused -- if the demo is allowed to fail, then it probably should not be something that we're running in the checks that are to be run for a PR.

If we're doing a negative test here, then make the "success" of the check be that the test is failing - that way you can still check that the failure is caught correctly, but not end up with the scenario where it looks like checks are failing for the PR.

[wenovus]

I'm confused -- if the demo is allowed to fail, then it probably should not be something that we're running in the checks that are to be run for a PR.

If we're doing a negative test here, then make the "success" of the check be that the test is failing - that way you can still check that the failure is caught correctly, but not end up with the scenario where it looks like checks are failing for the PR.

OK that makes sense to me, I've made the run passing now. It will still be part of the release process to make sure that with a breaking new test, that the minor version must be incremented. If unsure, simply increment the minor version and update in the model repos.

[wenovus]

@aashaikh Adding you for review of the "Demo" CI checks. If you open the one called Demo on openconfig/public / pattern statement, for the "Demo output on openconfig/public" step, you can see how the current set of tests perform on the current openconfig/public models. This serves as a way of verifying the effects of the tests on the current regexes. Maintainers of this repo would then increment the minor version in order to prevent a failure from occurring on the models repos until they're approved separately via a version increment on the models repos.

This is the workflow I'm proposing for adding new tests/changing tests for existing regexes. I haven't thought about how to deal with adding new regexes yet. I think Rob is largely ok with the proposal, but want to get your double check here.

[wenovus]

I'll assume that it's ok that the "Demo" CI steps pass despite regex failures. We can change this if this turns out to not be desired. I think the exact output from "Demo" that we're expecting has to be manually checked, so an improvement would be to post a comment or a link to a gist on what the failure messages are. I've opened #8 to track that.