Validation checklists

Existing approaches to client validation have always bugged me. In the past I've used specialty endpoints that accepted queries or submissions that validated patterns and if the pattern was not valid an error was generated. This is hard to use because it requires the user to constantly reconfigure their gear.

To start from the opposite end, when testing servers we construct a collection of tests that the Conformance Tool sends to the SUT. We have talked, and I think Steve did a bunch of work on, an approach where we use the Conformance tool layout to give instructions to the user.

I have an alternate approach which would be much easier to use but may not be useable for all cases.

For each test we construct two artifacts, documentation and a server-side validator. The documentation is presented in a checklist. This checklist allows the user to open the documentation for each test and plan their work. There is a validator for each item in the checklist. When the server receives transactions, it attempts to validate them against all validators in this system. Since each validator is tied to a checklist item when a validator is satisfies the user gets credit on the checklist.

The simplest case is easy. The validator is looking for a single transaction. When the that transaction arrives and is correctly validated then credit is shown on the checklist. You only know that the correct transaction has arrived because the validator tags it.

Some more complicated patterns reduce to the simple single transaction pattern. An example is Document Replace. To perform this test the SUT Document Source must submit a document and then replace it. Only the replace request needs to be tagged by the validator.