Validation tests for examples and test fixtures

[jpettitt]

amphtml files in /examples and /test/fixtures/e2e should be validated.

Errors in test pages can cause test flakes (see #25145). Having these files validated (if only as part of the local changes check) would help catch these hard to debug flakes.

[rsimha]

We currently run a whole host of validator checks via gulp validator. Would it be possible to add our test html fixtures to the list of files that are validated?

@honeybadgerdontcare @Gregable Thoughts?

/cc @ampproject/wg-caching

[honeybadgerdontcare]

The tests we have for the validator are explicitly to test validation scenarios (both PASS and FAIL). It sounds like the request here is to make sure that example files can only be valid AMP (PASS). Is that a correct reading?

If so, we could have a test that won't allow merging unless all files in certain directories pass validation. However, would this impede work that is in progress but not yet valid?

Are there other directories this could apply to other than /examples/.html and /test/fixtures/e2e/.html?

[cramforce]

We should do it, but allow for allow-listing expected failures.

[Gregable]

@rsimha I feel this is an infra feature request, not validation. Feel free to re-label if you disagree.

[jpettitt]

Yes the intent was that people making examples should make valid ones.

[rsimha]

The primary feature requested by this issue has been implemented. Existing invalid fixtures can be fixed piecemeal by following the instructions at #34000 (comment).