Improve validating sanitizer with context for why element/attribute is invalid

[westonruter]

Summary

• Introduce fine-grained error codes. Instead of just invalid_element when a required attribute is missing, the more specific error ATTR_REQUIRED_BUT_MISSING is raised which explains why it is invalid. Similarly, when an attribute value is incorrect, instead of invalid_attribute occurring when an attribute value violates a regex pattern, a more specific INVALID_ATTR_VALUE_REGEX pattern is raised. Methods which check for whether a node is valid now return an error code on failure rather than just false. These fine-grained error codes will be used in a subsequent PR to add detailed messages, though at the moment just codes are used.
• Remove redundant checks inAMP_Tag_And_Attribute_Sanitizer::validate_attr_spec_list_for_node() which prevent subsequent ability to raise specific error codes while sanitizing attributes.
• Discontinue premature removal of element which misses a mandatory attribute.
• Align error codes with those used by the actual AMP validator, where possible.
• The underlying spec_name is included in the validation error, allowing in a future PR to link a validation error directly to the spec which defines specifically why it is invalid.
• Add assertions when generating the spec to ensure the expected spec names are present, that all spec names are unique (which will be needed later), and that the error_message for blacklisted_cdata_regex is as expected (since we map to an error code).
• Update conversion sanitizers (image, video, audio, iframe) to emit ATTR_REQUIRED_BUT_MISSING validation errors when src is missing.
• Make AMP_Tag_And_Attribute_Sanitizer::sanitize_disallowed_attribute_values_in_node read only, deferring removal or emptying of invalid attribute to AMP_Base_Sanitizer::remove_invalid_attribute.
• Similarly, discontinue removing invalid descendants in AMP_Tag_And_Attribute_Sanitizer::validate_attr_spec_list_for_node() (which should be read-only) to a later point.
• Merge redundant get_disallowed_attributes_in_node into AMP_Tag_And_Attribute_Sanitizer::sanitize_disallowed_attribute_values_in_node.
• Eliminate missing_body_element validation error in Style sanitizer in favor of just adding the body.
• Make AMP_Tag_And_Attribute_Sanitizer::validate_attr_spec_list_for_node() always return integer scores, rather than use a float for the special case.
• Improve tests for AMP_Tag_And_Attribute_Sanitizer by eliminating duplicated assertions and adding error code checking to each test.
• Fix handling of special case URL __amp_source_origin, preventing it from being interpreted as a relative URL.

See #1420.

For Subsequent PRs

• Rework data model in class-amp-allowed-tags-generated.php to be more efficient to look up a tag spec by spec_name. See Refactor generated spec data to facilitate looking up by spec name #3817.
• Add translatable error messages for each specific error code, removing the logic from \AMP_Validation_Error_Taxonomy::get_error_title_from_code() and into the respective sanitizer that originated the validation error. Improve structure of validation error to make it easier to construct messages. For example, harmonize node_attributes and element_attributes to just element_attributes. See Improve architecture of composing error messages for each validation error code #4071.
• Send validation error messages back in the amp_validity REST API field to use in the block warnings, as opposed to duplicating the error message logic. See Improve displaying validation errors' invalid markup in Gutenberg block warning notice #3664.
• Include error details in validation error when failing to fetch external stylesheets. See Not enough information given when fetch_external_stylesheet failure happened #2449.
• Raise separate DISALLOWED_PROPERTY_IN_ATTR_VALUE validation errors for each invalid property, and update the value when sanitization is accepted. See Add AMP_DOM_Document & meta tag sanitizer #3758 (comment) and Add validation of individual properties in meta content attributes #4070.

Checklist

• My pull request is addressing an open issue (please create one otherwise).
• My code is tested and passes existing tests.
• My code follows the Engineering Guidelines (updates are often made to the guidelines, check it out periodically).

[schlessera]

I haven't looked at everything in detail yet, but added a few general comments and questions about the approach.

[schlessera]

The base class retrieves internals from one of its extending classes.

Can we extract all of the validation error constants into a separate class? I tend to create an interface with only constants in such a case. An interface with constants better communicates what is happening: two different pieces of code have a contract about how to refer to a specific problem. Contracts should be encoded in interface, not use internal logic of one another.

namespace Amp\AmpWP;

interface ValidationError {
	const DISALLOWED_TAG             = 'DISALLOWED_TAG';
	const DISALLOWED_CHILD_TAG       = 'DISALLOWED_CHILD_TAG';
	const DISALLOWED_FIRST_CHILD_TAG = 'DISALLOWED_FIRST_CHILD_TAG';
	// [...]
}

Thi is separate of any inheritance chain, and provides one central place where you retrieve constants from. The interface name also makes the syntax nicer to read:

$error['code'] = ValidationError::DISALLOWED_TAG;

If we need to have more grouping, we can have multiple interfaces, like CssValidationError, HtmlValidationError, ...?

[westonruter]

Good point. On one hand, I think this should just be removed entirely, and let the code be UNKNOWN_ERROR. Each sanitizer should be responsible for indicating the code, specifically.

Otherwise, I think this should be scoped with what you discussed in #3780 (comment). It's bound up with generating PHP objects from the spec.

So I think we should do it, but probably not in the scope of this PR.

[schlessera]

If you have 100 properties, you'd love to know exactly which ones are invalid and which ones are valid.

[westonruter]

Yes. Either the invalid properties could be communicated as a property of the one validation error, or there could be multiple validation errors each for a single property. That's the question here. I think the latter is better.

[westonruter]

Rebasing after large PR merged into develop…

[westonruter]

@schlessera There are 3 error codes left which do not have test assertions:

• CDATA_VIOLATES_BLACKLIST: The first one should never occur because it will only happen if the switch statement passes through all cases, which we currently catch when parsing the spec.
• CSS_SYNTAX_PARSE_ERROR: I am not aware of this being a possible error code in practice because the PHP CSS Parser is configured with “LenientParsing” enabled: “When used (which is true by default), the parser will not choke on unexpected tokens but simply ignore them.”
• STYLESHEET_INVALID_FILE_PATH: I couldn't find a way to cause this error scenario.

[schlessera]

STYLESHEET_INVALID_FILE_PATH does not make much sense in our current context. Could it be that this only relates to Optimized AMP SSR?

[schlessera]

😜

[schlessera]

Oh, maybe I should mention I searched for usage first:

• https://wpdirectory.net/search/01DV66Q5TGAA54FBQJA46N9D6R

[schlessera]

Maybe use a constant for this as well?

[schlessera]

Same here, use of a constant makes sense.

[schlessera]

Did a quick search, and no one seems to be using the constants in AMP_Validation_Error_Taxonomy, so BC should be fine.

[westonruter]

STYLESHEET_INVALID_FILE_PATH does not make much sense in our current context. Could it be that this only relates to Optimized AMP SSR?

Not really. It's an error that was returned (sometimes) when a file looked like it was on the filesystem but it could not be read. However, I've now eliminated this in 827659a by always falling back to an HTTP request when the file can't seem to be found on the filesystem.

[kienstra]

If it's alright, tomorrow might be the earliest I could review this.

[schlessera]

@westonruter You've moved this back into Ready for Review, but looking at it, it seems to be merged and there was no mention of this needing additional changes. Is this Done?

[westonruter]

I hadn't moved it to Done yet because #1420 isn't done yet and I wanted to make sure the other follow-ups had been filed as issues. They have now, so I'll move this to done.