Prevent schema.org duplicates #992

oscarssanchez · 2018-03-08T00:27:42Z

Request For Review

Could you use this PR, which addresses #962?

Originally, I used json_decode( $script->nodeValue ) to convert the schema.org json data to an array. However, that brought some issues with get_mustache_tag_placeholders(), as it adds some extra markup to the schema.org script tag.

I opted to use preg_match() as I saw the schema.org URL sometimes can either start with http or https.

Fixes #962.

westonruter

Thanks for the PR!

westonruter · 2018-03-08T07:23:15Z

includes/amp-post-template-actions.php

@@ -14,7 +14,6 @@ function amp_post_template_init_hooks() {
 	add_action( 'amp_post_template_head', 'amp_post_template_add_scripts' );
 	add_action( 'amp_post_template_head', 'amp_post_template_add_fonts' );
 	add_action( 'amp_post_template_head', 'amp_post_template_add_boilerplate_css' );
-	add_action( 'amp_post_template_head', 'amp_print_schemaorg_metadata' );


This should not be removed. The legacy post templates should remain as-is, I believe. The ensure_required_markup method below does not apply for the legacy post template actions.

Hi @westonruter

Thanks for your feedback. I corrected that mistake and restored the line.

westonruter · 2018-03-08T07:26:33Z

tests/test-class-amp-theme-support.php

+	 * @param boolean $expected The expected result.
+	 */
+	public function test_schema_org_present( $script, $expected ) {
+		$page = '<html><head><script>%s</script></head><body>Test</body></html>';


The <script> here is missing the type="application/ld+json".

Very good observation, I added the type attribute on the new tests for ensure_required_markup()

westonruter · 2018-03-08T07:28:19Z

includes/class-amp-theme-support.php

+	public static function schema_org_present( DOMDocument $dom ) {
+		$head = $dom->getElementsByTagName( 'head' )->item( 0 );
+		foreach ( $head->getElementsByTagName( 'script' ) as $script ) {
+			if ( 1 === preg_match( '/{"@context":"https?:[\\\]\/[\\\]\/schema\.org/', $script->nodeValue ) ) {


Before looking at the content of the element ($script->nodeValue) I think it should first check 'application/ld+json' === $script->getAttribute( 'type' ).

Also, I think the 1 === should be removed. As long as preg_match() returns a truthy value that's all we really care about.

Removed the integer comparison and added your suggestion, indeed it makes it more precise.

westonruter · 2018-03-08T07:32:24Z

includes/class-amp-theme-support.php

+	 * @param DOMDocument $dom Representation of the document.
+	 * @return bool
+	 */
+	public static function schema_org_present( DOMDocument $dom ) {


Let's rename this to is_schema_org_metadata_present. But actually, I think maybe the logic should just be inlined in ensure_required_markup like the other checks are done. True that it may be better design to break up the logic into separate methods, but since the existing logic is not in methods then I don't know if we should split out this one separately.

If you changed ensure_required_markup from protected to be public and then added the logic from schema_org_present to it, along with unit tests for ensure_required_markup, then that would be great. Later we may very well want to break up ensure_required_markup into smaller methods, but I think that would probably be done along with breaking up the AMP_Theme_Support class into separate classes.

I added your suggestions.

I also did the test for ensure_required_markup, however, I only made tests for the schema.org script part. Could you use tests for the entire function?

Yes, this method needs more tests, so if that's something you want to add that would be great.

westonruter · 2018-03-08T07:33:11Z

includes/class-amp-theme-support.php

@@ -829,6 +828,13 @@ protected static function ensure_required_markup( DOMDocument $dom ) {
 			) );
 			$head->insertBefore( $meta_viewport, $meta_charset->nextSibling );
 		}
+		if ( ! self::schema_org_present( $dom ) ) {


As noted below, I think this logic should be inlined (for now) as it is done above for $meta_charset and $meta_viewport and $rel_canonical below.

Check if schema.org script is present. Add schema.org only if it doesn't exist.

westonruter · 2018-03-09T06:56:19Z

includes/class-amp-theme-support.php

+		// Prevent schema.org duplicates.
+		$schema_org_script = null;
+		foreach ( $head->getElementsByTagName( 'script' ) as $script ) {
+			if ( 'application/ld+json' === $script->getAttribute( 'type' ) && preg_match( '/{"@context":"https?:[\\\]\/[\\\]\/schema\.org/', $script->nodeValue ) ) {


The regular expression here could be brittle. There's no guarantee that the @context will be the first key, and there's no guarantee there wouldn't be whitespace between the strings. I think doing a simple false !== strpos( $script->nodeValue, 'schema.org' ) could be sufficient here.

We may not even need to look at the value. Just checking if the type is application/ld+json may be just fine.

Hi @westonruter

I opted to search for both the attribute and the 'schema.org' substring as you mentioned it should check for any JSON-LD script in the head which mentions 'schema.org' in any form.

westonruter · 2018-03-09T06:57:05Z

includes/class-amp-theme-support.php

+		$schema_org_script = null;
+		foreach ( $head->getElementsByTagName( 'script' ) as $script ) {
+			if ( 'application/ld+json' === $script->getAttribute( 'type' ) && preg_match( '/{"@context":"https?:[\\\]\/[\\\]\/schema\.org/', $script->nodeValue ) ) {
+				$schema_org_script = $script->nodeValue;


Since the value isn't being used, it may be better to just store a flag like $has_schema_org_metadata = true.

Thanks, it is true, I made the corresponding correction.

westonruter · 2018-03-09T06:58:35Z

includes/class-amp-theme-support.php

+		}
+		if ( ! $schema_org_script ) {
+			$script = $dom->createElement( 'script', wp_json_encode( amp_get_schemaorg_metadata() ) );
+			AMP_DOM_Utils::add_attributes_to_node( $script, array(


Since you're using $dom->createElement() here then I think it makes more sense to just do $script->setAttribute( 'type', 'application/ld+json' )

Also true. Corrected it.

westonruter · 2018-03-09T07:00:20Z

tests/test-class-amp-theme-support.php

+		$dom  = new DOMDocument();
+		$dom->loadHTML( $page );
+		AMP_Theme_Support::ensure_required_markup( $dom );
+		$this->assertEquals( substr_count( $dom->saveHTML(), $schema_test_value ), 1 );


Also should add a test for when JSON is output but the slashes are not escaped.

And add a test for when @context is not the first key.

I made tests for when @ anothercontext is the first key and has schema.org present, I hope I understood correctly.

westonruter · 2018-03-09T07:16:39Z

One more higher-level question: when I first raised this issue I was thinking that there could only be one such metadata script in the document. However, apparently it's fine to have multiple: https://stackoverflow.com/questions/30723531/best-json-ld-practices-using-multiple-script-elements

The main concern here for the AMP plugin is to allow for other plugins to take control of the generation of the Schema.org metadata instead of the defaults that the AMP plugin outputs, but then to have the AMP plugin output a good fallback when others don't do so.

I'm feeling like if there is any JSON-LD script in the head (which mentions schema.org) then we can take this as the signal to just skip outputting the default metadata from the plugin.

westonruter · 2018-03-09T07:19:32Z

includes/class-amp-theme-support.php

+		$schema_org_script = null;
+		foreach ( $head->getElementsByTagName( 'script' ) as $script ) {
+			if ( 'application/ld+json' === $script->getAttribute( 'type' ) && preg_match( '/{"@context":"https?:[\\\]\/[\\\]\/schema\.org/', $script->nodeValue ) ) {
+				$schema_org_script = $script->nodeValue;


Why can't this do json_decode( $script->nodeValue, false ) and check if wp_parse_url( $script['@context'], PHP_URL_HOST ) === 'schema.org?

Hi @westonruter,

Trying to use wp_parse_url with json_decode( $script->nodeValue, false ) breaks the site, but it works fine when $script->nodeValue is converted to an array. The logic however stops working.

I've found that if you comment out these lines in class-amp-dom-utils.php, the logic works again:

$placeholders = self::get_mustache_tag_placeholders(); $document = str_replace( array_keys( $placeholders ), array_values( $placeholders ), $document );

I opted to keep your previous suggestions since i don't know how important are those lines.

OK, I think this makes sense. If I have some JSON that looks like:

{"foo": {"bar":}}

The last }} there is going to get replaced with a placeholder (since this token is returned by get_mustache_tag_placeholders) before parsing the DOM, to then get put back in AMP_DOM_Utils::get_content_from_dom_node(). That means we cannot parse JSON from the parsed DOM currently, which is not ideal. To fix this we'd need to limit the replacements to only happen inside of href and src attributes only, using something like this:

diff --git a/includes/utils/class-amp-dom-utils.php b/includes/utils/class-amp-dom-utils.php index 5288c04..f20fba1 100644 --- a/includes/utils/class-amp-dom-utils.php +++ b/includes/utils/class-amp-dom-utils.php @@ -77,9 +77,15 @@ class AMP_DOM_Utils { * elements since it is faster and wouldn't change the outcome. */ $placeholders = self::get_mustache_tag_placeholders(); - $document = str_replace( - array_keys( $placeholders ), - array_values( $placeholders ), + $document = preg_replace_callback( + '#\s(src|href)=("[^"]+"|\'[^\']+\')#s', + function( $matches ) use ( $placeholders ) { + return str_replace( + array_keys( $placeholders ), + array_values( $placeholders ), + $matches[0] + ); + }, $document );

That would probably be something good to add here to ensure we can read JSON out of the DOM.

I've merged the fix for this in #1019 via 4a10ee4

Logic Refactoring

Prevent schema.org duplicates

1357330

westonruter requested changes Mar 8, 2018

View reviewed changes

westonruter added this to the v0.7 milestone Mar 8, 2018

Prevent schema.org duplicates

c66bfc4

Check if schema.org script is present. Add schema.org only if it doesn't exist.

westonruter reviewed Mar 9, 2018

View reviewed changes

oscarssanchez force-pushed the add/962-prevent-schema-duplicates branch from 8b96d77 to c66bfc4 Compare March 11, 2018 00:15

oscarssanchez and others added 2 commits March 11, 2018 12:08

Schema.org prevent duplicates

67de7c3

Logic Refactoring

Merge branch 'develop' into add/962-prevent-schema-duplicates

fda9f87

westonruter approved these changes Mar 13, 2018

View reviewed changes

westonruter merged commit ab0ae8a into ampproject:develop Mar 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prevent schema.org duplicates #992

Prevent schema.org duplicates #992

oscarssanchez commented Mar 8, 2018 •

edited by westonruter

westonruter left a comment

westonruter Mar 8, 2018

oscarssanchez Mar 9, 2018

westonruter Mar 8, 2018

oscarssanchez Mar 9, 2018 •

edited

westonruter Mar 8, 2018

oscarssanchez Mar 9, 2018

westonruter Mar 8, 2018

oscarssanchez Mar 9, 2018 •

edited

westonruter Mar 9, 2018

westonruter Mar 8, 2018

westonruter Mar 9, 2018

westonruter Mar 9, 2018

oscarssanchez Mar 11, 2018

westonruter Mar 9, 2018

oscarssanchez Mar 11, 2018 •

edited

westonruter Mar 9, 2018

oscarssanchez Mar 11, 2018 •

edited

westonruter Mar 9, 2018

westonruter Mar 9, 2018

oscarssanchez Mar 11, 2018 •

edited

westonruter commented Mar 9, 2018

westonruter Mar 9, 2018

oscarssanchez Mar 11, 2018 •

edited

westonruter Mar 13, 2018 •

edited

westonruter Mar 14, 2018

Prevent schema.org duplicates #992

Prevent schema.org duplicates #992

Conversation

oscarssanchez commented Mar 8, 2018 • edited by westonruter

westonruter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oscarssanchez Mar 9, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oscarssanchez Mar 9, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oscarssanchez Mar 11, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oscarssanchez Mar 11, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oscarssanchez Mar 11, 2018 • edited

Choose a reason for hiding this comment

westonruter commented Mar 9, 2018

Choose a reason for hiding this comment

oscarssanchez Mar 11, 2018 • edited

Choose a reason for hiding this comment

westonruter Mar 13, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oscarssanchez commented Mar 8, 2018 •

edited by westonruter

oscarssanchez Mar 9, 2018 •

edited

oscarssanchez Mar 9, 2018 •

edited

oscarssanchez Mar 11, 2018 •

edited

oscarssanchez Mar 11, 2018 •

edited

oscarssanchez Mar 11, 2018 •

edited

oscarssanchez Mar 11, 2018 •

edited

westonruter Mar 13, 2018 •

edited