#804 Update Allowed Tags And Attributes #823

kienstra · 2017-12-07T09:39:52Z

This PR fixes and improves the AMP HTML tag and attributes updater, refer to the ACs for more info...

Steps To Manually Run Script

In order for amp_wp_build.py to generate class-amp-allowed-tags-generated.php, run these commands, using a Linux environment like VVV:

cd /path/to/your/vvv/wp-content/plugins/amp-wp
bash bin/amphtml-update.sh

Note: these steps were edited, as most of these steps are now in the file bin/amphtml-update.sh. That file also includes the step:
7. git apply this commit to re-add validator_gen_md.py, which this commit deleted

See #689 (Only the requirement "Minor: The script should do the git clone...")
Fixes #761
Fixes #772
Fixes #804
Closes #749
Closes #805

Generate class-amp-allowed-tags-generated.php, Using amp_wp_build.py. Followed the instructions at the top of the file. This is spec file revision 527.

Before, there was an error in: tag_spec.also_requires_tag This didn't always have the property. Also, update the directory of the extensions. The .protoascii file is no longer in the directory '0.1.'

Since the allowed tags and attributes are updated, Several assertions failed. Update them to reflect the new AMP spec. Since there is no longer a protocol whitelist, Remove the assertions for protocols.

This file was generated from a build script, In the repo https://github.com/ampproject/amphtml The spec file seems to require an empty value: extensions/amp-ad/validator-amp-ad.protoascii But examples of <amp-ad> seem to have the value. Like <amp-ad data-multi-size=320x50. @see examples/a4a-multisize.amp.html. So remove the line in the PHP file that requires it to be empty.

Having generated a new file for the allowed tags and attributes, Add these to the data for test_sanitizer(). Also, add unit test for is_missing_mandatory_attribute(). And rename that function from: is_mandatory_attribute_missing().

Before, this was inside an 'else' block. And it should be outside of this, So it applies more broadly. If there is a list of attributes, and one is missing, Remove the node. This was the previous behavior.

This was assigned a value that's used later. There's no need to store it in a variable.

There was some missing documentation. Add this for test_sanitizer and validate_tag_spec_for_node.

This has hundreds of errors and warnings. It's generated from a Python script.

In response to the warnings in Travis. And align an equal sign in test_sanitizer().

Before, there was a failed Travis build. This used boolval(), which caused a fatal error. Instead, simply compare the value to true.

In response to a failed Travis build. Fix an error in an array not being aligned. An move an = sign 2 spaces to the left.

There was an error in the Travis build. A function accessed a DOMElement property as an array. Instead, use the setAttribute() method.

kienstra · 2017-12-07T11:28:38Z

includes/sanitizers/class-amp-tag-and-attribute-sanitizer.php

-					$attr_spec_list = array_merge( $attr_spec_list, $rule_spec_list_to_validate[ $id ][AMP_Rule_Spec::ATTR_SPEC_LIST] );
+				foreach ( $spec_ids_sorted as $id ) {
+					$spec_list = isset( $rule_spec_list_to_validate[ $id ][ AMP_Rule_Spec::ATTR_SPEC_LIST ] ) ? $rule_spec_list_to_validate[ $id ][ AMP_Rule_Spec::ATTR_SPEC_LIST ] : null;
+					if ( ! $this->is_missing_mandatory_attribute( $spec_list, $node ) ) {


This addresses an edge case where there are more than one spec lists, and one of them requires a value. If one of them doesn't require a value, use that instead.

This detects that with $this->is_missing_mandatory_attribute(). If the node satisfies one of the specs, this doesn't remove it.

kienstra · 2017-12-07T11:29:50Z

includes/sanitizers/class-amp-tag-and-attribute-sanitizer.php

-			if ( $is_mandatory && ! $attribute_exists ) {
-				$this->remove_node( $node );
-				return;
-			}


This is mainly decomposed into the function below, is_missing_mandatory_attribute().

kienstra · 2017-12-07T11:30:23Z

includes/sanitizers/class-amp-tag-and-attribute-sanitizer.php

@@ -430,8 +449,7 @@ private function _sanitize_disallowed_attribute_values_in_node( $node, $attr_spe
 		foreach ( $attrs_to_remove as $attr_name ) {
 			if ( isset( $attr_spec_list[$attr_name][AMP_Rule_Spec::ALLOW_EMPTY] ) &&
 				( true == $attr_spec_list[$attr_name][AMP_Rule_Spec::ALLOW_EMPTY] ) ) {
-				$attr = $node->attributes;
-				$attr[ $attr_name ]->value = '';
+				$node->setAttribute( $attr_name, '' );


This raised an error, so it uses the setAttribute() method instead.

kienstra · 2017-12-07T11:35:46Z

phpcs.xml

@@ -19,4 +19,5 @@
 	<exclude-pattern>*/dev-lib/*</exclude-pattern>
 	<exclude-pattern>*/node_modules/*</exclude-pattern>
 	<exclude-pattern>*/vendor/*</exclude-pattern>
+	<exclude-pattern>*/includes/sanitizers/class-amp-allowed-tags-generated.php</exclude-pattern>


This file is generated by the Python file, and created hundreds of warnings and errors in PHPCS.

kienstra · 2017-12-07T11:36:39Z

tests/amp-tag-and-attribute-sanitizer-private-methods-tests.php

@@ -246,136 +246,6 @@ public function get_attr_spec_rule_data() {
 				),
 				'expected' => AMP_Rule_Spec::NOT_APPLICABLE,
 			),
-


The allowed protocol values are removed from the spec file, so this removes the tests for them.

….php

westonruter · 2017-12-08T03:52:40Z

Double-arrow PHPCS errors are suppressed for now via 2b1443e.

westonruter · 2017-12-08T03:55:46Z

@kienstra I think the next thing here is to look over gitHub issues that seem to be related to the spec being out of date, and confirm that the issues are now resolved with your latest changes.

Thanks, @westonruter. I'll look at the other issues surrounding the specification needing to be updated, and confirm if this pull request resolves them. -Ryan

ThierryA

Great job on the PR @kienstra. Below is a bash script draft which is still in WIP.

ThierryA · 2017-12-08T16:21:56Z

bin/amphtml-update.sh

@@ -0,0 +1,40 @@
+#!/bin/bash


Here is a draft the bash script to update AMP HTML. This script is meant to run in Linux environment (ex. VVV). To run the script, simply cd to the plugin root and run bash bin/amphtml-update.sh.

westonruter · 2017-12-08T16:47:31Z

bin/amphtml-update.sh

+# Go to the right location.
+if [[ '.' != $(dirname "$0") ]]; then
+	cd bin
+fi


What I usually do instead of this conditional is:

cd "$(dirname "$0")"

westonruter · 2017-12-08T16:48:32Z

bin/amphtml-update.sh

+sudo apt-get install git
+sudo apt-get install python
+sudo apt-get install protobuf-compiler
+sudo apt-get install python-protobuf


You can combine all of the dependencies into one call:

sudo apt-get install git python protobuf-compiler python-protobuf

westonruter · 2017-12-08T16:49:26Z

bin/amphtml-update.sh

+# Clone amphtml repo.
+if [[ ! -e $VENDOR_PATH/amphtml ]]; then
+	git clone https://github.com/ampproject/amphtml amphtml
+fi


Maybe add an else here which goes into $VENDOR_PATH/amphtml and does git pull?

westonruter · 2017-12-08T16:53:13Z

bin/amphtml-update.sh

+
+# Install dependencies.
+sudo apt-get install git
+sudo apt-get install python


Maybe we should make git and python just dependencies that we assume are installed, and exit if the command doesn't exist?

if ! command -v git >/dev/null 2>&1; then echo "Git is required" exit 1 fi if ! command -v python >/dev/null 2>&1; then echo "Python is required" exit 1 fi

If you are ok with it, I think it is fine to automatically install git or python if not installed, the check is pretty quick if it is already installed.

westonruter · 2017-12-08T16:54:41Z

bin/amphtml-update.sh

+PROJECT_PATH=$(dirname $PWD)
+VENDOR_PATH=$PROJECT_PATH/vendor
+
+if ! apt-get > /dev/null; then


Better to check if the command exists via: if ! command -v apt-get >/dev/null 2>&1; then

westonruter · 2017-12-08T16:54:55Z

bin/amphtml-update.sh

+
+if ! apt-get > /dev/null; then
+	echo -e "The AMP HTML uses the apt-get, make sure to run this script in a Linux environment"
+	exit


Do exit 1 so a failure error code is sent. An exit means just exit 0 which means success.

westonruter · 2017-12-08T16:55:28Z

bin/amphtml-update.sh

+VENDOR_PATH=$PROJECT_PATH/vendor
+
+if ! apt-get > /dev/null; then
+	echo -e "The AMP HTML uses the apt-get, make sure to run this script in a Linux environment"


No need for -e

ThierryA · 2017-12-08T17:24:34Z

@kienstra now that the bash script to update the HTML AMP tags is completed, you may update the PR description with the steps to run the script:

cd to the AMP plugin root
run bash bin/amphtml-update.sh

Note that the amphtml-update.sh bash script is meant to run in a Linux environment (ex. VVV).

Checklist

update the PR description with the steps above
add a contributing.md file in the root with the instruction to update the HTML AMP tags and attributes as described above

westonruter · 2017-12-09T00:34:21Z

@kienstra This PR will close PR #749 as obsolete, correct?

Thanks, @westonruter. You're right, PR #749 is obsolete now. -Ryan

This describes how to use the new script amphtml-update.sh. This is mainly modified from another open-source file. It could probably use more information. Including the workflow for submitting pull requests.

kienstra · 2017-12-09T03:09:10Z

Created contributing.md, Updated Script Steps,

Hi @westonruter and @ThierryA,
Thanks a lot for your work on this. The commit above adds a contributing.md file, with instructions to use your new script amphtml-update.sh. At some point, that file should probably have more details about contributing.

Still, I need to look at the remaining issues for AMP tag support, and see if this PR addresses them.

As harrisdavid mentioned, before this PR, the 'src' was mandatory. But with the updated AMP spec, it's not.

Before, there was an issue in the file that created: class-amp-allowed-tags-generated.php. amphtml-update.sh had a conditional: if isinstance(value, list) But the allowed_protocol attribute has a type of: google.protobuf.internal.containers.RepeatedScalarFieldContainer So that conditional failed. And allowed_protocol wasn't added. So instead, make that an 'else' statement. That change only resulted in 'allowed_tags' being added.

ThierryA · 2017-12-11T08:13:14Z

contributing.md

@@ -0,0 +1,30 @@
+# AMP Contributing Guide
+


Perhaps add a welcoming sentence (ex. "Thanks for taking the time to contribute!")

Thanks, @ThierryA. This commit adds your sentence to contributing.md:

Thanks for taking the time to contribute!

ThierryA · 2017-12-11T08:17:23Z

contributing.md

+2. run `bash bin/amphtml-update.sh`
+That script is intended for a Linux environment like [VVV](https://github.com/Varying-Vagrant-Vagrants/VVV).
+
+### PHPUnit Testing


The way the dev lib runs PHPUnit (run_phpunit_local) needs to run in an environment which has WordPress Unit Test installed, for example VVV. I would advise to add that as a note.

Yes, this came up in #828,.along with questions about how to get pre-commit hook working from wp-dev-lib.

Thanks, @ThierryA. This commit adds your point to contributing.md:

Please run these tests in an environment with WordPress unit tests installed, like [VVV](https://github.com/Varying-Vagrant-Vagrants/VVV).

ThierryA · 2017-12-11T09:38:18Z

@kienstra as you go through the PRs and issues that this PR address, could you add the Fixes #{number} and/or Closes #{number} to the description (I have already added a few). Thanks,

kienstra · 2017-12-11T18:23:50Z

Sure, @ThierryA. I'll add Fixes #{number} and or Closes #{number} to related tickets, as they apply.

@ThierryA

Add a point that this requires WP unit tests. Also, begin with the welcome that Thierry suggested. Props @ThierryA.

…tic/amp-wp into feature/804-allowed-tags

kienstra · 2017-12-11T20:29:33Z

Added Issues That Are Fixed

Hi @ThierryA and @westonruter,
Having looked at all of the issues with the labels validation-errors and sanitizers, I added these issues to the PR description as fixed by this PR:

#689 (Only the requirement "Minor: The script should do the git clone...")
#761
#772

The following issues also look to have been resolved, though not with this PR. I made comments to that effect on the tickets, requesting that the openers verify them:
#237
#253
#781

In phpcs.xml, retain edits from both branches. There were several conflicts in: class-amp-tag-and-attribute-sanitizer.php These were mainly conflicts with 03c12. Some were only DocBlocks, but some were logic. I mainly resovled them in favor of this branch: feature/804-allowed-tags.

When this has a type of 'application/json,' There is a required ancestor. So create an ancestor of <amp-analytics>.

In response to a failed Travis build. Also, change == to ===.

kienstra · 2017-12-12T03:11:14Z

Resolved Merge Conflicts

Hi @westonruter,
Thanks for waiting. This commit merges in develop, and resolves the merge conflicts. The commits that follow fix a PHPUnit test and PHPCS issues.

westonruter

Impressive work here! You wrestled with dragons. Let's get these changes in wider testing.

Ryan Kienstra added 14 commits December 4, 2017 18:17

804 : Update allowed AMP tags in sanitizer file.

0b987a6

Generate class-amp-allowed-tags-generated.php, Using amp_wp_build.py. Followed the instructions at the top of the file. This is spec file revision 527.

804 : Prevent errors on amp_wp_build.py, update extensions dir.

30064ca

Before, there was an error in: tag_spec.also_requires_tag This didn't always have the property. Also, update the directory of the extensions. The .protoascii file is no longer in the directory '0.1.'

Create is_mandatory_attribute_missing method

ba459a2

804 : Modify PHPUnit tests for tag and attribute validation.

8cff768

Since the allowed tags and attributes are updated, Several assertions failed. Update them to reflect the new AMP spec. Since there is no longer a protocol whitelist, Remove the assertions for protocols.

804 : PHPUnit tests for newly-allowed tags.

a698eea

Having generated a new file for the allowed tags and attributes, Add these to the data for test_sanitizer(). Also, add unit test for is_missing_mandatory_attribute(). And rename that function from: is_mandatory_attribute_missing().

804 : Move logic for missing attributes.

a066117

Before, this was inside an 'else' block. And it should be outside of this, So it applies more broadly. If there is a list of attributes, and one is missing, Remove the node. This was the previous behavior.

804 : Remove the unused $spec_value variable.

cfcf730

This was assigned a value that's used later. There's no need to store it in a variable.

804 : Improve variable alignment and add PHPDoc tags.

8f0a925

There was some missing documentation. Add this for test_sanitizer and validate_tag_spec_for_node.

804 : Exclude the generated allowed tags file from PHPCS checks.

a400fa5

This has hundreds of errors and warnings. It's generated from a Python script.

804 : Align 'array' keywords vertically.

cfc0247

In response to the warnings in Travis. And align an equal sign in test_sanitizer().

804 : Change the conditional to simply compare to true.

7f9a2db

Before, there was a failed Travis build. This used boolval(), which caused a fatal error. Instead, simply compare the value to true.

804 : Array and equal sign alignment fixes.

b4bfd81

In response to a failed Travis build. Fix an error in an array not being aligned. An move an = sign 2 spaces to the left.

804 : Prevent accessing an object as an array.

6f5e39d

There was an error in the Travis build. A function accessed a DOMElement property as an array. Instead, use the setAttribute() method.

kienstra commented Dec 7, 2017

View reviewed changes

kienstra requested a review from ThierryA December 7, 2017 11:34

kienstra assigned ThierryA Dec 7, 2017

kienstra commented Dec 7, 2017

View reviewed changes

kienstra requested a review from westonruter December 7, 2017 15:44

kienstra assigned westonruter Dec 7, 2017

Suppress DoubleError phpcs errors in test-tag-and-attribute-sanitizer…

2b1443e

….php

kienstra mentioned this pull request Dec 8, 2017

Proposing to re-add the deleted file validator_gen_md.py ampproject/amphtml#12371

Closed

ThierryA added 2 commits December 8, 2017 17:08

[WIP] AMP HTML bash updater draft

c73f92b

[WIP] AMP HTML bash updater improvements

f67041d

ThierryA reviewed Dec 8, 2017

View reviewed changes

westonruter reviewed Dec 8, 2017

View reviewed changes

Wrap up AMP HTML bash updater

236968b

804 : Add a contributing.md file, with steps to use build script.

eab6207

This describes how to use the new script amphtml-update.sh. This is mainly modified from another open-source file. It could probably use more information. Including the workflow for submitting pull requests.

kienstra changed the title ~~#804 : Update Allowed Tags And Attributes~~ Issue 804 : Update Allowed Tags And Attributes Dec 9, 2017

Ryan Kienstra and others added 3 commits December 9, 2017 11:47

804 : Test that 'amp-playbuzz' is sanitized properly.

eef4b8b

As harrisdavid mentioned, before this PR, the 'src' was mandatory. But with the updated AMP spec, it's not.

Fix WPCS warning

4597be5

ThierryA reviewed Dec 11, 2017

View reviewed changes

ThierryA changed the title ~~Issue 804 : Update Allowed Tags And Attributes~~ #804 : Update Allowed Tags And Attributes Dec 11, 2017

ThierryA changed the title ~~#804 : Update Allowed Tags And Attributes~~ #804 Update Allowed Tags And Attributes Dec 11, 2017

Ryan Kienstra added 2 commits December 11, 2017 14:15

804 : Add to contributing.md, including unit test information.

44554c5

Add a point that this requires WP unit tests. Also, begin with the welcome that Thierry suggested. Props @ThierryA.

Merge branch 'feature/804-allowed-tags' of https://github.com/Automat…

ed3c93d

…tic/amp-wp into feature/804-allowed-tags

westonruter added this to the v0.6 milestone Dec 11, 2017

mjangda mentioned this pull request Dec 12, 2017

Whitelist Sanitizer Improvements #689

Closed

6 tasks

Ryan Kienstra added 2 commits December 11, 2017 20:03

804 : Correct PHPUnit test for <script> validation.

4b3ec53

When this has a type of 'application/json,' There is a required ancestor. So create an ancestor of <amp-analytics>.

804 : Add spaces to correct PHPCS errors.

af7b830

In response to a failed Travis build. Also, change == to ===.

westonruter approved these changes Dec 12, 2017

View reviewed changes

westonruter merged commit 7f79a50 into develop Dec 12, 2017

westonruter deleted the feature/804-allowed-tags branch December 12, 2017 06:03

westonruter mentioned this pull request Dec 13, 2017

Support extra attributes for amp-iframe #800

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#804 Update Allowed Tags And Attributes #823

#804 Update Allowed Tags And Attributes #823

kienstra commented Dec 7, 2017 •

edited by westonruter

kienstra Dec 7, 2017 •

edited

kienstra Dec 7, 2017

kienstra Dec 7, 2017

kienstra Dec 7, 2017

kienstra Dec 7, 2017

westonruter commented Dec 8, 2017

westonruter commented Dec 8, 2017 •

edited by kienstra

ThierryA left a comment

ThierryA Dec 8, 2017

westonruter Dec 8, 2017

westonruter Dec 8, 2017

westonruter Dec 8, 2017

westonruter Dec 8, 2017

ThierryA Dec 8, 2017 •

edited

westonruter Dec 8, 2017

westonruter Dec 8, 2017

westonruter Dec 8, 2017

ThierryA commented Dec 8, 2017 •

edited by kienstra

westonruter commented Dec 9, 2017 •

edited by kienstra

kienstra commented Dec 9, 2017

ThierryA Dec 11, 2017

kienstra Dec 11, 2017

ThierryA Dec 11, 2017

westonruter Dec 11, 2017

kienstra Dec 11, 2017

ThierryA commented Dec 11, 2017

kienstra commented Dec 11, 2017

kienstra commented Dec 11, 2017 •

edited

kienstra commented Dec 12, 2017

westonruter left a comment

#804 Update Allowed Tags And Attributes #823

#804 Update Allowed Tags And Attributes #823

Conversation

kienstra commented Dec 7, 2017 • edited by westonruter

kienstra Dec 7, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

westonruter commented Dec 8, 2017

westonruter commented Dec 8, 2017 • edited by kienstra

ThierryA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThierryA Dec 8, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThierryA commented Dec 8, 2017 • edited by kienstra

Checklist

westonruter commented Dec 9, 2017 • edited by kienstra

kienstra commented Dec 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThierryA commented Dec 11, 2017

kienstra commented Dec 11, 2017

kienstra commented Dec 11, 2017 • edited

kienstra commented Dec 12, 2017

westonruter left a comment

Choose a reason for hiding this comment

kienstra commented Dec 7, 2017 •

edited by westonruter

kienstra Dec 7, 2017 •

edited

westonruter commented Dec 8, 2017 •

edited by kienstra

ThierryA Dec 8, 2017 •

edited

ThierryA commented Dec 8, 2017 •

edited by kienstra

westonruter commented Dec 9, 2017 •

edited by kienstra

kienstra commented Dec 11, 2017 •

edited