Work around with HTTPSE breaking ruleset format changes #3

cschanaj · 2018-01-11T07:07:55Z

Close EFForg/https-everywhere/issues/12590, Close brave/browser-laptop/issues/10976
Needed for brave/browser-laptop/issues/11598

Hopefully I did not break anything...

Bisaloo · 2018-02-12T15:41:12Z

@cschanaj, could you use GitHub auto-closing feature please? It works across folders

https://help.github.com/articles/closing-issues-using-keywords/

Also @diracdeltas, not sure if you were automatically pinged on this.

diracdeltas · 2018-02-12T15:43:44Z

Thanks, I did not see this. Will look at it today. Does this address brave/browser-laptop#10976 (comment) ?

diracdeltas · 2018-02-12T20:25:43Z

scripts/preloadHTTPSE.js

 const levelup = require('level')
 const rmDir = require('./util').rmDir
 const exec = require('child_process').exec

-const xpiVersion = '5.2.21' // Manually update this to latest version
+const xpiVersion = '2017.12.6' // Manually update this to latest version


please also bump the version in package.json to 6.0.0. this creates a new ruleset download in S3 instead of overwriting the current one (safer in case the new one has breaking changes).

diracdeltas · 2018-02-12T20:31:59Z

scripts/preloadHTTPSE.js

-  }
-
-  console.log('Writing httpse.json')
-  fs.writeFileSync('./out/httpse.json', JSON.stringify(rulesets), 'utf8')


we still need to create httpse.json since it's used on the Brave desktop browser (only mobile uses leveldb)

diracdeltas · 2018-02-12T20:32:46Z

scripts/preloadHTTPSE.js

@@ -33,84 +32,40 @@ const downloadRulesets = (dir, cb) => {
 }

 const buildDataFiles = () => {
-  // Manually exclude sites that are broken until they are fixed in the next
-  // HTTPS Everywhere release.
-  const exclusions = {


we don't want to delete these exclusions unless all of these rulesets are verified to be fixed in HTTPS Everywhere upstream

cschanaj · 2018-02-13T02:04:37Z

@diracdeltas I've made corresponding changes in my PR. Do I need update the xpiVersion (to 2018.1.29 or even latest) as well? It has been 2 months after 2017.12.6 was released.

diracdeltas · 2018-02-13T22:13:31Z

@cschanaj would be great to update it to 2018.1.29. we avoid latest because we want to pin the version number.

cschanaj · 2018-02-13T22:45:20Z

@diracdeltas please check. thanks!

PS. The original JSON format is not that straight forward...

{
   "ruleset":{
      "$":{
         "name":"101domain.com (partial)",
         "default_off":"failed ruleset test", // gone
         "f":"101domain.com.xml" // gone
      },
      "exclusion":[
         {
            "$":{
               "pattern":"^http://(?:portuguese|spanish)\\.101domain\\.com/+(?!css/|favicon\\.ico|images/)"
            }
         }
      ],
      "rule":[
         {
            "$":{
               "from":"^http://russian\\.101domain\\.com/[^?]*",
               "to":"https://www.101domain.ru/"
            }
         },
         {
            "$":{
               "from":"^http:",
               "to":"https:"
            }
         }
      ]
   }
}

cschanaj · 2018-02-13T23:19:16Z

scripts/preloadHTTPSE.js

+  }
+
+  for (const ruleset of rulesets) {
+    if (!ruleset.default_off && !ruleset.platform) {


the original code include default_off and platform ruleset in httpse.json, which is gone now.

diracdeltas · 2018-02-13T23:57:51Z

scripts/preloadHTTPSE.js

+    'Cisco.com (partial)': 'breaks http://www.cisco.com/c/m/en_us/training-events/events-webinars/techwise-tv/listings.html',
+    'GQ.com (partial)': 'mixed content on gq.com',
+    'Where 2 Get It (partial)': 'maps missing on http://us.coopertire.com/Customer-Care/Dealer-Locator.aspx?form=locator_search&addressline=92346',
+    'Thompson Hotels.com (partial)': 'missing stylesheets on http://www.thompsonhotels.com/'


thanks for going through and re-mapping these!


        [HTTPS Everywhere] simplify ruleset pattern for HTTPSE 6.0

Needed for brave/https-everywhere-builder#3 Note that this is a breaking change from HTTPS Everywhere 5.x's ruleset format.

diracdeltas · 2018-02-14T00:06:15Z

@cschanaj yup, there's no reason to have the HTTPSE.json be as complicated as it used to be. i opened brave/browser-laptop#13133 so you can simplify it somewhat by flattening out the $ properties.

so the new schema can be:

{
   "ruleset":{
      "name": "101domain.com (partial)",
      "exclusion":[
         {
               "pattern":"^http://(?:portuguese|spanish)\\.101domain\\.com/+(?!css/|favicon\\.ico|images/)"
         }
      ],
      "rule":[
         {
               "from":"^http://russian\\.101domain\\.com/[^?]*",
               "to":"https://www.101domain.ru/"
         },
         {
               "from":"^http:",
               "to":"https:"
         }
      ]
   }
}
`

diracdeltas · 2018-02-14T00:15:36Z

scripts/preloadHTTPSE.js

+        })
+      }
+
+      if (ruleset.securecookie) {


we don't have plans to use securecookie in browser-laptop so this can be omitted

thanks for submitting a PR upstream!! I have remove the securecookie properties and simplified the schema in 31d9b8f.

diracdeltas · 2018-02-15T07:39:59Z

This works great, thank you! Please run rm -r node_modules/ and npm install so that it updates package-lock.json. Then commit the new package-lock.json file.

cschanaj · 2018-02-15T11:38:09Z

@diracdeltas done. thanks!!

diracdeltas · 2018-02-15T17:25:11Z

@bsclifton this can be squashed and merged :). once that is done, please run npm run build and npm run upload to upload it to the S3 test bucket. I'll ping you again once it's ready for upload to prod. (https://github.com/brave/https-everywhere-builder/blob/master/scripts/uploadDataFiles.js#L44)


        Work around with HTTPSE breaking ruleset format changes

- Update preloadHTTPSE.js - Update package.json - Bump version in package.json to 6.0.0 - Update scripts/preloadHTTPSE.json - Update to HTTPS Everywhere 2018.1.29 - Changes as per npm run lint - Update scripts/preloadHTTPSE.js - Simplify HTTPSE.json schema - Update package-lock.json

bsclifton

Rebased / squashed per comments by @diracdeltas 😄

bsclifton · 2018-02-15T22:29:00Z

@diracdeltas built + uploaded 👍


        [HTTPS Everywhere] update to latest HTTPS Everywhere

Needed for brave/https-everywhere-builder#3 Note that this is a breaking change from HTTPS Everywhere 5.x's ruleset format.

bsclifton · 2018-02-16T06:18:01Z

pushed to prod with https://jenkins.brave.com/view/laptop%20updates/job/https-everywhere-builder/10/


        [HTTPS Everywhere] update to latest HTTPS Everywhere

Needed for brave/https-everywhere-builder#3 Note that this is a breaking change from HTTPS Everywhere 5.x's ruleset format.


        [HTTPS Everywhere] update to latest HTTPS Everywhere

Needed for brave/https-everywhere-builder#3 Note that this is a breaking change from HTTPS Everywhere 5.x's ruleset format.

diracdeltas assigned diracdeltas and unassigned diracdeltas Feb 12, 2018

diracdeltas self-requested a review Feb 12, 2018

diracdeltas reviewed Feb 12, 2018

View changes

cschanaj changed the title ~~Https everywhere 2017.12.6~~ Work around with HTTPSE breaking ruleset format changes Feb 13, 2018

cschanaj reviewed Feb 13, 2018

View changes

diracdeltas reviewed Feb 13, 2018

View changes

diracdeltas mentioned this pull request Feb 14, 2018

[HTTPS Everywhere] simplify ruleset pattern for HTTPSE 6.0 #13133

Merged

4 of 10 tasks complete

diracdeltas reviewed Feb 14, 2018

View changes

diracdeltas reviewed Feb 15, 2018

View changes

bsclifton dismissed diracdeltas’s stale review via 648c764 Feb 15, 2018

bsclifton approved these changes Feb 15, 2018

View changes

bsclifton merged commit 272d8c4 into brave:master Feb 15, 2018

cschanaj deleted the cschanaj:https-everywhere-2017.12.6 branch Feb 15, 2018

Bisaloo mentioned this pull request Feb 22, 2018

Brave using 5.2 release (outdated) for rulesets #12590

Closed

ryanml added a commit to ryanml/browser-laptop that referenced this pull request Feb 27, 2018

[HTTPS Everywhere] update to latest HTTPS Everywhere

eb1078a

Needed for brave/https-everywhere-builder#3 Note that this is a breaking change from HTTPS Everywhere 5.x's ruleset format.

diracdeltas mentioned this pull request May 10, 2018

Update to HTTPS Everywhere 6.0 #1460

Closed

brave / https-everywhere-builder

Work around with HTTPSE breaking ruleset format changes #3

Work around with HTTPSE breaking ruleset format changes #3

cschanaj commented Jan 11, 2018 •

edited by diracdeltas

Bisaloo commented Feb 12, 2018

diracdeltas commented Feb 12, 2018

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

cschanaj commented Feb 13, 2018 •

edited

diracdeltas commented Feb 13, 2018

cschanaj commented Feb 13, 2018 •

edited

This comment has been minimized.

This comment has been minimized.

diracdeltas commented Feb 14, 2018 •

edited

This comment has been minimized.

This comment has been minimized.

diracdeltas commented Feb 15, 2018 •

edited

cschanaj commented Feb 15, 2018

diracdeltas commented Feb 15, 2018

bsclifton left a comment •

edited

bsclifton commented Feb 15, 2018

bsclifton commented Feb 16, 2018

brave / https-everywhere-builder

Join GitHub today

Work around with HTTPSE breaking ruleset format changes #3

Work around with HTTPSE breaking ruleset format changes #3

Conversation

cschanaj commented Jan 11, 2018 • edited by diracdeltas

Bisaloo commented Feb 12, 2018

diracdeltas commented Feb 12, 2018

This comment has been minimized.

diracdeltas Feb 12, 2018

This comment has been minimized.

diracdeltas Feb 12, 2018

This comment has been minimized.

diracdeltas Feb 12, 2018

cschanaj commented Feb 13, 2018 • edited

diracdeltas commented Feb 13, 2018

cschanaj commented Feb 13, 2018 • edited

This comment has been minimized.

cschanaj Feb 13, 2018

This comment has been minimized.

diracdeltas Feb 13, 2018

diracdeltas commented Feb 14, 2018 • edited

This comment has been minimized.

diracdeltas Feb 14, 2018

This comment has been minimized.

cschanaj Feb 14, 2018

diracdeltas commented Feb 15, 2018 • edited

cschanaj commented Feb 15, 2018

diracdeltas commented Feb 15, 2018

bsclifton left a comment • edited

bsclifton commented Feb 15, 2018

bsclifton commented Feb 16, 2018

cschanaj commented Jan 11, 2018 •

edited by diracdeltas

cschanaj commented Feb 13, 2018 •

edited

cschanaj commented Feb 13, 2018 •

edited

diracdeltas commented Feb 14, 2018 •

edited

diracdeltas commented Feb 15, 2018 •

edited

bsclifton left a comment •

edited