Convert http references to https #3998

anselmbradford · 2018-04-11T17:43:50Z

I found a number of URLs in the codebase that are listed as http URLs but are accessible at https URLs, so updated those.

Changes

Update http:// to https:// URLs in codebase.

Testing

gulp test:acceptance should pass.
gulp test:unit should pass.
gulp audit:perf shouldn't show a redundant protocol http://https:// URL.

sebworks · 2018-04-11T18:16:42Z

protractorFailuresReport/protractorTestErrors-1.xml

@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="UTF-8"?>


This shouldn't be in the commit. I'll add it the folder to the git ignore.

Whoops! Removed it. Can we send those into the /test/ directory too?

chosak

See platform#2469; it looks like this PR at least partially addresses that.

chosak · 2018-04-11T18:08:02Z

cfgov/data_research/fixtures/mortgage_metadata.json

@@ -114,7 +114,7 @@
        "note": "",
        "updated": "2017-10-10",
        "name": "download_files",
-        "json_value": "{\"2017-03\":{\"percent_30_60\":{\"County\":{\"url\":\"http://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/CountyMortgagesPercent-30-89DaysLate-thru-2017-03.csv\",\"slug\":\"CountyMortgagesPercent-30-89DaysLate-thru-2017-03\",\"size\":\"40 KB\"},\"MetroArea\":{\"url\":\"http://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/MetroAreaMortgagesPercent-30-89DaysLate-thru-2017-03.csv\",\"slug\":\"MetroAreaMortgagesPercent-30-89DaysLate-thru-2017-03\",\"size\":\"148 KB\"},\"State\":{\"url\":\"http://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/StateMortgagesPercent-30-89DaysLate-thru-2017-03.csv\",\"slug\":\"StateMortgagesPercent-30-89DaysLate-thru-2017-03\",\"size\":\"25 KB\"}},\"thru_month\":\"March 2017\",\"percent_90\":{\"County\":{\"url\":\"http://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/CountyMortgagesPercent-90-plusDaysLate-thru-2017-03.csv\",\"slug\":\"CountyMortgagesPercent-90-plusDaysLate-thru-2017-03\",\"size\":\"40 KB\"},\"MetroArea\":{\"url\":\"http://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/MetroAreaMortgagesPercent-90-plusDaysLate-thru-2017-03.csv\",\"slug\":\"MetroAreaMortgagesPercent-90-plusDaysLate-thru-2017-03\",\"size\":\"148 KB\"},\"State\":{\"url\":\"http://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/StateMortgagesPercent-90-plusDaysLate-thru-2017-03.csv\",\"slug\":\"StateMortgagesPercent-90-plusDaysLate-thru-2017-03\",\"size\":\"25 KB\"}},\"pub_date\":\"October 2017\"}}"
+        "json_value": "{\"2017-03\":{\"percent_30_60\":{\"County\":{\"url\":\"https://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/CountyMortgagesPercent-30-89DaysLate-thru-2017-03.csv\",\"slug\":\"CountyMortgagesPercent-30-89DaysLate-thru-2017-03\",\"size\":\"40 KB\"},\"MetroArea\":{\"url\":\"https://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/MetroAreaMortgagesPercent-30-89DaysLate-thru-2017-03.csv\",\"slug\":\"MetroAreaMortgagesPercent-30-89DaysLate-thru-2017-03\",\"size\":\"148 KB\"},\"State\":{\"url\":\"https://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/StateMortgagesPercent-30-89DaysLate-thru-2017-03.csv\",\"slug\":\"StateMortgagesPercent-30-89DaysLate-thru-2017-03\",\"size\":\"25 KB\"}},\"thru_month\":\"March 2017\",\"percent_90\":{\"County\":{\"url\":\"https://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/CountyMortgagesPercent-90-plusDaysLate-thru-2017-03.csv\",\"slug\":\"CountyMortgagesPercent-90-plusDaysLate-thru-2017-03\",\"size\":\"40 KB\"},\"MetroArea\":{\"url\":\"https://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/MetroAreaMortgagesPercent-90-plusDaysLate-thru-2017-03.csv\",\"slug\":\"MetroAreaMortgagesPercent-90-plusDaysLate-thru-2017-03\",\"size\":\"148 KB\"},\"State\":{\"url\":\"https://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/StateMortgagesPercent-90-plusDaysLate-thru-2017-03.csv\",\"slug\":\"StateMortgagesPercent-90-plusDaysLate-thru-2017-03\",\"size\":\"25 KB\"}},\"pub_date\":\"October 2017\"}}"


URLs like http://files.consumerfinance.gov.s3.amazonaws.com/foo need to become just https://files.consumerfinance.gov/foo

Fixed in 6f01018

chosak · 2018-04-11T18:08:56Z

cfgov/data_research/mortgage_utilities/s3_utils.py

@@ -18,7 +18,7 @@
 S3_SECRET = os.getenv('AWS_S3_SECRET_ACCESS_KEY')
 BASE_BUCKET = settings.AWS_STORAGE_BUCKET_NAME
 MORTGAGE_SUB_BUCKET = "data/mortgage-performance"
-PUBLIC_ACCESS_BASE = 'http://{}.s3.amazonaws.com/{}'.format(
+PUBLIC_ACCESS_BASE = 'https://{}.s3.amazonaws.com/{}'.format(


See above; the HTTPS format for S3 would be something like

'https://{}/{}'.format(BASE_BUCKET, MORTGAGE_SUB_BUCKET)

Is as you have written okay, or does it need testing to verify? Fixed in 6f01018

We could add a new test but it'd just be testing that strings are appended correctly. I tested this manually on your branch:

$ convert_http_to_https* $ ALLOWED_HOSTS='["*"]' DJANGO_SETTINGS_MODULE=cfgov.settings.production cfgov/manage.py shell >>> from data_research.mortgage_utilities.s3_utils import * >>> PUBLIC_ACCESS_BASE, S3_SOURCE_BUCKET (u'https://files.consumerfinance.gov/data/mortgage-performance', u'https://files.consumerfinance.gov/data/mortgage-performance/source')

This looks correct to me.

chosak · 2018-04-11T18:10:04Z

cfgov/data_research/scripts/export_public_csvs.py

@@ -117,7 +117,7 @@ def export_downloadable_csv(geo_type, late_value):
    Each CSV is to start with a National row for comparison.

    CSVs are posted at
-    http://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/  # noqa: E501
+    https://files.consumerfinance.gov.s3.amazonaws.com/data/mortgage-performance/downloads/  # noqa: E501


See above; remove s3.amazonaws.com.

Fixed in 6f01018

chosak · 2018-04-11T18:12:41Z

cfgov/legacy/static/nemo/_/js/search.js

@@ -1,32 +1,31 @@

 $(function() {
-    /* this allows us to pass in HTML tags to autocomplete. Without this they get escaped 
+    /* this allows us to pass in HTML tags to autocomplete. Without this they get escaped


Maybe remove extraneous whitespace changes to reduce the size of this PR?

You can view it without whitespace changes by adding ?w=1 to the PR URL, see https://blog.github.com/2011-10-21-github-secrets/#whitespace

Does this work for you?

chosak · 2018-04-11T18:13:47Z

cfgov/unprocessed/apps/analytics-gtm/js/enterprise-attribution-cookies.js

@@ -201,7 +201,7 @@ const sourceCookies = {
   */
  parseOrganicSearch: function( host, query, referrer ) {
    // Referrer must not be the same host
-    if ( hostsAreEqual( 'http://' + host, referrer ) === false ) {
+    if ( hostsAreEqual( 'https://' + host, referrer ) === false ) {


Is this a file we should be modifying?

Yes, host is set by document.location.host on https://github.com/cfpb/cfgov-refresh/blob/master/cfgov/unprocessed/apps/analytics-gtm/js/enterprise-attribution-cookies.js#L414 and https://github.com/cfpb/cfgov-refresh/blob/master/cfgov/unprocessed/apps/analytics-gtm/js/enterprise-attribution-cookies.js#L53, which would be our pages where this script runs, which will be https.

chosak · 2018-04-11T18:14:34Z

cfgov/unprocessed/apps/owning-a-home/js/explore-rates/rate-checker.js

@@ -1,5 +1,12 @@
 import $ from 'jquery';
 import 'rangeslider.js';
+import {


Same comment as above; would you consider removing extraneous whitespace changes to reduce the scope of this PR?

Whoops! #3991 appears to have snuck in here. I'll see if I can get that one reviewed and rebase here.

This is the non-whitespace diff URL https://github.com/cfpb/cfgov-refresh/pull/3998/files?w=1

chosak · 2018-04-11T18:16:41Z

docs/caching.md

@@ -2,15 +2,15 @@

 ### Akamai

-We use [Akamai](https://www.akamai.com/), a content delivery network, to cache the entirety of [www.consumerfinance.gov](https://www.consumerfinance.gov/) (but not our development servers). We invalidate any given page in Wagtail when it is published or unpublished (by hooking up the custom class [`AkamaiBackend`](https://github.com/cfpb/cfgov-refresh/blob/master/cfgov/v1/models/akamai_backend.py) to [Wagtail's frontend cache invalidator](http://docs.wagtail.io/en/v1.9/reference/contrib/frontendcache.html). By default, we clear the Akamai cache any time we deploy. 
+We use [Akamai](https://www.akamai.com/), a content delivery network, to cache the entirety of [www.consumerfinance.gov](https://www.consumerfinance.gov/) (but not our development servers). We invalidate any given page in Wagtail when it is published or unpublished (by hooking up the custom class [`AkamaiBackend`](https://github.com/cfpb/cfgov-refresh/blob/master/cfgov/v1/models/akamai_backend.py) to [Wagtail's frontend cache invalidator](http://docs.wagtail.io/en/v2.0.1/reference/contrib/frontendcache.html). By default, we clear the Akamai cache any time we deploy.


It makes me sad that docs.wagtail.io can't be HTTPS; this looks to be a ReadTheDocs thing.

chosak · 2018-04-11T18:17:34Z

protractorFailuresReport/protractorTestErrors-1.xml

@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="UTF-8"?>


Did you mean to commit these files?

chosak · 2018-04-11T18:17:56Z

refresh-data.sh

@@ -42,7 +42,7 @@ if [[ -z "$refresh_dump_name" ]]; then

 Or you can define the location of a dump and this script will download it for you:

-	export CFGOV_PROD_DB_LOCATION=http://some-bucket.s3.amazonaws.com/wherever/production_django.sql.gz
+	export CFGOV_PROD_DB_LOCATION=https://some-bucket.s3.amazonaws.com/wherever/production_django.sql.gz


This won't work with HTTPS if you include the s3.amazonaws.com (see above).

Removed in 225af8f

chosak · 2018-04-11T18:51:24Z

TERMS.md

  co-created with Jonathan Neal, is licensed under the MIT License.
- [Slick](http://kenwheeler.github.io/slick/) by Ken Wheeler is licensed under


If you're cleaning up this reference to Slick (which I guess is no longer used), can we also remove this one?

chosak · 2018-04-11T19:18:36Z

refresh-data.sh

@@ -42,7 +42,7 @@ if [[ -z "$refresh_dump_name" ]]; then

 Or you can define the location of a dump and this script will download it for you:

-	export CFGOV_PROD_DB_LOCATION=http://some-bucket.s3.amazonaws.com/wherever/production_django.sql.gz
+	export CFGOV_PROD_DB_LOCATION=https://some-bucket.com/wherever/production_django.sql.gz


Now that we're removing references to our explicit use of S3, how would you feel about removing the word "bucket" here? Since in theory the production dumps could live anywhere.

Updated to example.com

feedback

chosak

Thanks for responding to my feedback @anselmbradford. This generally looks good to me, as there isn't much that would impact backend code besides the one change to the mortgage utilities code.

anselmbradford requested review from rosskarchner, chosak, sebworks and willbarton April 11, 2018 17:43

accesslint bot approved these changes Apr 11, 2018

View reviewed changes

Convert http references to https

6e49488

anselmbradford force-pushed the convert_http_to_https branch from babc696 to 6e49488 Compare April 11, 2018 17:45

Add missing migrations

f1a6529

sebworks reviewed Apr 11, 2018

View reviewed changes

chosak previously requested changes Apr 11, 2018

View reviewed changes

anselmbradford added 3 commits April 11, 2018 14:18

Remove test error reports

fc47653

Remove AWS URL from files.cf.gov URLs

6f01018

Remove AWS URL from example URL

225af8f

chosak reviewed Apr 11, 2018

View reviewed changes

Merge branch 'master' into convert_http_to_https

c0a629d

chosak reviewed Apr 11, 2018

View reviewed changes

anselmbradford added 2 commits April 11, 2018 15:29

Remove slick from gitignore

cbf42f6

Update example URL

707ab79

chosak approved these changes Apr 11, 2018

View reviewed changes

Merge branch 'master' into convert_http_to_https

816f474

anselmbradford merged commit 582f0ac into master Apr 11, 2018

anselmbradford deleted the convert_http_to_https branch April 11, 2018 23:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert http references to https #3998

Convert http references to https #3998

anselmbradford commented Apr 11, 2018

sebworks Apr 11, 2018 •

edited

Loading

anselmbradford Apr 11, 2018

chosak left a comment

chosak Apr 11, 2018

anselmbradford Apr 11, 2018 •

edited

Loading

chosak Apr 11, 2018

anselmbradford Apr 11, 2018

chosak Apr 11, 2018

chosak Apr 11, 2018

anselmbradford Apr 11, 2018

chosak Apr 11, 2018

anselmbradford Apr 11, 2018

chosak Apr 11, 2018

anselmbradford Apr 11, 2018

chosak Apr 11, 2018

anselmbradford Apr 11, 2018

anselmbradford Apr 11, 2018

chosak Apr 11, 2018

anselmbradford Apr 11, 2018

chosak Apr 11, 2018

chosak Apr 11, 2018

anselmbradford Apr 11, 2018

chosak Apr 11, 2018

anselmbradford Apr 11, 2018

chosak Apr 11, 2018

anselmbradford Apr 11, 2018

chosak left a comment •

edited

Loading

		co-created with Jonathan Neal, is licensed under the MIT License.
		- [Slick](http://kenwheeler.github.io/slick/) by Ken Wheeler is licensed under

Convert http references to https #3998

Convert http references to https #3998

Conversation

anselmbradford commented Apr 11, 2018

Changes

Testing

sebworks Apr 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chosak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anselmbradford Apr 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chosak left a comment • edited Loading

Choose a reason for hiding this comment

sebworks Apr 11, 2018 •

edited

Loading

anselmbradford Apr 11, 2018 •

edited

Loading

chosak left a comment •

edited

Loading