Release 8.2.0

Functional SRCH-2324 update newrelic_rpm (SRCH-2324 update newrelic_rpm #771) … SRCH-2453 verify host when connecting to Twitter's streaming API (SRCH-2453 verify host when connecting to Twitter's streaming API #769) … Testing or Process SRCH-2457 add pull request template (SRCH-2457 add pull request template #774) SRCH-2110 consolidate cucumber tests (SRCH-2110 consolidate cucumber tests #773)
GSA · Sep 2, 2021 · 552de77 · 552de77
2 parents 5cb8557 + e921c3a
commit 552de77
Show file tree

Hide file tree

Showing 15 changed files with 1,015 additions and 144 deletions.
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -0,0 +1,28 @@
+## Summary
+- Brief summary of the changes included in this PR
+- Any additional information or context which may help the reviewer
+
+### Checklist
+Please ensure you have addressed all concerns below before marking a PR "ready for review" or before requesting a re-review:
+
+#### Functionality Checks
+
+- [ ] Code is functional.
+
+- [ ] Automated checks pass, if applicable. If Code Climate checks do not pass, explain reason for failures:
+
+- [ ] If your changes will be tested manually, you have run `bundle update` and committed your changes to Gemfile.lock.
+
+- [ ] You have merged the latest changes from the target branch (usually `master` or `main`) into your branch.
+
+- [ ] If your target branch is NOT `master` or `main`, specify the reason:
+
+- [ ] PR title is either of the format **SRCH-#### \<description\>** matching the associated Jira ticket (i.e. "SRCH-123 implement feature X"), or **Release #.#.#** matching the release number
+
+- [ ] You have squashed your commits into a single commit (exception: your PR includes commits with formatting-only changes, such as required by Rubocop)
+
+- [ ] Your primary commit message is of the format **SRCH-#### \<description\>** matching the associated Jira ticket
+
+#### Process Checks
+
+- [ ] You have specified an "Assignee", and if necessary, additional reviewers
diff --git a/Gemfile b/Gemfile
@@ -27,13 +27,16 @@ gem 'resque-scheduler', '~> 4.3.1'
 # Paperclip is deprecated: https://cm-jira.usa.gov/browse/SRCH-702
 gem 'paperclip', '~> 5.2.0'
 gem 'googlecharts', '~> 1.6.12'
-gem 'tweetstream', '~> 2.6.1' # no longer maintained?
+# Using custom fork of tweetstream until vulnerability is resolved
+# (or until we switch to Twitter's API v2):
+# https://github.com/tweetstream/tweetstream/issues/212
+gem 'tweetstream', git: 'https://github.com/GSA/tweetstream'
 gem 'twitter', git: 'https://github.com/GSA/twitter.git', branch: '5-stable'
 gem 'flickraw', '~> 0.9.9'
 gem 'active_scaffold', '~> 3.5.0'
 gem 'active_scaffold_export', git: 'https://github.com/naaano/active_scaffold_export'
 gem "recaptcha", '~> 4.6.3', :require => "recaptcha/rails"
-gem 'newrelic_rpm', '~> 5.0.0'
+gem 'newrelic_rpm', '~> 6.15.0'
 gem 'american_date', '~> 1.1.1'
 gem 'sass', '~> 3.3.0'
 gem 'sass-rails', '~> 5.0.7'
@@ -78,7 +81,7 @@ gem 'retry_block', '~> 1.2.0'
 gem 'aws-sdk', '< 3.0'
 gem 'colorize', '~> 0.8.1'
 gem 'dogstatsd-ruby', '~> 3.2.0'
-gem 'http', '~> 1.0'
+gem 'http', '~> 4.0'
 gem 'robots_tag_parser', '~> 0.1.0', git: 'https://github.com/GSA/robots_tag_parser'
 gem 'loofah', '~> 2.9.0'
 # Locking ref, as later versions (after being renamed & released as "medusa-crawler")

diff --git a/Gemfile.lock b/Gemfile.lock
@@ -39,18 +39,29 @@ GIT
       activesupport (>= 3.2)
       i18n (~> 0.6)
 
+GIT
+  remote: https://github.com/GSA/tweetstream
+  revision: 1feb4d788fe246ac45f7d8a7e427a2119040d33f
+  specs:
+    tweetstream (2.6.1)
+      daemons (~> 1.1)
+      em-http-request (>= 1.1.1)
+      em-twitter (~> 0.3)
+      multi_json (~> 1.3)
+      twitter (~> 5.12)
+
 GIT
   remote: https://github.com/GSA/twitter.git
-  revision: c829ec4c7fbd4e2c52fe6fb7d968fd6926fcd7f5
+  revision: a1379f0bbaf37e92a1ba65bb4c7c11eeb7995877
   branch: 5-stable
   specs:
     twitter (5.17.0)
       addressable (~> 2.3)
       buftok (~> 0.2.0)
       equalizer (= 0.0.10)
       faraday (~> 0.9.0)
-      http (~> 1.0)
-      http_parser.rb (~> 0.6.0)
+      http (~> 4.0)
+      http_parser.rb (~> 0.7.0)
       memoizable (~> 0.4.0)
       naught (~> 1.0)
       simple_oauth (~> 0.3.0)
@@ -395,15 +406,17 @@ GEM
     html_truncator (0.4.2)
       nokogiri (~> 1.5)
     htmlentities (4.3.4)
-    http (1.0.4)
+    http (4.4.1)
       addressable (~> 2.3)
       http-cookie (~> 1.0)
-      http-form_data (~> 1.0.1)
-      http_parser.rb (~> 0.6.0)
+      http-form_data (~> 2.2)
+      http-parser (~> 1.2.0)
     http-cookie (1.0.4)
       domain_name (~> 0.5)
-    http-form_data (1.0.3)
-    http_parser.rb (0.6.0)
+    http-form_data (2.3.0)
+    http-parser (1.2.3)
+      ffi-compiler (>= 1.0, < 2.0)
+    http_parser.rb (0.7.0)
     httparty (0.18.1)
       mime-types (~> 3.0)
       multi_xml (>= 0.5.2)
@@ -487,7 +500,7 @@ GEM
     mysql2 (0.4.10)
     naught (1.1.0)
     net-http-persistent (2.9.4)
-    newrelic_rpm (5.0.0.342)
+    newrelic_rpm (6.15.0)
     nio4r (2.5.8)
     nokogiri (1.11.7)
       mini_portile2 (~> 2.5.0)
@@ -557,7 +570,7 @@ GEM
     rails-dom-testing (2.0.3)
       activesupport (>= 4.2.0)
       nokogiri (>= 1.6)
-    rails-html-sanitizer (1.4.1)
+    rails-html-sanitizer (1.4.2)
       loofah (~> 2.3)
     rails-i18n (5.1.3)
       i18n (>= 0.7, < 2)
@@ -666,7 +679,7 @@ GEM
       rubocop-ast (>= 1.2.0, < 2.0)
       ruby-progressbar (~> 1.7)
       unicode-display_width (>= 1.4.0, < 3.0)
-    rubocop-ast (1.10.0)
+    rubocop-ast (1.11.0)
       parser (>= 3.0.1.1)
     rubocop-performance (1.11.5)
       rubocop (>= 1.7.0, < 2.0)
@@ -749,12 +762,6 @@ GEM
     turbolinks (5.2.1)
       turbolinks-source (~> 5.2)
     turbolinks-source (5.2.0)
-    tweetstream (2.6.1)
-      daemons (~> 1.1)
-      em-http-request (>= 1.1.1)
-      em-twitter (~> 0.3)
-      multi_json (~> 1.3)
-      twitter (~> 5.5)
     twitter-typeahead-rails (0.11.1)
       actionpack (>= 3.1)
       jquery-rails
@@ -848,7 +855,7 @@ DEPENDENCIES
   hashie (~> 3.3.0)
   html_truncator (~> 0.4.2)
   htmlentities (~> 4.3.4)
-  http (~> 1.0)
+  http (~> 4.0)
   i18n-tasks (~> 0.9.19)
   iso8601 (~> 0.10.1)
   jbuilder (~> 2.6.4)
@@ -862,7 +869,7 @@ DEPENDENCIES
   medusa!
   mysql2 (~> 0.4.4)
   net-http-persistent (~> 2.9.3)
-  newrelic_rpm (~> 5.0.0)
+  newrelic_rpm (~> 6.15.0)
   nokogiri (~> 1.11.1)
   omniauth-rails_csrf_protection (~> 0.1.2)
   omniauth_login_dot_gov!
@@ -911,7 +918,7 @@ DEPENDENCIES
   therubyracer (~> 0.12.3)
   truncator (~> 0.1.7)
   turbolinks (~> 5.2.1)
-  tweetstream (~> 2.6.1)
+  tweetstream!
   twitter!
   twitter-typeahead-rails (~> 0.11.1)
   typhoeus (~> 1.3.0)

diff --git a/config/newrelic.yml b/config/newrelic.yml
@@ -6,6 +6,7 @@
   apdex_t: <%= Rails.application.secrets.newrelic[:apdex_t] || 0.5 %>
   capture_params: false
   disable_curb: true
+  backport_fast_active_record_connection_lookup: true
   transaction_tracer:
     enabled: true
     transaction_threshold: apdex_f

diff --git a/config/tweetstream.yml b/config/tweetstream.yml
@@ -0,0 +1,15 @@
+default: &DEFAULT
+  verify_peer: true
+  cert_chain_file: <%= ENV['SSL_CERT_FILE'] || OpenSSL::X509::DEFAULT_CERT_FILE %>
+
+development:
+  <<: *DEFAULT
+
+test:
+  <<: *DEFAULT
+
+production:
+  <<: *DEFAULT
+  # Workaround for "OpenSSL::X509::DEFAULT_CERT_FILE" pointing to a missing file in Ubuntu
+  # https://github.com/actions/virtual-environments/issues/235
+  cert_chain_file: '/etc/ssl/certs/ca-certificates.crt'
diff --git a/features/mobile_searches.feature b/features/mobile_searches.feature
@@ -1,6 +1,8 @@
 # DO NOT ADD NEW TESTS TO THIS FILE!
-# These tests need to be consolidated with those in searches.feature
-# https://cm-jira.usa.gov/browse/SRCH-2110
+# New tests should be added to searches.feature.
+# The separate files are a leftover from the days of the legacy SERP.
+# We are preserving this old file for sake of future git blame-rs.
+
 Feature: Searches using mobile device
   Scenario: Web search
     Given the following Affiliates exist:
@@ -274,20 +276,6 @@ Feature: Searches using mobile device
     And I should see "Cualquier fecha" within the current time filter
     And I should see "Más recientes" within the current sort by filter
 
-  Scenario: Media RSS search
-    Given the following Affiliates exist:
-      | display_name | name          | contact_email    | first_name | last_name |
-      | English site | en.agency.gov | admin@agency.gov | John       | Bar     |
-    And affiliate "en.agency.gov" has the following RSS feeds:
-      | name   | url                              | is_navigable | show_only_media_content |
-      | Images | http://en.agency.gov/feed/images | true         | true                    |
-    And there are 10 image news items for "Images"
-    When I am on en.agency.gov's "Images" news search page
-    And I fill in "Enter your search term" with "image"
-    And I press "Search" within the search box
-    Then I should see exactly "10" image search results
-    And I should see "Powered by Search.gov"
-
   Scenario: Video news search
     Given the following Affiliates exist:
       | display_name | name          | contact_email    | first_name | last_name | locale | youtube_handles         |
@@ -316,6 +304,7 @@ Feature: Searches using mobile device
     And I should see "Previous"
     And I should see a link to "2" with class "pagination-numbered-link"
     And I should see a link to "Next"
+    And I should see "Refine your search"
 
     When I follow "Next"
     Then I should see exactly "20" video search results
@@ -346,31 +335,33 @@ Feature: Searches using mobile device
     Then I should see "Generado por Search.gov"
     And I should see at least "5" video search results
 
-  Scenario: Site search
+  Scenario: Collections search
     Given the following Affiliates exist:
       | display_name | name          | contact_email    | first_name | last_name | locale |
       | English site | en.agency.gov | admin@agency.gov | John       | Bar       | en     |
       | Spanish site | es.agency.gov | admin@agency.gov | John       | Bar       | es     |
 
     And affiliate "en.agency.gov" has the following document collections:
-      | name    | prefixes           |
+      | name    | prefixes            |
       | USA.gov | https://www.usa.gov |
 
     And affiliate "es.agency.gov" has the following document collections:
-      | name            | prefixes                       |
-      | GobiernoUSA.gov | https://gobierno.usa.gov |
+      | name         | prefixes                     |
+      | Gobierno USA | https://www.usa.gov/espanol/ |
 
-    When I am on en.agency.gov's "USA.gov" mobile site search page
+    When I am on en.agency.gov's "USA.gov" docs search page
     And I fill in "Enter your search term" with "gov"
     And I press "Search"
     Then I should see Powered by Bing logo
     And I should see at least "10" web search results
+    And every result URL should match "www.usa.gov"
 
-    When I am on es.agency.gov's "GobiernoUSA.gov" mobile site search page
+    When I am on es.agency.gov's "Gobierno USA" docs search page
     And I fill in "Ingrese su búsqueda" with "gobierno"
     And I press "Buscar"
     Then I should see Generado por Bing logo
     And I should see at least "7" web search results
+    And every result URL should match "www.usa.gov/espanol"
 
   Scenario: Site navigations without dropdown menu
     Given the following Affiliates exist:
@@ -439,7 +430,7 @@ Feature: Searches using mobile device
     And I press "Search"
     And I should see at least "1" web search results
 
-    When I am on en.agency.gov's "Inactive site search" mobile site search page
+    When I am on en.agency.gov's "Inactive site search" docs search page
     Then I should see "Inactive site search" within the SERP active navigation
 
     When I am on en.agency.gov's "Inactive news search" mobile news search page
@@ -488,55 +479,30 @@ Feature: Searches using mobile device
     And I should see at least 1 job posting
     And I should see a link to "More GSA job openings on USAJobs.gov" with url for "https://www.usajobs.gov/Search/Results?a=GS&hp=public"
 
-  Scenario: Searching with matching med topic query
-    Given the following Medline Topics exist:
-      | medline_tid | medline_title     | medline_url                                               | locale | summary_html                                                                                                                                                                                                                                                                                                         |
-      | 1558        | Alcohol           | https://www.nlm.nih.gov/medlineplus/alcohol.html          | en     | <p>If you are like many Americans, you drink alcohol at least occasionally. For many people, moderate drinking is probably safe.  It may even have health benefits, including reducing your risk of certain heart problems.</p>                                                                                      |
-      | 1313        | Underage Drinking | https://www.nlm.nih.gov/medlineplus/underagedrinking.html | en     | <p>It is possible to drink legally and safely - when you're over 21. But if you're under 21, or if you drink too much at any age, alcohol can be especially risky. </p>                                                                                                                                              |
-      | 1732        | Alcohol           | https://www.nlm.nih.gov/medlineplus/spanish/alcohol.html  | es     | <p>Si usted es como muchos estadounidenses, quizás consuma bebidas alcohólicas por lo menos ocasionalmente. Para muchas personas, beber moderadamente probablemente sea sano. Quizá hasta puede tener beneficios para la salud, entre los que se incluye disminuir el riesgo de padecer algunos problemas cardiacos. |
-    And the following Related Medline Topics for "Alcohol" in English exist:
-      | medline_title     | medline_tid | url                                                       |
-      | Underage drinking | 1313        | https://www.nlm.nih.gov/medlineplus/underagedrinking.html |
-    And the following Related Medline Topics for "Alcohol" in Spanish exist:
-      | medline_title | medline_tid | url                                                         |
-      | Alcoholismo   | 1733        | https://www.nlm.nih.gov/medlineplus/spanish/alcoholism.html |
-    And the following Medline Sites exist:
-      | medline_title | locale | title      | url                                                              |
-      | Alcohol       | en     | Alcoholism | http://clinicaltrials.gov/search/open/condition=%22Alcoholism%22 |
-    And the following Affiliates exist:
-      | display_name | name          | contact_email    | first_name |  last_name | locale | is_medline_govbox_enabled |
-      | English site | en.agency.gov | admin@agency.gov | John       | Bar        | en     | true                      |
-      | Spanish site | es.agency.gov | admin@agency.gov | John       | Bar        | es     | true                      |
-    When I am on en.agency.gov's mobile search page
-    And I fill in "Enter your search term" with "alcohol"
-    And I press "Search"
-    Then I should see a link to "Alcohol" with url for "https://www.nlm.nih.gov/medlineplus/alcohol.html" within the med topic govbox
-    And I should see a link to "Underage drinking" with url for "https://www.nlm.nih.gov/medlineplus/underagedrinking.html" within the med topic govbox
-    And I should see a link to "Alcoholism" with url for "http://clinicaltrials.gov/search/open/condition=%22Alcoholism%22" within the med topic govbox
-
-    When I am on es.agency.gov's mobile search page
-    And I fill in "Ingrese su búsqueda" with "alcohol"
-    And I press "Buscar"
-    Then I should see a link to "Alcohol" with url for "https://www.nlm.nih.gov/medlineplus/spanish/alcohol.html" within the med topic govbox
-    And I should see a link to "Alcoholismo" with url for "https://www.nlm.nih.gov/medlineplus/spanish/alcoholism.html" within the med topic govbox
-
   Scenario: Searching with sitelimit
     Given the following Affiliates exist:
       | display_name | name          | contact_email    | first_name | last_name | locale | domains |
       | English site | en.agency.gov | admin@agency.gov | John       | Bar       | en     | .gov    |
       | Spanish site | es.agency.gov | admin@agency.gov | John       | Bar       | es     | .gov    |
+    And affiliate "en.agency.gov" has the following document collections:
+      | name | prefixes                 | is_navigable |
+      | Blog | https://search.gov/blog/ | true         |
     When I am on en.agency.gov's search page with site limited to "usa.gov"
-    And I fill in "Enter your search term" with "gov"
-    And I press "Search"
+    And I search for "gov"
+    Then every result URL should match "usa.gov"
     Then I should see "We're including results for gov from usa.gov only."
     And I should see "Do you want to see results for gov from all sites?"
     When I follow "gov from all sites" within the search all sites row
     Then I should not see "We're including results for gov from usa.gov only."
+    When I follow "Blog" in the search navbar
+    Then I should see at least "1" web search results
+    And every result URL should match "search.gov/blog"
 
     When I am on es.agency.gov's search page with site limited to "usa.gov"
     And I fill in "Ingrese su búsqueda" with "gobierno"
     And I press "Buscar"
-    Then I should see "Los resultados para gobierno son solo de usa.gov."
+    Then every result URL should match "usa.gov"
+    And I should see "Los resultados para gobierno son solo de usa.gov."
     And I should see "¿Quiere ver resultados para gobierno de todos los sitios?"
     When I follow "gobierno de todos los sitios" within the search all sites row
     Then I should not see "Los resultados para gobierno son solo de usa.gov."
@@ -696,7 +662,7 @@ Feature: Searches using mobile device
   Scenario: Web search using Bing engine
     Given the following Affiliates exist:
       | display_name | name          | contact_email    | first_name   | last_name | locale | search_engine | domains |
-      | English site | en.agency.gov | admin@agency.gov | John         | Bar       | en     | BingV6        | .gov    |
+      | English site | en.agency.gov | admin@agency.gov | John         | Bar       | en     | BingV7        | .gov    |
     And affiliate "en.agency.gov" has the following document collections:
       | name    | prefixes            |
       | USA.gov | https://www.usa.gov |

diff --git a/features/responsive_search.feature b/features/responsive_search.feature