Feature/background fetch super admin #100

ethanewing · 2018-07-02T22:45:43Z

Not finished: searchgov_url_fetcher_job still WIP

MothOnMars · 2018-07-03T20:34:19Z

features/admin.feature

@@ -226,7 +226,8 @@ Feature:  Administration
    Then I should see "search.gov/oops"
    And I follow "Fetch"
    And I wait for ajax
-    Then I should see "404"
+    Then I should see "Your URL has been added to the fetching queue"
+    And I follow "Close" within "div.info-message.message"


@ethanewing , remember to keep cucumber features as 'human-readable' as possible. For within steps, you'll need to define one a 'human-readable' page area. See https://github.com/GSA/search-gov/blob/master/features/step_definitions/within_steps.rb

In this case, you might say something like within the first URLs row. I suspect that div.info-message.message isn't the best CSS id for that; let me know if you need help finding a better identifier.

MothOnMars · 2018-07-03T20:38:00Z

spec/jobs/searchgov_url_fetcher_job.rb

+require 'spec_helper'
+
+describe SearchgovUrlFetcherJob do
+  subject(:perform) { SearchgovUrlFetcherJob.perform_now(args[:searchgov_url]) }


Because you aren't using keyword arguments for this job, you can just say:
subject(:perform) { SearchgovUrlFetcherJob.perform_now(searchgov_url) }
...and remove the args variable.

MothOnMars · 2018-07-03T20:38:25Z

spec/jobs/searchgov_url_fetcher_job.rb

+    { searchgov_url: searchgov_url }
+  end
+
+  it_behaves_like 'a searchgov job'


MothOnMars · 2018-07-03T20:41:35Z

spec/jobs/searchgov_url_fetcher_job.rb

+
+describe SearchgovUrlFetcherJob do
+  subject(:perform) { SearchgovUrlFetcherJob.perform_now(args[:searchgov_url]) }
+  let!(:searchgov_url) { SearchgovUrl.create(url: 'https://agency.gov/') }


are you sure you need the bang method here?

I think I do, as the tests don't seem to pass if I don't have it. Maybe there's a better way of organizing it?

MothOnMars · 2018-07-03T20:44:24Z

spec/jobs/searchgov_url_fetcher_job.rb

+    it 'fetches a searchgov_url' do
+      # byebug
+      perform
+      expect(searchgov_url.last_crawl_status).to_not be(nil)


We don't really need the record to be fetched in our test, nor do we need the test to know anything about what fetch does (such as changing the last_crawl_status). A cleaner (and faster) way to test this would be to use a mock, i.e.:

expect(searchgov_url).to receive(:fetch) perform

I've changed this example to what you've suggested. I'm not sure if you meant to scrap it completely and were just giving me an example of how it should be done were we trying to test it, so if you meant for me to get rid of it I can do that real quick.

The change you made is what I was looking for 👍

MothOnMars · 2018-07-03T22:06:14Z

spec/jobs/searchgov_url_fetcher_job.rb

@@ -1,7 +1,7 @@
 require 'spec_helper'

 describe SearchgovUrlFetcherJob do
-  subject(:perform) { SearchgovUrlFetcherJob.perform_now(args[:searchgov_url]) }
+  subject(:perform) { SearchgovUrlFetcherJob.perform_now(searchgov_url) }
  let!(:searchgov_domain) { SearchgovDomain.create(domain: 'agency.gov', status: '200') }
  let(:searchgov_url) { SearchgovUrl.create(url: 'https://agency.gov/') }
  let(:args) do


Per my earlier comment, you can remove the args variable, which you are no longer using.

I believe the args variable is necessary for the shared example, it_behaves_like 'a searchgov job', to work correctly?

Ah, you are correct. I was thinking you could just pass the searchgov_url record as the argument, rather than a hash, because you are not using keyword arguments.

However...let's change that. I am not always consistent about using keyword args, but I do like how explicit they are about complaining when you are missing an argument. Using keyword args would also be consistent with the other job arguments:

$ gg 'def perform' app/jobs/ app/jobs/searchgov_domain_indexer_job.rb: def perform(searchgov_domain:, delay:) app/jobs/searchgov_domain_preparer_job.rb: def perform(searchgov_domain:) app/jobs/sitemap_indexer_job.rb: def perform(searchgov_domain:)

So, instead of:
def perform(searchgov_url)

You'd use:
def perform(searchgov_url:)

And the way to initiate that job would be:
SearchgovUrlFetcherJob.perform_now(searchgov_url: searchgov_url)

It's a little more wordy than just SearchgovUrlFetcherJob.perform_now(searchgov_url), but it gives us more flexibility in case we need to add more args down the line. Sorry for making this specification late in the game; I was thinking I didn't mind the inconsistency with the other jobs, but I do think we should make this change.

Gotcha. So I now have it that perform uses a keyword argument. Since the args variable is still necessary, would it be cleaner to write SearchgovUrlFetcherJob.perform_now(args) in the subject declaration in the spec?

MothOnMars · 2018-07-03T22:07:41Z

spec/jobs/searchgov_url_fetcher_job.rb

@@ -1,7 +1,7 @@
 require 'spec_helper'

 describe SearchgovUrlFetcherJob do
-  subject(:perform) { SearchgovUrlFetcherJob.perform_now(args[:searchgov_url]) }
+  subject(:perform) { SearchgovUrlFetcherJob.perform_now(searchgov_url) }
  let!(:searchgov_domain) { SearchgovDomain.create(domain: 'agency.gov', status: '200') }


I don't see this variable used anywhere...?

I've removed this variable. It seemed to be necessary before when I was trying to test the fetching of a record, but now that I've changed that to what you recommended it isn't needed anymore.

MothOnMars · 2018-07-03T22:16:50Z

features/step_definitions/within_steps.rb

@@ -60,7 +60,8 @@
  'in the twitter govbox' => '#twitter_govbox',
  'in the site header' => '.l-site-header',
  'in the active site main navigation' => '.l-site-nav.main .active',
-  'in the active site sub navigation' => '.l-site-nav.sub .active'
+  'in the active site sub navigation' => '.l-site-nav.sub .active',
+  'in the SearchgovDomain URLs table' => 'tbody.messages #as_admin__searchgov_urls-messages'


@ethanewing , that selector appears to be for just the section that holds the flash messages for that table. As we may want to use the in the SearchgovDomain URLs table selector more generically, I'd try the as_admin__searchgov_urls-active-scaffold div.

MothOnMars · 2018-07-13T16:41:02Z

spec/controllers/searchgov_urls_controller_spec.rb

-        to have_enqueued_job(SearchgovUrlFetcherJob).on_queue("searchgov")
+      it 'enqueues a searchgov_url_fetcher_job to the searchgov queue' do
+        expect{ post :fetch, params }.
+          to have_enqueued_job(SearchgovUrlFetcherJob).on_queue("searchgov")


@ethanewing , TDD is helpful for bug fixes as well. This test doesn't actually test this change, because it's not checking the job arguments: e7b3adb.

Would that test to make sure it only takes named arguments go in searchgov_url_fetcher_job_spec.rb since that test is specifically about the functionality of the perform function within the model? I've already got a test in that spec that tests that the perform is invalid if it isn't given an argument -- should I just add another that tests that it's invalid if it's not given a named argument?

This test is covering the bug you fixed with e7b3adb. It ensures we're passing the args correctly to the job. You can do that by specifying the args in your test:

expect{ post :fetch, params }. to have_enqueued_job(SearchgovUrlFetcherJob).on_queue("searchgov"). with(searchgov_url: searchgov_url)

It couldn't hurt to also make your searchgov_url_fetcher_job_spec.rb spec more specific, by testing that it requires asearchgov_url keyword argument.

MothOnMars · 2018-07-13T16:42:28Z

spec/controllers/searchgov_urls_controller_spec.rb

-    #   post :fetch, params
-    #   p response.body
-    # end
+  context 'when logged in as an admin' do


@ethanewing , you should be able to use the shared 'super admin logged in' context for this:
https://github.com/GSA/search-gov/blob/master/spec/support/shared_sites_controller_shared_behavior.rb#L46
https://relishapp.com/rspec/rspec-core/docs/example-groups/shared-context

MothOnMars · 2018-07-13T17:07:24Z

spec/controllers/searchgov_urls_controller_spec.rb

+  fixtures :users
+
+  let(:admin) { users(:affiliate_admin) }
+  let(:searchgov_url) { SearchgovUrl.create(url: 'agency.gov/test') }


Try to use bang methods in your tests, to ensure that your setup steps don't fail silently: SearchgovUrl.create!(url: 'agency.gov/test')

MothOnMars · 2018-07-13T17:07:46Z

spec/jobs/searchgov_url_fetcher_job_spec.rb

+
+describe SearchgovUrlFetcherJob do
+  subject(:perform) { SearchgovUrlFetcherJob.perform_now(args) }
+  let!(:searchgov_url) { SearchgovUrl.create(url: 'https://agency.gov/') }


See above comment re. bang methods

MothOnMars · 2018-07-13T18:18:02Z

spec/jobs/searchgov_url_fetcher_job_spec.rb

@@ -15,6 +15,11 @@
        to raise_error(ArgumentError)
    end

+    it 'must have a named argument' do
+      expect { SearchgovUrlFetcherJob.perform_now searchgov_url }.
+        to raise_error(ArgumentError)


The more specific you can be in your specs, the better. Here, you can specify the error message as well as the error class:
expect....to raise_error(ArgumentError, 'missing keyword: searchgov_url')

MothOnMars · 2018-07-13T18:24:26Z

spec/controllers/searchgov_urls_controller_spec.rb

  let(:params) do
    { association: 'searchgov_urls', parent_scaffold: 'admin/searchgov_domains',
    searchgov_domain_id: searchgov_url.searchgov_domain.id, id: searchgov_url.id }
  end

-  context 'when logged in as an admin' do
+  include_context 'super admin logged in' do


MothOnMars · 2018-07-13T18:24:34Z

spec/controllers/searchgov_urls_controller_spec.rb


      it 'enqueues a searchgov_url_fetcher_job to the searchgov queue' do
        expect{ post :fetch, params }.
-          to have_enqueued_job(SearchgovUrlFetcherJob).on_queue("searchgov")
+          to have_enqueued_job(SearchgovUrlFetcherJob).on_queue("searchgov").
+          with(searchgov_url: searchgov_url)


MothOnMars · 2018-07-13T21:33:41Z

spec/controllers/searchgov_urls_controller_spec.rb

+
+  include_context 'super admin logged in' do
+    describe '#fetch' do
+


@ethanewing - I just noticed this. Please remove this extra line break

MothOnMars · 2018-07-13T21:35:07Z

spec/controllers/searchgov_urls_controller_spec.rb

+describe Admin::SearchgovUrlsController do
+  fixtures :users
+
+  let(:admin) { users(:affiliate_admin) }


@ethanewing , now that you're using the shared context, this line can be removed

MothOnMars · 2018-07-13T21:57:49Z

spec/controllers/searchgov_urls_controller_spec.rb

@@ -3,7 +3,6 @@
 describe Admin::SearchgovUrlsController do
  fixtures :users


whoops - I missed this too! You can remove this line and the line break following.

My specs are failing when I comment that line out, unless I transfer that line (fixtures :users) into the shared context itself.

As the users fixtures are used throughout the various shared contexts in spec/support/shared_sites_controller_shared_behavior.rb, it makes sense to add the fixtures :users line to the top of that file.

MothOnMars · 2018-07-16T18:42:15Z

@ethanewing , please move spec/controllers/searchgov_urls_controller_spec.rb to spec/controllers/admin/searchgov_urls_controller_spec.rb

MothOnMars reviewed Jul 3, 2018

View reviewed changes

ethanewing added 16 commits July 5, 2018 14:14

created searchgov_url_fetcher_job; edited searchgov_urls_controller

b9707f7

"fetches url" and shared example working for urlfetcherspec

1abcada

rearranged searchgov_url_fetcher_job spec

c6ae4ef

edited cucumber test for admin/searchgov_domains to test for flash msg

c0f3eda

not passing; socket error

51e51dd

sgov_url_fetcher_job specs now passing

99c161e

revised subject(:perform) declaration

e140978

updated cucumber test; wrote new "within" clause

defb0ae

generalized 'searchgov domain urls' selector

5029a1c

revised spec

fe32b0d

'perform' now takes keyword argument

503ca83

cleaned up 'subject' declaration

64716b6

reverted to old test write-up

887b64e

correctly named searchgov_url_fetcher_job spec file

852efc0

revised it 'fetches a searchov_url' example

6e9edb4

fixed controller to use named argument when calling fetch

e7b3adb

MothOnMars self-assigned this Jul 12, 2018

ethanewing added 2 commits July 12, 2018 13:47

tentative searchgov_urls_controller_spec

fe747cb

all passing

3c80e46

MothOnMars reviewed Jul 13, 2018

View reviewed changes

ethanewing added 4 commits July 13, 2018 13:45

added activate_authlogic to shared example 'super admin is logged in'

d524f69

uses shared example; test arguments of serachgov_url_fetcher_job

481a18f

uses bang method for 'create' now

b417c11

added test to make sure 'perform' uses a named argument

b2e1170

MothOnMars reviewed Jul 13, 2018

View reviewed changes

passing revision

fd1ce47

MothOnMars reviewed Jul 13, 2018

View reviewed changes

revise

6cf83bb

MothOnMars reviewed Jul 13, 2018

View reviewed changes

declare 'fixtures :users' in shared examples file

e1c5ddb

moved searchgov_urls_controller_spec to spec/controllers/admin

fa910cb

MothOnMars merged commit d4bdb0a into GSA:master Jul 16, 2018

MothOnMars deleted the feature/background_fetch_super_admin branch July 16, 2018 19:19

MothOnMars mentioned this pull request Jul 16, 2018

The fetch link in Super Admin should enqueue a background job #98

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/background fetch super admin #100

Feature/background fetch super admin #100

ethanewing commented Jul 2, 2018

MothOnMars Jul 3, 2018

MothOnMars Jul 3, 2018

MothOnMars Jul 3, 2018

MothOnMars Jul 3, 2018

MothOnMars Jul 5, 2018

ethanewing Jul 5, 2018

MothOnMars Jul 3, 2018

ethanewing Jul 5, 2018

MothOnMars Jul 5, 2018

MothOnMars Jul 3, 2018

ethanewing Jul 5, 2018

MothOnMars Jul 5, 2018

ethanewing Jul 5, 2018 •

edited

Loading

MothOnMars Jul 5, 2018

MothOnMars Jul 3, 2018

ethanewing Jul 5, 2018

MothOnMars Jul 3, 2018

MothOnMars Jul 13, 2018

ethanewing Jul 13, 2018

MothOnMars Jul 13, 2018

MothOnMars Jul 13, 2018

MothOnMars Jul 13, 2018

MothOnMars Jul 13, 2018

MothOnMars Jul 13, 2018

MothOnMars Jul 13, 2018

MothOnMars Jul 13, 2018

MothOnMars Jul 13, 2018

MothOnMars Jul 13, 2018

MothOnMars Jul 13, 2018

ethanewing Jul 13, 2018

MothOnMars Jul 13, 2018

MothOnMars commented Jul 16, 2018


		include_context 'super admin logged in' do
		describe '#fetch' do

		@@ -3,7 +3,6 @@
		describe Admin::SearchgovUrlsController do
		fixtures :users

Feature/background fetch super admin #100

Feature/background fetch super admin #100

Conversation

ethanewing commented Jul 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ethanewing Jul 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MothOnMars commented Jul 16, 2018

ethanewing Jul 5, 2018 •

edited

Loading