WorkableFormFiller and seedfile updates #224

daniel-sussman · 2024-07-31T21:02:49Z

WorkableFormFiller is pretty solid now.
Added a rakefile for testing purposes. To test FormFiller, type 'rake ff:[ats_name]', e.g. rake ff:greenhouse.
The seedfile now includes Ashby, Bamboo, Greenhouse and Workable jobs.

Ches-ctrl

Looks good overall

Ches-ctrl · 2024-08-01T05:35:49Z

app/models/concerns/ats/ashbyhq/job_details.rb

@@ -18,7 +18,7 @@ def job_url_api(url_api, ats_identifier, _job_id)
        "#{url_api}#{ats_identifier}?includeCompensation=true"
      end

-      def fetch_url(job_data)
+      def fetch_url(job_data, _company_id)


What is _company_id doing here?

Nothing useful! It was a lazy solution to the problem that only BambooHR needs info extrinsic to the all_jobs json in order to build the posting_url, but I've found a workaround and removed it.

Ches-ctrl · 2024-08-01T05:37:13Z

app/models/concerns/ats/bamboohr/job_details.rb

@@ -18,6 +18,10 @@ def fetch_id(job_data)
        job_data['id']
      end

+      def fetch_url(job_data, company_id)


What is the purpose/context of the fetch_url method again?

Ches-ctrl · 2024-08-01T05:37:58Z

storage/csv/unparseable_urls.csv

What is the process that keeps outputting unparseable_urls?

Think it's the DevITJobs updater

Ches-ctrl · 2024-08-01T05:40:18Z

app/services/importer/get_relevant_job_urls.rb

-      data&.each do |job|
-        title, location = @ats.fetch_title_and_location(job)
-        relevant_jobs << @ats.fetch_url(job) if relevant?(title, location)
+      data&.each do |job_data|


Just wondering if there is a better way to filter relevant jobs whilst we look at this. @daniel-sussman / @patanj101 - any thoughts on how else we might only draw in relevant jobs from each company's API endpoint?

One thought is that we could just pull all the jobs at this point and then add the filter to the query later. I'd be interested in what effect this had on performance - you don't have to check whether each job is relevant, but you do have to import more jobs.

Ches-ctrl

2nd review

Ches-ctrl · 2024-08-02T05:52:55Z

app/models/concerns/ats/bamboohr/job_details.rb

@@ -18,6 +18,10 @@ def fetch_id(job_data)
        job_data['id']
      end

+      def fetch_posting_url(job_data, company_id)
+        url_base.sub('XXX', company_id) + job_data['id']


We should convert in due course to proper variable names e.g. ${company_id}

Ches-ctrl · 2024-08-02T05:53:14Z

app/services/importer/get_relevant_job_urls.rb

@@ -1,9 +1,13 @@
 module Importer
+  # This is a helper class that's called from the seedfile. Its purpose is to receive a list of ats_identifiers and return a list of posting urls for relevant jobs to seed.


Very helpful - cheers

Ches-ctrl · 2024-08-02T05:53:34Z

app/services/importer/get_relevant_job_urls.rb

      end
      relevant_jobs
    end
+
+    def additional_args


What's the story here?

Okay, I made the name more explanatory and added a comment.

…g companies called 'j'

…okie

Ches-ctrl

3rd Review - great stuff Dan

Ches-ctrl · 2024-08-06T06:37:13Z

Gemfile

@@ -100,6 +100,7 @@ gem 'rails-html-sanitizer'
 # HTTP Clients
 # gem 'httparty' # Installed as a dependency via Avo
 # gem 'faraday' # Installed as a dependency via Cloudinary
+gem 'faraday-follow_redirects', '~> 0.3.0'


Presumably does what it says on the tin - easiest way of achieving this?

Yep, think this is easiest if we want to handle all requests with Faraday rather than HTTParty.

Ches-ctrl · 2024-08-06T06:39:35Z

app/models/concerns/ats/workable/job_details.rb

@@ -12,6 +12,12 @@ def fetch_id(job_data)
        job_data['shortcode']
      end

+      def fetch_posting_url(job_data, company_id)
+        # build long-form posting url that includes company_id


If you put the comment above the method then it will generate docs correctly. I like having these in and would be keen to include these on more methods. FAO also @patanj101

Ches-ctrl · 2024-08-06T06:40:10Z

app/models/concerns/ats/workable/parse_url.rb

@@ -3,8 +3,8 @@ module Workable
    module ParseUrl
      def parse_url(url, _saved_ids = nil)
        regex_formats = [
-          %r{https://apply\.workable\.com/([^/]+)(?:/j/([^/]+))?},
-          %r{https://apply\.workable\.com/api/v1/accounts/([^/]+)(?:/jobs/([^/?]+)(?:\?.*)?)?}
+          %r{^https://apply\.workable\.com/(?:([^/]+)/)?j/([^/]+)},


Impact of changes here? Does this also handle HTTP?

This should prevent the parser bug that made CompanyCreator try to create Workable companies named 'j'. It only handles https, not http (none of the parsing methods do).

Ches-ctrl · 2024-08-06T06:41:26Z

app/services/applier/ashby_form_filler.rb

@@ -43,158 +38,5 @@ def hidden_element
    def response_field
      find(:css, "label[for='#{@locator}']").ancestor('fieldset', match: :first)
    end
-


This is no longer required presumably as we have the actual payload? Is it all working?

I just moved this from the children to the parent class, actually. Will implement a functional #click_submit_button method once we're finished with development testing.

Ches-ctrl · 2024-08-06T06:42:13Z

app/services/applier/form_filler.rb

@@ -45,15 +45,16 @@ def attach_file_to_application
      attach_file(@locator, @filepath)


Can we add comments to this file please

Ches-ctrl · 2024-08-06T06:56:11Z

lib/tasks/admin.rake

+    # apply_url = 'https://adobe.wd5.myworkdayjobs.com/en-US/external_experienced/job/Sr-Machine-Learning-Engineer--Adobe-Firefly_R147574/apply'
+    apply_url = 'https://maxar.wd1.myworkdayjobs.com/en-US/MAXAR/job/Senior-Automation-Engineer_R21029/apply'
+    Importer::GetWorkdayFields.call(apply_url)
+  end


Can you add one of these for each ATS integration Dan?

Also can we add some validation condition(s) so that we know it's worked? e.g. Capybara checks all input fields are filled. Or checks a particular input field contains the value in AQS. Or we could deliberately not fill one of the required fields and click the submit button - if the form returns >1 error then there is an issue we haven't noticed

Could also add similar functionality in the rspec folder in due course?

Yep, the formfiller ones are all in the ff.rake file. I've been using the admin.rake file just for convenience of testing during development (restoring the file before pushing to main).

For classes that rely on Capybara, I'm not sure you want me to write rspec tests because they'll be slow. The rakefile works well for informal testing in development, where I give Capybara sleep instructions so I can verify visually that the form's correct.

Ches-ctrl · 2024-08-06T06:56:46Z

lib/tasks/ff.rake

+  group[response]
+end
+
+ASHBY_PAYLOADS =


Can we have the payloads sit in a separate file so that this is a bit cleaner?

Ches-ctrl · 2024-08-06T06:57:15Z

lib/tasks/ff.rake

+  end
+end
+
+def prompt_user(group)


Generally this logic would sit in a service class rather than in the rake file for cleanness - can you move?

Ches-ctrl · 2024-08-06T06:57:38Z

lib/tasks/ff.rake

@@ -0,0 +1,1076 @@
+namespace :ff do


Similarly with this testing @patanj101

Really nice solution btw Dan

Ches-ctrl · 2024-08-06T06:59:12Z

spec/models/company_jobs_fetcher_spec.rb

@@ -3,7 +3,7 @@
 RSpec.describe CompanyJobsFetcher do
  context "with an existing company" do
    before do
-      allow($stdout).to receive(:write) # suppresses terminal clutter
+      # allow($stdout).to receive(:write) # suppresses terminal clutter


What changed here?

It was a lazy test that relied on capturing what's printed to the console ($stdout). One problem is that diverting $stdout means it doesn't display anymore, so you have no info on why the test failed. I changed this to evaluate method result rather than its printed output.

Ches-ctrl

Approved - cheers both

added Bamboo jobs to seedfile

b12bb05

Ches-ctrl assigned daniel-sussman Aug 1, 2024

Ches-ctrl reviewed Aug 1, 2024

View reviewed changes

daniel-sussman added 4 commits August 1, 2024 10:50

added comments for legibility

e120a55

Charlie feedback: added comments and minor refactor

ab6f3df

WorkableFormFiller can handle education questions

45f682d

bugfix to #handle_date

3c9567f

Ches-ctrl reviewed Aug 2, 2024

View reviewed changes

WorkableFormFiller can handle education and experience questions

af09d15

daniel-sussman changed the title ~~added Bamboo jobs to seedfile~~ WorkableFormFiller and seedfile updates Aug 2, 2024

daniel-sussman added 11 commits August 2, 2024 15:18

added Workable to seedfile, fixed Workable parser so it stops creatin…

e6ee9ec

…g companies called 'j'

added ff rakefile

9bdbdac

minor bugfix to rakefile

681c3f8

Updated Workable#get_application_question_set

51b650a

Updated it properly this time.

4b3ac37

fixed namespace issue

210b39f

GetWorkdayFields can get fields when no login required

1056b87

fixed job_creation bug with Workable

009f34e

WorkdayCookieFetcher is basically working

3f13e81

WorkdayCookieFetcher can create a user account in order to fetch a co…

b83411f

…okie

caches authentication cookie for 1 hour

7b7d6f3

Ches-ctrl requested changes Aug 6, 2024

View reviewed changes

daniel-sussman added 2 commits August 6, 2024 12:28

Charlie feedback

514f9cf

removed Workday stuff

9aec373

Ches-ctrl approved these changes Aug 6, 2024

View reviewed changes

Ches-ctrl merged commit 4788cb5 into main Aug 6, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WorkableFormFiller and seedfile updates #224

WorkableFormFiller and seedfile updates #224

daniel-sussman commented Jul 31, 2024 •

edited

Loading

Ches-ctrl left a comment

Ches-ctrl Aug 1, 2024

daniel-sussman Aug 1, 2024

Ches-ctrl Aug 1, 2024

Ches-ctrl Aug 1, 2024

daniel-sussman Aug 1, 2024

Ches-ctrl Aug 1, 2024

Ches-ctrl left a comment

Ches-ctrl Aug 2, 2024

Ches-ctrl Aug 2, 2024

Ches-ctrl Aug 2, 2024

daniel-sussman Aug 2, 2024

Ches-ctrl left a comment •

edited

Loading

Ches-ctrl Aug 6, 2024

daniel-sussman Aug 6, 2024

Ches-ctrl Aug 6, 2024

daniel-sussman Aug 6, 2024

Ches-ctrl Aug 6, 2024

daniel-sussman Aug 6, 2024

Ches-ctrl Aug 6, 2024

daniel-sussman Aug 6, 2024

Ches-ctrl Aug 6, 2024

daniel-sussman Aug 6, 2024

Ches-ctrl Aug 6, 2024

Ches-ctrl Aug 6, 2024

daniel-sussman Aug 6, 2024

Ches-ctrl Aug 6, 2024

Ches-ctrl Aug 6, 2024

Ches-ctrl Aug 6, 2024

Ches-ctrl Aug 6, 2024

Ches-ctrl Aug 6, 2024

daniel-sussman Aug 6, 2024

Ches-ctrl left a comment

		@@ -1,9 +1,13 @@
		module Importer
		# This is a helper class that's called from the seedfile. Its purpose is to receive a list of ats_identifiers and return a list of posting urls for relevant jobs to seed.

		@@ -45,15 +45,16 @@ def attach_file_to_application
		attach_file(@locator, @filepath)

WorkableFormFiller and seedfile updates #224

WorkableFormFiller and seedfile updates #224

Conversation

daniel-sussman commented Jul 31, 2024 • edited Loading

Ches-ctrl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ches-ctrl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ches-ctrl left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ches-ctrl left a comment

Choose a reason for hiding this comment

daniel-sussman commented Jul 31, 2024 •

edited

Loading

Ches-ctrl left a comment •

edited

Loading