Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Company crawler #178

Draft
wants to merge 54 commits into
base: main
Choose a base branch
from
Draft

Company crawler #178

wants to merge 54 commits into from

Conversation

matt-manes
Copy link
Collaborator

Implements overrides for spider components to modify url crawl order and company_crawler core functionality.

@matt-manes matt-manes requested a review from Ches-ctrl May 31, 2024 20:11
@matt-manes
fix: remove leading space from paycom url_identifier
4786cd1
@matt-manes
refactor: load ats stubs from csv
5061add
@matt-manes
perf: update ats stubs
6c821ab
@matt-manes
fix: fix failing scan when url redirects to 'www.' version
93acbc6
@matt-manes
refactor: add number of urls scanned and scan length to end of scan o…
8490990 
…utput
@matt-manes
refactor: include in Spidr module namespace
e7cace4
@matt-manes
refactor: extract base Crawler class
73d983d
@matt-manes
Initial commit
47c74d9
@matt-manes
chore: fix comment typo
7aa3125
@matt-manes
chore: remove debugging 'puts' statement
dd3e1d0
@matt-manes
refactor: move status code print out to Crawler.crawl()
29d6523
@matt-manes
refactor: rename load_priority_stubs to load_stubs
c9fbec5
@matt-manes
Initial commit
e65392c
@matt-manes
refactor: move files from app/services to lib directory and add `…
52ed607 
…Crawlers` module namespace
@matt-manes
chore: add EasyApply and Flair HR entries
ec6f9e0
@matt-manes
refactor: add XXX placeholders to url entries that had them implied
aea04a0
@matt-manes
chore: add missing url_base fields from jobglob data
67c52f8
@matt-manes
refactor: replace XXX placehoders with named placehoders
f70ad2b
Ches-ctrl
Ches-ctrl requested changes Jun 26, 2024
View reviewed changes
Copy link
Owner

@Ches-ctrl Ches-ctrl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know what you think of these changes

lib/crawlers/base_crawler.rb Outdated Show resolved Hide resolved
lib/crawlers/base_crawler.rb Outdated Show resolved Hide resolved
lib/crawlers/base_crawler.rb Outdated Show resolved Hide resolved
lib/crawlers/company_crawler/company_crawler.rb Outdated Show resolved Hide resolved
lib/crawlers/company_crawler/company_crawler.rb Outdated Show resolved Hide resolved
lib/crawlers/company_crawler/company_list_crawler.rb Outdated Show resolved Hide resolved
lib/crawlers/netzero_crawler/netzero_crawler.rb Outdated Show resolved Hide resolved
lib/priority_spidr/priority_agent.rb Outdated Show resolved Hide resolved
Gemfile Outdated Show resolved Hide resolved
Gemfile Show resolved Hide resolved
matt-manes and others added 25 commits June 26, 2024 15:42
@matt-manes
refactor: remove unneccesary requires
d33f215
@matt-manes
refactor: change output file name generation and location
e23d09a
@matt-manes
refactor: revert changes to url_base column format and instead an e…
2bff384 
…xtra `board_template_url` column
@matt-manes
Merge branch 'main' into company-crawler
72c2bfe
@matt-manes
fix: rerun bundle install after removing pathname gem
c57ef1a
@matt-manes
fix: fix calling super with args style violation
ce77d84
@matt-manes
fix: fix uninitialized constant error in pr checks
78da304
@matt-manes
fix: remove stray relative imports
f8414dc
@matt-manes
fix: change spidr entry point to PrioritySpidr module
33e7928
@matt-manes
refactor: move PriorityAgent to Crawlers module and remove `Prior…
d403c63 
…itySpidr`
@matt-manes
Init commit
d43f1f4
@matt-manes
fix: edit various typos and errored patterns
28437d3
@matt-manes
perf: improve board url from job url extraction
3f2716c
@matt-manes
tests: add tests for most ats systems that have documented template b…
0e8a17a 
…oard urls
@matt-manes
tests: remove redundancies
fdd0474
@matt-manes
perf: add bamboohr translator for their 'js/embed.js' urls
6355e55
@matt-manes
perf: add board extractor usage to crawl workflow
a6b521d
@matt-manes
tests: add bamboohr crawl test
2510133
@matt-manes
refactor: pass agent instance to member variable of the same name t…
d03752f 
…o avoid needing to pass it around as an argument
@matt-manes
feat: add post crawl chores base method
95aa1fa
@matt-manes
perf: remove duplicate urls from @hits at the end of the crawl
398aaab
@matt-manes
refactor: trim bamboohr identifier
72c5e04
@matt-manes
chore: check in default company list
64c2ba0
@matt-manes
refactor: add args for starting point and number of crawls to `crawl_…
d012caa 
…list`
@matt-manes
feat: add rake task
0710b3b
@matt-manes matt-manes requested a review from Ches-ctrl July 5, 2024 21:48
@Ches-ctrl Ches-ctrl added paused for review For review by CC and removed paused labels Jul 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Ches-ctrl Ches-ctrl Awaiting requested review from Ches-ctrl

Requested changes must be addressed to merge this pull request.

Assignees

@matt-manes matt-manes

Labels
for review For review by CC
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@matt-manes @Ches-ctrl