Repository navigation
v0.620-beta
Pre-release
Pre-release
- Added an end of script report that gave basic information about the data that was gathered
- Switched the scrape logic from only doing "past week" to only doing "past day"
- Note that in the next version I will make it so that "past week" is the default but if the script sees that there have been previous runs done, ie if all_jobs.json exists, then it will switch to "past 24 hours" so as to ensure that newly posted jobs are not lost in the endless sea of postings
- Updated the
no_resultsxpath to match what LinkedIn has it as currently - Added more job description xpath and css locators to help the script during the content grabbing phase
- Changed the logic of the
max_jobs_per_searchto bemax_jobs_per_search_location. This means that it will grab X valid jobs for each location rather than stopping once it reaches X regardless of how many locations it has gone through. This means a lot more jobs can be scraped so documentation needs to be updated. - Updated
config.yamlandcustomizations_example.yamlto properly put""around strings - Added ability to turn on or off html style reports and where to save said reports
- Cleaned up imports and organized them
- Changed all .75 sleep statements to 1 to help further reduce the chance of getting 429-ed
- Changed
job['content']from string to BeautifulSoup so that if the user chooses html output, the content page will be pretty and easy to read - Added JSON encoder class that will properly output the jobs variable to JSON
- Polished up the
save_job_report_htmlfunction to make it so that the output was more human friendly and less button clicking - Changed the selenium chrome drive setup so that the window always starts maximized in a preferred 1920x1080 size. This helps make sure that elements are not hidden and thus helps prevent a lot of edge cases.
- Added in timers to track the scraping, content grabbing, and data processing sections of the code