Skip to content

v0.620-beta

Pre-release
Pre-release

Choose a tag to compare

@JustOneMoreDog JustOneMoreDog released this 05 Jun 16:57
274d3f0
  • Added an end of script report that gave basic information about the data that was gathered
  • Switched the scrape logic from only doing "past week" to only doing "past day"
    • Note that in the next version I will make it so that "past week" is the default but if the script sees that there have been previous runs done, ie if all_jobs.json exists, then it will switch to "past 24 hours" so as to ensure that newly posted jobs are not lost in the endless sea of postings
  • Updated the no_results xpath to match what LinkedIn has it as currently
  • Added more job description xpath and css locators to help the script during the content grabbing phase
  • Changed the logic of the max_jobs_per_search to be max_jobs_per_search_location. This means that it will grab X valid jobs for each location rather than stopping once it reaches X regardless of how many locations it has gone through. This means a lot more jobs can be scraped so documentation needs to be updated.
  • Updated config.yaml and customizations_example.yaml to properly put "" around strings
  • Added ability to turn on or off html style reports and where to save said reports
  • Cleaned up imports and organized them
  • Changed all .75 sleep statements to 1 to help further reduce the chance of getting 429-ed
  • Changed job['content'] from string to BeautifulSoup so that if the user chooses html output, the content page will be pretty and easy to read
  • Added JSON encoder class that will properly output the jobs variable to JSON
  • Polished up the save_job_report_html function to make it so that the output was more human friendly and less button clicking
  • Changed the selenium chrome drive setup so that the window always starts maximized in a preferred 1920x1080 size. This helps make sure that elements are not hidden and thus helps prevent a lot of edge cases.
  • Added in timers to track the scraping, content grabbing, and data processing sections of the code