v4.21

• Updated license
• Small bug fix with EOL characters

v4.2

• Variety of small bug fixes

Full Changelog: v4.1...v4.2

v4.1

• See #14 for list of changes

v4.0

No longer babies first Python project

v3.0

Merge pull request #10 from JustOneMoreDog/flask

Implementing Flask

v2.0

• Now re-written to be run as a container
• v3.0 will have Web UI Front End

v1.0

• Added the ability to search for remote positions
• General code cleanup
• Increased logging
• Adjusted back-off sleeps to be less aggressive

v0.622-beta

• Fixed HTML export not creating the required files and or folders
• Updated validate_page function logic to handle HTTP Error Code 429

beta-v621

• Added Past Month button to the config file
• Changed the scraping logic yet again. It will keep scraping jobs until it has enough valid jobs or has run out of jobs to scrape. It will now also always perform one loop of the scrape for every location even if we are over the threshold. This ensures that we always pull something from every location
• Fixed requirements.txt
• In regards to main.py
  • Fix logs directory not being made with the right name
  • Added a whole bunch more of logging. This is in an effort to make troubleshooting a lot easier. I want the user to have all the information I need to help them already ready to go
  • Added the search to our output. Now we can see which searches are yielding which results (if any)
  • Added further protection against LinkedIn's bot detection redirects
  • Change the timespan logic
    • The user no longer says if they want to do past day, week, or month. It will now always start with day and keep going until it gets enough valid jobs. However, if it runs out of jobs and does not have enough, it will re-do the search but this time with past week and then past month. As soon as it gets enough jobs from past week or month it will break out and not continue.
  • Added further protections against erroring out when there are no results
  • Added stat and log tracking for how many duplicates, previously found, and dud job postings we find. This will help the user understand why this script takes so long to run
  • Added more safeguards
  • Added a post script report that dumps all the stats about the run
• Figured out exactly what I needed in order to setup cron and now can add it to documentation

v0.620-beta

• Added an end of script report that gave basic information about the data that was gathered
• Switched the scrape logic from only doing "past week" to only doing "past day"
  • Note that in the next version I will make it so that "past week" is the default but if the script sees that there have been previous runs done, ie if all_jobs.json exists, then it will switch to "past 24 hours" so as to ensure that newly posted jobs are not lost in the endless sea of postings
• Updated the no_results xpath to match what LinkedIn has it as currently
• Added more job description xpath and css locators to help the script during the content grabbing phase
• Changed the logic of the max_jobs_per_search to be max_jobs_per_search_location. This means that it will grab X valid jobs for each location rather than stopping once it reaches X regardless of how many locations it has gone through. This means a lot more jobs can be scraped so documentation needs to be updated.
• Updated config.yaml and customizations_example.yaml to properly put "" around strings
• Added ability to turn on or off html style reports and where to save said reports
• Cleaned up imports and organized them
• Changed all .75 sleep statements to 1 to help further reduce the chance of getting 429-ed
• Changed job['content'] from string to BeautifulSoup so that if the user chooses html output, the content page will be pretty and easy to read
• Added JSON encoder class that will properly output the jobs variable to JSON
• Polished up the save_job_report_html function to make it so that the output was more human friendly and less button clicking
• Changed the selenium chrome drive setup so that the window always starts maximized in a preferred 1920x1080 size. This helps make sure that elements are not hidden and thus helps prevent a lot of edge cases.
• Added in timers to track the scraping, content grabbing, and data processing sections of the code