Skip to content
A web application which scrapes several German real estate listing websites and visualizes offers in a map.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.vscode
app
images
scraper
transport
utils
.bowerrc
.eslintrc
.gitignore
README.md
bower.json
config.template.js
index.js
package-lock.json
package.json

README.md

Wohnung Scraper

This tool scrapes several online real estate listing websites and visualizes offers in a map.

Screenshot

Screenshot

Features:

  • new offers are scraped regularly
  • properties like the price, size and other stuff (balcony available) are extracted, parsed and stored
  • addresses are converted into geo coordinates
  • existing offers are checked regularly, so that offers which are offline, will be hidden
  • all offers are shown on the map:
    • the color shows the price per m² from low (green) to red (high)
    • the number of edges of the polygon visualizes the number of rooms + 1 (so 2 rooms = triangle, 3 rooms = square, ...)
    • the size of the polygon shows the total size (the more m² the flat has, the bigger it is)
  • the map allows filtering by properties (default properties are configurable)
  • the map includes several helpful map layers
    • the usual boring street and satellite layers
    • public transport layer
    • noise map (thanks to the great work of Berliner Morgenpost)
    • custom OpenStreetMaps layer generated by Mapbox shows churches (might be noisy!)
    • more layers possible: schools (noisy in the morning), markets (shouldn't be too far away), other public transport (bike sharing stations, ...)
  • it's possible to visualize how far you can get using public transport in x minutes from a selected marker on the map (there's a slider as well - big thanks to the great work of Mapnificent)
  • it offers direct links to specific locations using public transport (BVG in Berlin only for now...) when selecting a marker (e.g. how long does it take to get from home to work)
  • a telegram bot sends new offers as soon as they arrive, if they match given filters
  • it's possible to set favorites for easier filtering and tracking

Set up

  1. Create your own config.js by using the config.template.js (especially fill API keys)
  2. Run npm start
  3. Get lucky on localhost:3000

Roadmap

This was privately developed to get a new flat in Berlin (which worked very quickly and well btw). The following features might be very helpful:

  • fix scrapers (some broke due to some missing dynamics in the screen scraping process)
  • make the set up process easier (e.g. create db automatically)
  • Fix frontend
    • Rewrite in React (it's a quick and dirty Angular and Bower solution currently)
    • Localize frontend
    • Fix map layers
  • add support for more cities (more dynamic map layers & service integration... also, let users create scripts for cities which can be reused)
  • there's definitely more 🚀

Remarks

Please note that the usage might be illegal as you scrape data which is not yours! I'm not affiliated with any of the sites scraped or tools used here and everything you do with it is on your own risk.

You can’t perform that action at this time.