Skip to content
react-rewrite
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Wohnung Scraper

This tool scrapes several online real estate listing websites and visualizes offers in a map.

Screenshot

Screenshot

Features

  • new offers are scraped regularly
  • properties like the price, size and other stuff (balcony available) are extracted, parsed and stored
  • addresses are converted into geo coordinates
  • existing offers are checked regularly, so that offers which are offline, will be hidden
  • all offers are shown on the map:
    • the color shows the price per m² from low (green) to red (high)
    • the number of edges of the polygon visualizes the number of rooms + 1 (so 2 rooms = triangle, 3 rooms = square, ...)
    • the size of the polygon shows the total size (the more m² the flat has, the bigger it is)
  • the map allows filtering by properties (default properties are configurable)
  • the map includes several helpful map layers
    • the usual boring street and satellite layers
    • public transport layer
    • noise map (thanks to the great work of Berliner Morgenpost)
    • custom OpenStreetMaps layer generated by Mapbox shows churches (might be noisy!)
    • more layers possible: schools (noisy in the morning), markets (shouldn't be too far away), other public transport (bike sharing stations, ...)
  • it's possible to visualize how far you can get using public transport in x minutes from a selected marker on the map (there's a slider as well - big thanks to the great work of Mapnificent)
  • it offers direct links to specific locations using public transport (BVG in Berlin only for now...) when selecting a marker (e.g. how long does it take to get from home to work)
  • a telegram bot sends new offers as soon as they arrive, if they match given filters
  • it's possible to set favorites for easier filtering and tracking

Set up

  1. Create your own config.js by using the config.template.js (especially fill API keys)
  2. Use one of the following alternatives:
    1. Run docker run -d -p 3000:80 -v /path/to/your/local_config.js:/app/config.js -v /path/to/local/data/folder:/app/data sibbl/wohnung-scraper:react-rewrite to use the prebuilt Docker image from Docker Hub (requires Docker to be installed)
    2. Run docker-compose up to build and run the image locally (requires Docker to be installed)
    3. Run npm run server:dev to get the dev servers of both the backend and frontend up and running (requires node.js to be installed)
  3. Get lucky on localhost:3000

Roadmap

This was privately developed to get a new flat in Berlin (which worked very quickly and well btw). The following features might be very helpful:

  • fix scrapers (some broke due to some missing dynamics in the screen scraping process)
  • make the set up process easier (e.g. create db automatically)
  • Fix frontend
    • Rewrite in React (it's a quick and dirty Angular and Bower solution currently)
      • This is WIP in the react-rewrite branch. There are still some things missing, especially the public transport integration.
    • Localize frontend
    • Fix map layers
      • This is WIP in the react-rewrite branch. The whole map integration was rewritten.
  • add support for more cities (more dynamic map layers & service integration... also, let users create scripts for cities which can be reused)
    • This is WIP in the react-rewrite branch. This still needs to be more flexible.
  • there's definitely more 🚀

Remarks

Please note that the usage might be illegal as you scrape data which is not yours! I'm not affiliated with any of the sites scraped or tools used here and everything you do with it is on your own risk.

About

A web application which scrapes several German real estate listing websites and visualizes offers in a map.

Resources

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •  

Languages