Tool which collects (scraps from web) statistics for programing languages in need for my site (https://jaki-jezyk-programowania.pl).
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.idea
demo
src
.gitattributes
.gitignore
LICENSE
README.md
pom.xml
stats-scraper.bat
stats-scraper.iml

README.md

NOTE: I do not use this version anymore. It may not work. I have rewrited it using Kotlin -> stats-scraper

Programing languages statistics scraper

Tool which collects (scraps from web) statistics for programing languages in need for my site jaki-jezyk-programowania.pl.

Currently, the tool is fetching data for each language from:

  • Github
    • top 10 projects
    • number of projects
    • number of projects with more than 500 stars
  • Meetup.com
    • number of members
    • number of meetups
  • StackOverflow
    • number of tagged questions
  • Wikipedia
    • latest language version
  • Tiobe INDEX
    • position at last year
    • position at this year
  • Spectrum ranking
    • position at last year
    • position at this year

Everything is stored in two json files: - statistics.json - languagesVersions.json

If statistics.json already exists, then it will be renamed (date will be appended) and fresh one will be created as statistics.json.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

- Java 8+
- Maven (Development)
- Github account (if you want to collect data from Github)

Demo

If you want to just demo test it, download demo release, put it to demo folder and run this bat file:

demo/stats-scraper-demo.bat

or manually in cmd from project directory

java -jar stats-scraper-1.0.0.jar -no-git

(Github data will be skipped)

Note:

If you will run this tool more than once within short time then errors occurs due to api restrictions. 

Also you cannot have more than one file created in the same day. You need to remove or move old file with appended date if you want to get fresh data in that day.

Installing

Install maven dependencies

mvn install

Provide github authentication token under src\main\resources\config.properties if u want to fetch data from Github

Follow this guide if u don't have token.

GithubAuthToken=token 22sadasdsa34r32412342134214324123

Otherwise, you need to pass parameter -no-git when you are running program:

java -jar stats-scraper-1.0.0.jar -no-git

or remove lines where GithubDataScraper is added to StatisticsBuilder at App class.

If you want to regenerate jar file to use it as single file application run:

mvn packege

(it goes under target/stats-scraper-1.0.0.jar)

Sample output of tool:

Sample output of tool Sample output of tool 2

Built With