detectem - detect software and its version on websites.
Julia Python
Clone or download
Latest commit 5984a4f Jul 20, 2018
Permalink
Failed to load latest commit information.
.github Added initial version Nov 15, 2016
data Add first generic plugin: Wordpress Generic Plugin Mar 1, 2018
detectem Add timeout to webservice Jul 19, 2018
docs Add missing newlines at end of file Jul 27, 2017
extras/docker Add support to run detectem with Docker Jan 4, 2018
scripts Update add_new_plugin script with support for xpath Oct 11, 2017
tests Update pyyaml from 3.12 to 3.13 Jul 17, 2018
.bumpversion.cfg Bump version: 0.6.0 → 0.6.1 Sep 7, 2017
.editorconfig Bumping version updates changelog too Aug 31, 2017
.gitignore Added initial version Nov 15, 2016
.gitlab-ci.yml Add parsel to requirements.txt Oct 10, 2017
.pyup.yml create pyup.io config file Jan 17, 2018
.style.yapf Update codestyle and requirements.txt Apr 9, 2018
.travis.yml Update travis configuration to test python 3.5 and 3.6. Jan 17, 2018
AUTHORS.rst Update travis configuration to test python 3.5 and 3.6. Jan 17, 2018
CHANGELOG.md Bump version: 0.6.0 → 0.6.1 Sep 7, 2017
CONTRIBUTING.rst Added initial version Nov 15, 2016
HISTORY.rst Added initial version Nov 15, 2016
LICENSE Added initial version Nov 15, 2016
MANIFEST.in Add script.lua template Feb 12, 2017
Makefile Remove doc generation from root Makefile since it corrupts docs direc… Feb 13, 2017
README.rst Update travis configuration to test python 3.5 and 3.6. Jan 17, 2018
requirements.txt Update codestyle and requirements.txt Apr 9, 2018
setup.cfg Bumping version updates changelog too Aug 31, 2017
setup.py Update travis configuration to test python 3.5 and 3.6. Jan 17, 2018
tox.ini Add support for python 3.6 Jul 17, 2017

README.rst

detectem

Updates

detectem is a specialized software detector. Let's see it in action.

$ det http://domain.tld
[{'name': 'phusion-passenger', 'version': '4.0.10'},
 {'name': 'apache-mod_bwlimited', 'version': '1.4'},
 {'name': 'apache-mod_fcgid', 'version': '2.3.9'},
 {'name': 'jquery', 'version': '1.11.3'},
 {'name': 'crayon-syntax-highlighter', 'version': '_2.7.2_beta'}]

Using a serie of indicators, it's able to detect software running on a site and extract accurately its version information. It uses Splash API to render the website and start the detection routine. It does full analysis on requests, responses and even on the DOM!

There are two important articles to read:

Features

  • Detect software in modern web technologies.
  • Browser support provided by Splash.
  • Analysis on requests made and responses received by the browser.
  • Get software information from the DOM.
  • Great performance (less than 10 seconds to get a fingerprint).
  • Plugin system to add new software easily.
  • Test suite to ensure plugin result integrity.
  • Continuous development to support new features.

Installation

  1. Install Docker and add your user to the docker group, then you avoid to use sudo.

  2. Pull the image:

    $ docker pull scrapinghub/splash
    
  3. Create a virtual environment with Python >= 3.5 .

  4. Install detectem:

    $ pip install detectem
    
  5. Run it against some URL:

    $ det http://domain.tld
    

Other installation method

detectem as Docker Container

Let's see it in action.

$ docker-compose run --rm detectem http://domain.tld
[{'name': 'phusion-passenger', 'version': '4.0.10'},
 {'name': 'apache-mod_bwlimited', 'version': '1.4'},
 {'name': 'apache-mod_fcgid', 'version': '2.3.9'},
 {'name': 'jquery', 'version': '1.11.3'},
 {'name': 'crayon-syntax-highlighter', 'version': '_2.7.2_beta'}]

But first that all we must do:

Installation

  1. Install the last Docker CE Stable version.

  2. Add your user to the docker group and logout:

    $ sudo usermod -a -G docker you
    
  3. Make sure you have logout to apply changes, then log in again.

  4. Install Docker Compose

  5. Download to your workspace the docker-compose building files.

    Dockerfile-alternate docker-compose.yml

  6. Build the required docker images for detectem at the same directory as the previous point:

    $ docker-compose up -d
    
  7. Run detectem against some URL:

    $ docker-compose run --rm detectem http://domain.tld
    

Documentation

The documentation is at ReadTheDocs.