Skip to content
A utility to collect a history of key numbers for each SonarQube analysis triggered by maven builds.
Java
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
sonar-collector-webhook
.editorconfig
.gitignore
.travis.yml
LICENSE
README.org
pom.xml

README.org

SonarQube metrics collector

SonarQube is a code analysis tool that shows key numbers about code quality, e.g. code coverage, code complexity and various code practices.

SonarQube has a web GUI that allows exploring the analysis results.

However, SonarQube has no storage of build quality history. To keep statistics about code quality one either have to manually type the key numbers of each analysed application into a spreadsheet, or use something like this tool.

This utility consists of a servlet that serves as a webhook that is called by Sonar when completing an analysis. The webhook POST data doesn’t have the necessary information (which are some key metrics of the build).

So when receiving a POST, the servlet will do a callback to the SonarQube REST API to retrieve the metrics, which will then be stored in a PostgreSQL database.

The servlet has been written as a microservice that can be installed into an apache karaf instance.

Status of the project

https://travis-ci.org/steinarb/sonar-collector.svg?branch=master https://coveralls.io/repos/steinarb/sonar-collector/badge.svg https://sonarcloud.io/api/project_badges/measure?project=no.priv.bang.sonar.sonar-collector%3Asonar-collector&metric=alert_status#.svg https://maven-badges.herokuapp.com/maven-central/no.priv.bang.sonar.sonar-collector/sonar-collector/badge.svg

SonarCloud

https://sonarcloud.io/api/project_badges/measure?project=no.priv.bang.sonar.sonar-collector%3Asonar-collector&metric=ncloc#.svg https://sonarcloud.io/api/project_badges/measure?project=no.priv.bang.sonar.sonar-collector%3Asonar-collector&metric=bugs#.svg https://sonarcloud.io/api/project_badges/measure?project=no.priv.bang.sonar.sonar-collector%3Asonar-collector&metric=vulnerabilities#.svg https://sonarcloud.io/api/project_badges/measure?project=no.priv.bang.sonar.sonar-collector%3Asonar-collector&metric=code_smells#.svg https://sonarcloud.io/api/project_badges/measure?project=no.priv.bang.sonar.sonar-collector%3Asonar-collector&metric=coverage#.svg

Release history

VersionDateDescription
1.1.0<2019-11-14 Thu>Use pax-jdbc-config to set up the database, build with openjdk 11, jackson security upgrade
1.0.0<2017-12-18 Mon>First release

How to build the application

(this assumes that you have an apache karaf already installed on your development computer, as well as git, maven, and a Java development kit)

Procedure:

  1. Clone the project
    mkdir -p ~/git
    cd ~/git
    git clone https://github.com/steinarb/sonar-collector.git
        
  2. Build the project with maven
    cd ~/git/sonar-collector
    mvn clean install
        
  3. Give the following commands from the karaf console to install the REST service:
    feature:repo-add mvn:no.priv.bang.sonar.sonar-collector/sonar-collector-webhook/LATEST/xml/features
    feature:install sonar-collector-webhook
        

Note: You will need to have a suitable PostgreSQL database to write to for this servlet to be meaningful. See the sections Create the database and Using a database running on a different host for more detail. The database just have to be a blank, freshly created database, that the servlet can write to, either on localhost with the curent user, or using JDBC credentials configured from the karaf console (this is what’s covered in Using a database running on a different host).

How to install and run the application on a debian server

(This procedure doesn’t require development tools or building the project yourself. The servlet, and its attached karaf feature has been deployed to maven central)

This describes how to install and run the program on a debian GNU/linux system.

Install the required software

As root, do the following command:

apt-get update
apt-get install postgresql

Create the database

Procedure:

  1. Create a PostgreSQL user matching the karaf user:
    /usr/bin/sudo -u postgres createuser --pwprompt karaf
        
    1. At the prompt “Enter password for new role”, enter the JDBC password for user “karaf”
    2. At the prompt “Enter it again”, enter the same password again

    Make a note of this password, since it will be needed later, when setting up a password authenticated connection

  2. Create an empty database owned by the karaf user:
    /usr/bin/sudo -u postgres createdb -O karaf sonarcollector
        

Install apache karaf

Do the following steps as root

  1. Add a key for the apt repo containing the karaf package
    wget -O - https://apt.bang.priv.no/apt_pub.gpg | apt-key add -
        
  2. Add the repo containing karaf by adding the following lines to /etc/apt/sources.list :
    # APT archive for apache karaf
    deb http://apt.bang.priv.no/public stable main
        
  3. Install the debian package
    apt-get update
    apt-get install karaf
        

Install the application in karaf

Procedure:

  1. SSH into karaf
    ssh -p 8101 karaf@localhost
        

    The password is “karaf” (without the quotes)

  2. Install the application
    feature:repo-add mvn:no.priv.bang.sonar.sonar-collector/sonar-collector-webhook/LATEST/xml/features
    feature:install sonar-collector-webhook-with-postgresql
        

(the application has been installed into maven central, which is a repository that is builtin to karaf)

Using a database running on a different host

The above example shows connecting to a PostgreSQL database running on localhost, authenticating with ident authentication (ie. no password).

This example shows how to connect to a PostgreSQL database running on a different host, authenticating using username and password.

Procedure:

  1. SSH into karaf
    ssh -p 8101 karaf@localhost
        

    The password is “karaf” (without the quotes)

  2. In the karaf command shell, create configuration for the JDBC connection:
    config:edit org.ops4j.datasource-sonar-collector
    config:property-set osgi.jdbc.driver.name "PostgreSQL JDBC Driver"
    config:property-set dataSourceName "jdbc/sonar-collector"
    config:property-set url "jdbc:postgresql://lorenzo.hjemme.lan/sonarcollector"
    config:property-set user "karaf"
    config:property-set password "karaf"
    config:property-set org.apache.karaf.features.configKey "org.ops4j.datasource-sonar-collector"
    config:update
        

    (this assumes the username/password combination karaf/karaf, it is recommended to use a different password in a real setting with PostgreSQL accepting network connections)

The “config:update” command will cause the sonar collector to be restarted, it will pick up the new configuration, and connect to the remote server, and if the “sonar-collector” database exists as a blank database, create the schema and be ready to store data there.

Side note: The configuration will be stored in standard .properties file format, in the file /etc/karaf/no.priv.bang.sonar.collector.webhook.SonarCollectorServlet.cfg and be persistent across restarts and reinstallations of the karaf .deb package (the .deb package will only uninstall/reinstall unchanged known files in this directory, and won’t touch unknown files at all).

Allowing network connections in PostgreSQL on debian

Note that PostgreSQL out of the box on debian only accepts domain connections and localhost connections.

To make PostgreSQL listen on all network connections, two files must be edited and the PostgreSQL daemon must be restarted.

Procedure, do the following, logged in as root on the server:

  1. Do “su” to user postgres to get the right ownership on the files
    su - postgres
        
  2. Edit the /etc/postgresql/9.6/main/postgresql.conf file, uncomment the listen_address line and edit it to look like this
    listen_addresses = '*'                  # what IP address(es) to listen on;
        
  3. Edit the /etc/postgresql/9.6/main/pg_hba.conf, add the following lines
    # IPv4 network connection allow password authentication
    host    all             all             0.0.0.0/0               md5
        
  4. Log out from user postgres (only root can restart the daemon):
    exit
        
  5. Restart the postgresql daemon
    systemctl restart postgresql
        

Using a different database than PostgreSQL

WARNING! This is not regularily tested (i.e. won’t be tested before releases) and I don’t plan to actually use sonar-collector with anything except PostgreSQL myself.

To use JDBC against a RDBMS other than PostgreSQL, do the following from the karaf console command line (derby in-memory database used in the examples):

  1. Load the component providing the DataSourceFactory OSGi service:
    feature:install pax-jdbc-derby
        
  2. Add karaf configuration selecting the correct DataSourceFactory and JDBC connection info (url, user and password):
    config:edit org.ops4j.datasource-sonar-collector
    config:property-set osgi.jdbc.driver.name "PostgreSQL JDBC Driver"
    config:property-set dataSourceName "jdbc/sonar-collector"
    config:property-set url "jdbc:derby:data/example/derby;create=true"
    config:property-set osgi.jdbc.driver.name derby
    config:property-set org.apache.karaf.features.configKey "org.ops4j.datasource-sonar-collector"
    config:update
        
  3. Load sonar-collector using a feature that doesn’t unnecessarily pull in the PostgreSQL DataSourceFactory:
    feature:repo-add mvn:no.priv.bang.sonar.sonar-collector/sonar-collector-webhook/LATEST/xml/features
    feature:install sonar-collector-webhook-with-jdbc
        

Add a webhook to Sonar

Add a webhook to SonarCloud

Procedure:

  1. Open your SonarCloud project in a web browser and log in as a user with ownership to the project (I do login as github user)
  2. In the project select the menu Administration->General Settings
  3. Select the webhooks tab in the tab bar on the left side of the page (you may have to scroll down to see it)
  4. In “Name:”, write:
    sonar-collecttor
        
  5. In “URL”, write:
    https://mydowmain.com:8181/sonar-collector
        
  6. Click the button “Save”

Add a webhook to a hosted SonarQube instance

In a hosted SonarQube the webhook can be set globally across all projects.

Procedure:

  1. Open your SonarCloud instance in a web browser, e.g. http://localhost:9000 and log in as an admin user (admin/admin in a test instance)
  2. In the top menu, select Administration
  3. Select the tab “Webhooks” in the list to the left of the page (you may have to scroll down to see the tab)
  4. In “Name”, type:
    sonar-collector
        
  5. In “URL”, type:
    http://localhost:8181/sonar-collector
        
  6. Click the button “Save”

License

This utility is licensend under the Apache license v. 2. See the LICENSE file for details.

Development stuff

Testing and debugging

To run the servlet locally and debug into the servlet, the following software is required:

  1. A locally installed apache karaf (see the apache karaf quick start guide )
  2. A locally installed SonarQube (see SonarQube Get Started in Two Minutes )
  3. A locally installed (or at least reachable, see Using a database running on a different host ) PostgreSQL database
  4. An IDE that can do remote debugging

Preparation for debugging

  1. create user and empty database in PostgreSQL
  2. Add http://localhost:8181/sonar-collector as a webhook in SonarQube
  3. Clone and build the sonar-collector
    mkdir -p ~/git
    cd ~/git/
    git clone https://github.com/steinarb/sonar-collector.git
    cd ~/git/sonar-collector/
    mvn clean install
        
  4. Start karaf with setup for remote debugging (cd to an unpacked downloaded karaf installation, start karaf as the user you used to do “mvn clean install”)
    cd ~/Downloads/apache-karaf-4.1.4/
    bin/karaf debug
        
  5. Install the sonar-collector in karaf, with the following commands in the karaf console:
    feature:repo-add mvn:no.priv.bang.sonar.sonar-collector/sonar-collector-webhook/LATEST/xml/features
    feature:install sonar-collector-webhook
        
  6. Connect the IDE to a debugging connection on localhost port 5005 (see your IDE’s documentation for this) and set the breakpoint at the desired code

Then just trigger an analysis in the locally installed SonarQube and debug when the breakpoint is triggered:

mvn clean org.jacoco:jacoco-maven-plugin:prepare-agent package sonar:sonar -Dsonar.host.url=http://localhost:9000 -Dsonar.login=a51f2ab9a8790abd91773f0a7d2f6d2dc9d97975

(as the sonar.login argument, use the token that SonarQube returns when using the setup wizard of the quick start)

You can’t perform that action at this time.