Backend API for Jenkins Plugin Site
Switch branches/tags
Nothing to show
Clone or download
Pull request Compare This branch is 13 commits ahead of michaelmccaskill:develop.
daniel-beck Merge pull request #48 from daniel-beck/selector
Sometimes it's a table, sometimes it's a span
Latest commit 51c7b5c Mar 9, 2018

README.adoc

Plugin Site API

This is the backend API for Jenkins Plugin Site frontend. It provides a REST API to allow searching for plugins and retrieving specific plugin information.

Hacking

In order to run the plugin-site-api tests, a service needs to be online which can serve up a DATA_FILE_URL. Assuming docker-compos is installed on the system, the following commands will run the tests properly:

  • in another terminal: make run

  • make check

Inspect the Makefile for more targets, such as all which will run tests and build a working container.

Architecture

The REST API calls are powered by Jersey 2.x running inside of Tomcat 8. Plugins are stored and searched via embedded Elasticsearch.

Plugin information sources include:

  • Jenkins Update Center

    • Main source of plugin data

  • Jenkins Stats

    • Installation statistics

  • Jenkins Wiki

    • Scrape wiki content

Elasticsearch mappings are initialized at runtime using a temporary directory that is cleaned up at shutdown. The data is generated via a separate Maven task and stored on an external URL outside of the application. The data file is downloaded from said URL and extracted at boot time. Every 12 hours the application will download the file again and check if it contains new data. If so the application will the reindex the Elasticsearch data using the new data file.

Run Local Plugin Site API

DATA_FILE_URL="http://url.to/plugins.json.gzip" mvn jetty:run

This will launch an embedded Jetty container accessible at http://localhost:8080.

Run Docker Plugin Site API

docker build -t jenkinsciinfra/plugin-site-api .
docker run -p 8080:8080 -it -e DATA_FILE_URL="http://url.to/plugins.json.gzip" jenkinsciinfra/plugin-site-api

Rebuild Elasticsearch data

mvn -P generatePluginData

This will generate a new file in target/plugins.json.gzip consisting of plugin information and installation statistics. This file should be uploaded to DATA_FILE_URL.

REST API Reference

Warning
This API is still in flux as the frontend and backend are still being developed. As such this API is likely to change in the near future.

Unless otherwise indicated, all API calls:

  • Produce JSON

  • Do not require authentication

  • Have no parameters or request body

Possible error codes for all calls:

  • 500

GET /categories

Retrieve categories with their labels. It will be necessary to call /labels to get the titles

Sample Response

{
  "categories":
  [
    {
      "id": "languagesPlatforms",
      "title": "Platforms",
      "description": "Jenkins plugins that are designed to give added support for building, testing or deploying to specific languages or platforms.",
      "labels":
      [
        "ios", "dotnet", "android", "ruby", "scala"
      ]
    },
    ...
  ],
  "total": 6
}

GET /maintainers

Retrieve unique maintainers in the plugin data.

Sample Response

{
    "maintainers": [
        "Kohsuke Kawaguchi",
        "Jesse Farinacci",
        "Stephen Connolly",
        "Gregory Boissinot",
        "Oliver Gondža",
        "Oleg Nenashev",
        "Seiji Sogabe",
        "Lucie Votypkova",
        "Vojtech Juranek",
        "Nicolas De Loof",
        "Bap",
        "Praqma Josra",
        "Robert Sandell",
        "Stefan Brausch",
        .....
    ],
    "limit": 909
}

GET /labels

Retrieve available plugin labels. "title" is an optional field so it may be missing from some labels.

Sample Response

{
  "labels" :
  [
    {
      "id": "ios",
      "title": "iOS development"
    },
    {
      "id": "dotnet",
      "title": "Azure and .NET"
    },
    {
      "id": "android",
      "title": "Android development"
    },
    {
      "id": "ruby",
      "title": "Ruby development"
    },
    {
      "id": "scala",
      "title": "Scala plugins"
    },
    ....
  ],
  "limit": 30
}

GET /plugin/:name

Retrieves information about a plugin

Could return 404 if plugin is not found

Sample Response

{
  "buildDate": "Jul 04, 2016",
  "categories": [
    "scm"
  ],
  "dependencies": [
    {
      "name": "matrix-project",
      "optional": false,
      "version": "1.6"
    },
    ...
  ],
  "maintainers": [
    {
      "id": "kohsuke",
      "name": "Kohsuke Kawaguchi",
      "email": null
    },
    ...
  ],
  "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.",
  "gav": "org.jenkins-ci.plugins:git:2.5.2",
  "labels": [
    "scm"
  ],
  "name": "git",
  "previousTimestamp": "2016-07-02T20:46:28.00Z",
  "previousVersion": "2.5.1",
  "releaseTimetamp": null,
  "requiredCore": "1.609.3",
  "scm": "github.com",
  "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=",
  "stats": {
    "installations": [
      {
        "timestamp": 1322697600000,
        "total": 8906
      },
      ...
    ],
    "installationsPercentage": [
      {
        "timestamp": 1459468800000,
        "percentage": 61.16896694248365
      },
      ...
    ],
    "installationsPerVersion": [
      {
        "version": "2.0.3",
        "total": 141
      },
      ...
    ],
    "installationsPercentagePerVersion": [
      {
        "version": "2.2.7",
        "percentage": 0
      },
      ...
    ],
    "currentInstalls": 89232,
    "trend": 2990
  },
  "title": "Jenkins Git plugin",
  "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi",
  "version": "2.5.2",
  "wiki": {
    "content": <HTML content>,
    "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin"
}

GET /plugins

Search for plugins

Table 1. Parameters
Name Required Description Default Value Possible Values Example

page

false

Specify page number to return. This is not zero based

1

3

q

false

Search plugin name, title, excerpt if given

workflow

sort

false

Define how results are sorted

name

name,updated

name

categories

false

Filter by categories

scm

labels

false

Filter by labels

scm,ios

maintainers

false

Filter by maintainers

kohsuke

core

false

Filter by required core version

2.13

limit

false

Specify page limit for results

50

25

Sample Request

GET /plugins?q=git&sort=name&limit=3&page=1

Sample Response

{
  "page": 1,
  "pages": 3,
  "plugins":
  [
    {
      "buildDate": "Jul 04, 2016",
      "categories": [
        "scm"
      ],
      "dependencies": [
        {
          "name": "matrix-project",
          "optional": false,
          "version": "1.6"
        },
        ...
      ],
      "maintainers": [
        {
          "id": "kohsuke",
          "name": "Kohsuke Kawaguchi",
          "email": null
        },
        ...
      ],
      "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.",
      "gav": "org.jenkins-ci.plugins:git:2.5.2",
      "labels": [
        "scm"
      ],
      "name": "git",
      "previousTimestamp": "2016-07-02T20:46:28.00Z",
      "previousVersion": "2.5.1",
      "releaseTimetamp": null,
      "requiredCore": "1.609.3",
      "scm": "github.com",
      "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=",
      "stats": {
        "installations": [
          {
            "timestamp": 1322697600000,
            "total": 8906
          },
          ...
        ],
        "installationsPercentage": [
          {
            "timestamp": 1459468800000,
            "percentage": 61.16896694248365
          },
          ...
        ],
        "installationsPerVersion": [
          {
            "version": "2.0.3",
            "total": 141
          },
          ...
        ],
        "installationsPercentagePerVersion": [
          {
            "version": "2.2.7",
            "percentage": 0
          },
          ...
        ],
        "currentInstalls": 89232,
        "trend": 2990
      },
      "title": "Jenkins Git plugin",
      "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi",
      "version": "2.5.2",
      "wiki": {
        "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin"
      }
    },
    ...
  ],
  "limit": 3,
  "total": 7
}

GET /plugins/installed

Get top "limit" install plugins

Table 2. Parameters
Name Required Description Default Value Possible Values Example

limit

false

Specify limit for results

10

5

Sample Request

GET /plugins/installed

Sample Response

{
  "page": 1,
  "pages": 3,
  "plugins":
  [
    {
      "buildDate": "Jul 04, 2016",
      "categories": [
        "scm"
      ],
      "dependencies": [
        {
          "name": "matrix-project",
          "optional": false,
          "version": "1.6"
        },
        ...
      ],
      "maintainers": [
        {
          "id": "kohsuke",
          "name": "Kohsuke Kawaguchi",
          "email": null
        },
        ...
      ],
      "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.",
      "gav": "org.jenkins-ci.plugins:git:2.5.2",
      "labels": [
        "scm"
      ],
      "name": "git",
      "previousTimestamp": "2016-07-02T20:46:28.00Z",
      "previousVersion": "2.5.1",
      "releaseTimetamp": null,
      "requiredCore": "1.609.3",
      "scm": "github.com",
      "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=",
      "stats": {
        "installations": [
          {
            "timestamp": 1322697600000,
            "total": 8906
          },
          ...
        ],
        "installationsPercentage": [
          {
            "timestamp": 1459468800000,
            "percentage": 61.16896694248365
          },
          ...
        ],
        "installationsPerVersion": [
          {
            "version": "2.0.3",
            "total": 141
          },
          ...
        ],
        "installationsPercentagePerVersion": [
          {
            "version": "2.2.7",
            "percentage": 0
          },
          ...
        ],
        "currentInstalls": 89232,
        "trend": 2990
      },
      "title": "Jenkins Git plugin",
      "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi",
      "version": "2.5.2",
      "wiki": {
        "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin"
      }
    },
    ...
  ],
  "limit": 3,
  "total": 7
}

GET /plugins/trend

Get top "limit" trending plugins

Table 3. Parameters
Name Required Description Default Value Possible Values Example

limit

false

Specify limit for results

10

5

Sample Request

GET /plugins/trend

Sample Response

{
  "page": 1,
  "pages": 3,
  "plugins":
  [
    {
      "buildDate": "Jul 04, 2016",
      "categories": [
        "scm"
      ],
      "dependencies": [
        {
          "name": "matrix-project",
          "optional": false,
          "version": "1.6"
        },
        ...
      ],
      "maintainers": [
        {
          "id": "kohsuke",
          "name": "Kohsuke Kawaguchi",
          "email": null
        },
        ...
      ],
      "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.",
      "gav": "org.jenkins-ci.plugins:git:2.5.2",
      "labels": [
        "scm"
      ],
      "name": "git",
      "previousTimestamp": "2016-07-02T20:46:28.00Z",
      "previousVersion": "2.5.1",
      "releaseTimetamp": null,
      "requiredCore": "1.609.3",
      "scm": "github.com",
      "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=",
      "stats": {
        "installations": [
          {
            "timestamp": 1322697600000,
            "total": 8906
          },
          ...
        ],
        "installationsPercentage": [
          {
            "timestamp": 1459468800000,
            "percentage": 61.16896694248365
          },
          ...
        ],
        "installationsPerVersion": [
          {
            "version": "2.0.3",
            "total": 141
          },
          ...
        ],
        "installationsPercentagePerVersion": [
          {
            "version": "2.2.7",
            "percentage": 0
          },
          ...
        ],
        "currentInstalls": 89232,
        "trend": 2990
      },
      "title": "Jenkins Git plugin",
      "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi",
      "version": "2.5.2",
      "wiki": {
        "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin"
      }
    },
    ...
  ],
  "limit": 3,
  "total": 7
}

GET /plugins/updated

Get top "limit" recently updated plugins

Table 4. Parameters
Name Required Description Default Value Possible Values Example

limit

false

Specify limit for results

10

5

Sample Request

GET /plugins/updated

Sample Response

{
  "page": 1,
  "pages": 3,
  "plugins":
  [
    {
      "buildDate": "Jul 04, 2016",
      "categories": [
        "scm"
      ],
      "dependencies": [
        {
          "name": "matrix-project",
          "optional": false,
          "version": "1.6"
        },
        ...
      ],
      "maintainers": [
        {
          "id": "kohsuke",
          "name": "Kohsuke Kawaguchi",
          "email": null
        },
        ...
      ],
      "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.",
      "gav": "org.jenkins-ci.plugins:git:2.5.2",
      "labels": [
        "scm"
      ],
      "name": "git",
      "previousTimestamp": "2016-07-02T20:46:28.00Z",
      "previousVersion": "2.5.1",
      "releaseTimetamp": null,
      "requiredCore": "1.609.3",
      "scm": "github.com",
      "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=",
      "stats": {
        "installations": [
          {
            "timestamp": 1322697600000,
            "total": 8906
          },
          ...
        ],
        "installationsPercentage": [
          {
            "timestamp": 1459468800000,
            "percentage": 61.16896694248365
          },
          ...
        ],
        "installationsPerVersion": [
          {
            "version": "2.0.3",
            "total": 141
          },
          ...
        ],
        "installationsPercentagePerVersion": [
          {
            "version": "2.2.7",
            "percentage": 0
          },
          ...
        ],
        "currentInstalls": 89232,
        "trend": 2990
      },
      "title": "Jenkins Git plugin",
      "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi",
      "version": "2.5.2",
      "wiki": {
        "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin"
      }
    },
    ...
  ],
  "limit": 3,
  "total": 7
}

GET /versions

Retrieve unique required Jenkins versions in the plugin data.

Sample Response

{
  "limit": 226,
  "versions": [
    "1.580.1",
    "1.625.3",
    "1.424",
    "1.609.3",
    "1.609.1",
    "1.398",
    "1.466",
    "1.480",
    "1.596.1",
    "1.532.3",
    "1.580",
    "1.509.4",
    "1.642.3",
    "1.580.3",
    "1.447",
    "1.532",
    "1.609",
    "1.509",
    "1.554.1",
    "1.480.3",
    "1.509.3",
    "1.392",
    .....
  ]
}

Deployment to production

This project is containerized via the Dockerfile that is located in the deploy/ directory. The Jenkinsfile uses this Dockerfile in conjunction with the front-end code located in the jenkins-infra/plugin-site repository to create a container fit for deployment.

Unfortunately, the build of this container must occur on a Jenkins cluster which is publicly inaccessible for security reasons. The private job polls SCM every 10 minutes.

Deploying:

  1. Have code merged to the master branch of this repository

  2. Wait pateiently 10-15 minutes (coffee time!)

  3. Verify that a new container tag has been published here.

  4. Submit a pull request to the jenkins-infra/jenkins-infra repository updating the profile::pluginsite::image_tag value to the latest container’s tag (here)

  5. Once that is merged the changes will be live