Skip to content
Provisioning app, which triggers project and component provisions (including Jira / Confluence / BitBucket and OCP resource creation)
Branch: master
Clone or download
tjaeschke Merge pull request #140 from opendevstack/tjaeschke-test-fix
- Implemented: Fixed failing test, added mock client
Latest commit e1da7a8 May 21, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docker Build directly Aug 2, 2018
docs updated navigation Apr 12, 2019
gradle/wrapper Update Gradle to 4.9 Jul 30, 2018
ocp-config Added missing default template key May 14, 2019
src - Implemented: Fixed failing test, added mock client May 21, 2019
.gitignore make crowd sso cookie name configurable (#82) Jan 28, 2019
.travis.yml change jdk to openjdk8 for travis config Apr 26, 2019
CHANGELOG.md Update of changelog with springboot changes May 10, 2019
Jenkinsfile Bug - fix latest sqfindings (#133) May 7, 2019
LICENSE
README.md Update README.md May 13, 2019
build.gradle resolves #119: update spring boot version to 2.1.4 May 10, 2019
gradlew Initial Commit Jul 27, 2018
gradlew.bat Initial Commit Jul 27, 2018
sonar-project.properties Feature/increase cov (#45) Oct 27, 2018

README.md

OpenDevStack Provisioning Application

Build Status

This application creates new OpenDevStack digital projects. It is the central entrypoint to get started with a new project / or provision new components based on quickstarters. It delegates the tasks to create / update resources to several services such as jira, confluence, bitbucket and rundeck.

Basic idea & usage:

  1. An admin (user in a group defined in property crowd.admin.group) creates new ODS project. This in turn creates

    • a Jira Project (name based on project key & name)
    • a Confluence Space (name based on project's key)
    • the required Openshift projects named key-dev, key-test and key-cd - in case openshiftproject == true. Internally this is done thru a rest call to rundeck triggering the create-projects rundeck job
    • a Bitbucket Project (name based on project key) - in case openshiftproject == true. Within this project two default repositories are created key-oc-config-artifacts for all yaml resources as well as key-design for any design artifacts (e.g. sketches)
  2. A normal user (user in a group defined in property crowd.user.group) creates all resources required for a working component - this happens thru the user interface - in going to modify project / picking your project and then the wanted quickstarter. Internally this is done thru a rest call to rundeck - with the picked job as parameter - here

    • Bitbucket repository within the chosen project named key-boilerplate name
    • Openshift components based on the chosen boilerplate, coming from ods-quickstarters
  3. The involved people receive an email with the setup, URLs to components etc. - in case mail.enabled == true

Integration with Bitbucket (webhooks)

Next to the provision app creating the bitbucket repository for a chosen quickstarter - it also creates a webhook on that repo, which triggers on three events

    List<String> events = new ArrayList<String>();
        events.add("repo:refs_changed");
        events.add("pr:merged");
        events.add("pr:declined");
    webhook.setEvents(events);

This webhook calls the webhook proxy which in turn creates an openshift build config of type pipeline in the name-cd project and executes it.

Permissions

By default no special permissions are set on either confluence / jira / bitbucket or openshift, only system-wide settings are inherited.

However there is a special knob to tighten security (which can be passed with the project input createpermissionset : boolean) - based on three groups that need to be provided as part of the API call / from the userinterface.

  1. admin group: admin rights on the generated projects / spaces / repositories
  2. user group: read / write rights on the generated projects / spaces / repositories
  3. readonly group: read rights on the generated projects / spaces / repositories

The configuration for the permission sets are configured:

  1. JIRA Project is provisioned with its own permissionset defined in src/main/resources/permission-templates/jira.permission.all.txt
  2. Confluence Project is provisioned with special permission set defined in src/main/resources/permission-templates/confluence.permission.*
  3. Bitbucket Project is provisioned with tight read & write roles
  4. Openshift Project roles linked to the passed groups (READONLY - view, ADMINGROUP - admin, USERS - edit)

Project/Space types based on templates

The default jira / confluence project' types are defined in src/main/resources/application.properties - and correspondingly in the config maps

project.template.key.names=default

jira.project.template.key=com.pyxis.greenhopper.jira:gh-scrum-template
jira.project.template.type=software

confluence.blueprint.key=com.atlassian.confluence.plugins.confluence-software-project:sp-space-blueprint

To add a new template - copy, and add your config, based on a new <name>

jira.project.template.key.<name>=com.pyxis.greenhopper.jira:gh-scrum-template
jira.project.template.type.<name>=software

# optional, can stay as is
confluence.blueprint.key.<name>=com.atlassian.confluence.plugins.confluence-software-project:sp-space-blueprint

and add the new from above to the existing property project.template.key.names

# list of templates surfaced to the UI and API
project.template.key.names=default,<name>

Using the provision application via API / thru direct REST calls

PROVISION_API_HOST=<host name>

curl -D headers.txt -k -H "Content-Type: application/x-www-form-urlencoded" \
-X POST ${PROVISION_API_HOST}/j_security_check \
-d username=<username> -d password=<password>

# grab the login status, and exit if error
login_status=$(cat headers.txt | grep ${PROVISION_API_HOST}/login?error)

if [[ $login_status != "" ]]; then echo "Login Error"; exit 1; fi;

# grab the needed IDs and bake the cookies
JSESSION_ID=$(cat headers.txt | grep "Set-Cookie: JSESSION" | cut -d ';' -f1 | cut -d ":" -f2)";" 
CROWD_COOKIE=$(cat headers.txt | grep "Set-Cookie: crowd" | cut -d ';' -f1 | cut -d ":" -f2)

COOKIES=${JSESSION_ID}${CROWD_COOKIE}

# sample provision file >> create.txt
{
  "name" : "<Mandatory name>",
  "key" : "<Mandatory key>",
  "createpermissionset" : true,
  "jiraconfluencespace" : true,
  "admin" : "<admin user>",
  "adminGroup" : "<admin group>",
  "userGroup" : "<user group>",
  "readonlyGroup" : "<readonly group>",
  "openshiftproject" : false
}

provisionfile=create.txt

# invoke the provision API to create a new project
curl -k -X POST --cookie "$COOKIES" -d @"$provisionfile" \
-H "Content-Type: application/json; charset=utf-8" -v ${PROVISION_API_HOST}/api/v1/project

Internal architecture

The Project is based on Spring Boot, using several technologies which can be seen in the build.gradle.

The provision app is merely an orchestrator that does HTTP REST calls to Atlassian Crowd, Jira, Confluence, Bitbucket and Rundeck (for openshift interaction).

The APIs exposed for direct usage, and also for the UI are in the controller package. The connectors to the various tools to create resources are in the services package

If you want to build / run locally - create gradle.properties in the project's root to configure connectivity to OpenDevStack NEXUS

- nexus_url=<NEXUS HOST>
- nexus_user=<NEXUS USER>
- nexus_pw=<NEXUS_PW> 

If you want to build / run locally without NEXUS, you can disable NEXUS by adding the following property to gradle.properties:

no_nexus=true

Alternatively, you can also configure the build using environment variables:

Gradle property Environment variable
nexus_url NEXUS_HOST
nexus_user NEXUS_USERNAME
nexus_pw NEXUS_PASSWORD
no_nexus NO_NEXUS

You can start the application with the following command:

# to run the server execute
gradle bootRun

To overwrite the provided application.properties a configmap is created out of them and injected into /config/application.properties within the container. The base configuration map as well as the deployment yamls can be found in ocp-config, and overwrite parameters from application.

Frontend Code

The frontend is based on jquery and thymeleaf. All posting to the API happens out of java script (client.js)

Backend Code

The backend is based on Spring Boot, authenticates against Atlassian Crowd and exposes consumable APIs (api/v1/project). Storage of created projects happens on the filesystem thru the StorageAdapter. Both frontend (html) and backend are tested thru Junit & Mockito

Consuming REST APIs in Java

Generally this is a pain. To ease development, a few tools are in use:

  • Jackson (see link below)
  • OKTTP3 Client (see link below)
  • jsonschema2pojo generator (see link below)

The process for new operations to be called is:

  1. Look up the API call that you intend to make
  2. see if there is a JSON Schema available
  3. Generate (a) Pojo('s) for the Endpoint
  4. Use the pojo to build your request, convert it to JSON with Jackson and send it via OKHTTP3, and the Provision Application's RestClient

Link collection

Atlassian API's

FAQ:

  1. Where is the provision app deployed?
    A. the provision application is deployed on openshift, in both prov-dev and prov-test. prov-dev is the development environment in case you want to change / enhance the application, while the production version of the application is deployed in prov-test. The URL to get to the provision application, is defined thru a route. Ít's https://prov-app-test..

  2. Why are three Openshift projects created when I provision a new project?
    A: The project-name-dev & -test ones are runtime namespaces. Depending on which branch you merge / commit your code into, images will be built & deployed in one of the two (further information on how this is done - can be found in the jenkins-shared-library
    In contrast to this, the project-name-cd namespace hosts a project specific instance of the ODS Jenkins and also of the Webhook Proxy. When a built is triggered, builder pods (=deployments of Jenkins slaves) are created in this project.
    This was a cautious design choice to give a project team as much power as possible - when it comes to configuration of jenkins.

  3. What is RUNDECK used for?
    A: Rundeck is used as orchestration engine when the provision application triggers provision jobs (e.g. create new projects, create components). This architecture is subject to change likely in release 2.0, to dramatically reduce complexity in multi cluster scenarios.

  4. Where do I find the logs, if something went wrong?
    A. Within the Openshift pod of the provision app (in projectdev/test, namely in /opt/provision/history/logs a logfile is created per project)

  5. Where is the real configuration of the provision application?
    A. The base configuration in the the application.properties in the codebase, the setup specific one is in a config map deployed within the prov-dev/test project.

You can’t perform that action at this time.