Structure

Dieter Adriaenssens edited this page Nov 1, 2015 · 53 revisions

Scripts (python-client)

Each script (except init.sh) checks if init.sh was run.

Scripts (service)

  • service.py : powers Buildtime Trend as a Service on Heroku, see README.md for more information.
  • utils/start_worker.py : starts the Celery worker executing jobs on the task queue, see README.md for more information.

Files and data structures

Timestamp file

filename : timestamps.csv

"stage1",timestamp1
"stage2",timestamp2
"end",timestamp3

Data file in native mode

Output of analysis.py, filename : trends/buildtimes.xml :

<builds>
  <build
      repo="ruleant/buildtime-trend"    # repo name (since version 0.1)
      build="1"                         # build id (since version 0.1)
      job="1.1"                         # job id (since version 0.1)
      result="passed"                   # build result : passed, failed (since version 0.1)
      branch="master"                   # repo branch (since version 0.1)
      started_at="2014-07-25T14:03:51Z" # timestamp when build started (since version 0.1)
      ci_platform="travis">             # Continuous integration platform : 
                                        #   travis, local, ... (since version 0.1)
     <stages>
       <stage
           name="stage1"                # stage name (since version 0.1)
           duration="value2_1" />       # stage duration [sec] (since version 0.1)
       ...
     </stages>
  </build>
  ...
<builds>

Data structures in keen mode

Sent to Keen.io by analysis.py (python-client) or by Buildtime Trend as a Service.

Structure of each eventtype :

  • buildtime_trend
    • project_name : project name is the unique identifier for this project, usually the repo name (since version 0.1 : keen)
    • version : Buildtime trend version generating the data (since version 0.1 : keen, renamed to lib_version in version 0.2)
    • lib_version : Buildtime trend library version generating the data (since version 0.2 : keen, renamed from version)
    • client_version : Buildtime trend client version generating the data (since version 0.2 : keen)
    • client : Name of Buildtime Trend client generating the data (since version 0.2 : keen)
    • schema_version : Version of the schema structure (since version 0.1 : keen)
  • keen (added by Keen.io API)
    • timestamp : timestamp, equals to created_at if not specified
    • created_at : when the event was created
    • id : unique ID

In the build_jobs event collection, these are added :

  • job : see Build/job properties (since version 0.1 : keen, renamed in version 0.2 (previously build))
    • stages : list of stages (since version 0.1 : keen)
      • stage : see Stage properties (since version 0.1 : keen)

In the build_stages event collection, these are added :

  • job : see Build/job properties (since version 0.1 : keen, renamed in version 0.2 (previously build))
  • stage : see Stage properties (since version 0.1 : keen)

In the build_substages event collection, these are added :

  • job : see Build/job properties (since version 0.1 : keen, renamed in version 0.2 (previously build))
  • stage : see Stage properties (since version 0.1 : keen)

Build/job properties

  • repo : repo name (since version 0.1 : native, keen)
  • branch : repo branch (since version 0.1 : native, keen)
  • build : build id (since version 0.1 : native, keen)
  • job : job id (since version 0.1 : native, keen)
  • result : build result : passed, failed (since version 0.1 : native, keen)
  • worker : CI worker information (since version 0.2 : keen) :
    • hostname: worker hostname (since version 0.2 : keen)
    • os: worker OS (since version 0.2 : keen)
  • started_at : timestamp when build started, see Timestamp properties for structure (since version 0.1, updated in version 0.2 : native, keen)
  • finished_at : timestamp when build finished, see Timestamp properties for structure (since version 0.1, updated in version 0.2 : keen)
  • duration : build duration [sec] (since version 0.1 : keen)
  • ci_platform : Continuous integration platform : travis, local, ... (since version 0.1 : native, keen)
  • build_trigger : what triggered the build : push, pull_request (since version 0.3 : keen)
  • build_matrix : build matrix parameters (since version 0.3 : keen) :
    • language : programming language, see Travis CI documentation for full list (since version 0.3 : keen)
    • language_version : programming language version (since version 0.3 : keen)
    • language_components : language components, fe.: android_20 (since version 0.3 : keen)
    • jdk : Java JDK version (Java, Android, Groovy, Ruby, Scala), fe. : openjdk7 (since version 0.3 : keen)
    • ghc : Haskell compiler (since version 0.3 : keen)
    • lein : Lein version (Clojure) (since version 0.3 : keen)
    • mono : Mono version (C#, F#, Visual Basic) (since version 0.3 : keen)
    • node_js : Node JS version (Javascript) (since version 0.3 : keen)
    • otp_release : OTP release version (Erlang) (since version 0.3 : keen)
    • rvm: RVM version (Ruby, Objective-C) (since version 0.3 : keen)
    • gemfile: Gemfile (Ruby, Objective-C) (since version 0.3 : keen)
    • os : Operating system, fe. : linux, OS X (since version 0.3 : keen)
    • compiler : Compiler, fe. : clang, gcc (since version 0.3 : keen)
    • parameters : extra build matrix environment parameters (since version 0.3 : keen)
    • xcode_sdk : Objective-C SDK (since version 0.3 : keen)
    • xcode_scheme : Objective-C Scheme (since version 0.3 : keen)
    • xcode_project : Objective-C project (since version 0.3 : keen)
    • xcode_workspace : Objective-C workspace (since version 0.3 : keen)
    • summary : Concatenation of all build matrix parameters into one string (since version 0.3 : keen)
  • pull_request : pull request information (since version 0.3 : keen)
    • is_pull_request : true, false (since version 0.3 : keen)
    • title : Pull request title (since version 0.3 : keen)
    • number : Pull request number (since version 0.3 : keen)

Future build properties :

  • language : java, C, python, php, ... (not implemented yet)
  • language version : openJDK 7, Python 2.7, PHP 5.5, ... (not implemented yet)
  • framework : Spring, Android, Django, ... (not implemented yet)
  • buildtool : make, ant, maven, gradle, ... (not implemented yet)
  • commit (not implemented yet)

Stage properties

  • name : stage name (since version 0.1 : native, keen)
  • started_at : timestamp when build started, see Timestamp properties for structure (since version 0.1, updated in version 0.2 : keen)
  • finished_at : timestamp when build finished, see Timestamp properties for structure (since version 0.1, updated in version 0.2 : keen)
  • duration : build duration [sec] (since version 0.1 : keen)

Timestamp properties

  • timestamp : iso formatted timestamp (since version 0.2 : keen)
  • year : Year as 4 digit number (since version 0.2 : keen)
  • month : Month as 2 digit number, with leading zero (01, 02, ..., 11, 12) (since version 0.2 : keen)
  • month_full_en : Full name of the month in en_US (Januari, Februari, ...) (since version 0.2 : keen)
  • month_short_en : Abbreviated name of the month in en_US (Jan, Feb, ...) (since version 0.2 : keen)
  • day_of_month : Day of the month as 2 digit number, with leading zero (01, 02, ..., 30, 31) (since version 0.2 : keen)
  • day_of_week : Day of week as a number (0 = Sunday, 1 = Monday, ..., 6 = Saturday) (since version 0.2 : keen)
  • day_of_week_short_en : Abbreviated name of day of the week in en_US (Mon, Tue, ...) (since version 0.2 : keen)
  • day_of_week_full_en : Full name of day of the week in en_US (Sunday, Monday, ...) (since version 0.2 : keen)
  • hour_24 : Hour in 24-hour format as 2 digit number, with leading zero (00, 01, ..., 22, 23) (since version 0.2 : keen)
  • hour_12 : Hour in 12-hour format as 2 digit number, with leading zero (01, 02, ..., 11, 12) (since version 0.2 : keen)
  • hour_ampm : AM/PM (since version 0.2 : keen)
  • minute : Minute as 2 digit number, with leading zero (00, 01, 02, ..., 58, 59) (since version 0.2 : keen)
  • second : Second as 2 digit number, with leading zero (00, 01, 02, ..., 58, 59) (since version 0.2 : keen)

Event collection build_stages or build_substages example :

{
    "buildtime_trend": {
        "project_name": "ruleant/getback_gps",
        "client": "buildtimetrend/service",
        "client_version": "0.2-dev",
        "lib_version": "0.2-dev",
        "schema_version": "2"
    },
    "keen": {
        "timestamp": "2014-08-22T15:01:49.782Z",
        "created_at": "2014-08-22T15:01:49.782Z",
        "id": "53f75b5d7d8cb9684aa6a904"
    },
    "job": {
        "result": "passed",
        "finished_at": {
            "timestamp": "2014-08-22T15:01:46",
            "day_of_week_short_en": "Fri",
            "day_of_month": "22",
            "day_of_week": "5",
            "month": "08",
            "month_full_en": "August",
            "second": "46",
            "month_short_en": "Aug",
            "year": "2014",
            "hour_24": "15",
            "hour_12": "03",
            "day_of_week_full_en": "Friday",
            "minute": "01",
            "hour_ampm": "PM"
        },
        "worker": {
            "hostname": "worker-linux-docker-1bb6ca34.prod.travis-ci.org",
            "os": "travis-linux-5"
        },
        "repo": "ruleant/getback_gps",
        "job": "438.2",
        "build": "438",
        "branch": "test_travis_cache",
        "duration": 522,
        "ci_platform": "travis",
        "started_at": {
            "timestamp": "2014-08-22T14:51:15+00:00",
            "day_of_week_short_en": "Fri",
            "day_of_month": "22",
            "day_of_week": "5",
            "month": "08",
            "month_full_en": "August",
            "second": "15",
            "month_short_en": "Aug",
            "year": "2014",
            "hour_24": "14",
            "hour_12": "02",
            "day_of_week_full_en": "Friday",
            "minute": "51",
            "hour_ampm": "PM"
        }
    },
    "stage": {
        "name": "update_javadoc",
        "duration": 31,
        "finished_at": {
            "timestamp": "2014-08-22T15:01:46",
            "day_of_week_short_en": "Fri",
            "day_of_month": "22",
            "day_of_week": "5",
            "month": "08",
            "month_full_en": "August",
            "second": "46",
            "month_short_en": "Aug",
            "year": "2014",
            "hour_24": "15",
            "hour_12": "03",
            "day_of_week_full_en": "Friday",
            "minute": "01",
            "hour_ampm": "PM"
        },
        "started_at": {
            "timestamp": "2014-08-22T15:01:15",
            "day_of_week_short_en": "Fri",
            "day_of_month": "22",
            "day_of_week": "5",
            "month": "08",
            "month_full_en": "August",
            "second": "15",
            "month_short_en": "Aug",
            "year": "2014",
            "hour_24": "15",
            "hour_12": "03",
            "day_of_week_full_en": "Friday",
            "minute": "01",
            "hour_ampm": "PM"
        },
   }

}

Event collection build_jobs example :

{
    "buildtime_trend": {
        "project_name": "ruleant/buildtime-trend",
        "client": "buildtimetrend/service",
        "client_version": "0.2-dev",
        "lib_version": "0.2-dev",
        "schema_version": "2"
    },
    "keen": {
        "timestamp": "2014-08-22T10:20:15.295Z",
        "created_at": "2014-08-22T10:20:15.295Z",
        "id": "53f7195fc9e16340d0eed747"
    },
    "job": {
        "result": "passed",
        "started_at": {
            "timestamp": "2014-08-22T10:17:44+00:00",
            "day_of_week_short_en": "Fri",
            "day_of_month": "22",
            "day_of_week": "5",
            "month": "08",
            "month_full_en": "August",
            "second": "44",
            "month_short_en": "Aug",
            "year": "2014",
            "hour_24": "10",
            "hour_12": "10",
            "day_of_week_full_en": "Friday",
            "minute": "17",
            "hour_ampm": "AM"
        },
        "finished_at": {
            "timestamp": "2014-08-22T10:20:12",
            "day_of_week_short_en": "Fri",
            "day_of_month": "22",
            "day_of_week": "5",
            "month": "08",
            "month_full_en": "August",
            "second": "12",
            "month_short_en": "Aug",
            "year": "2014",
            "hour_24": "10",
            "hour_12": "10",
            "day_of_week_full_en": "Friday",
            "minute": "20",
            "hour_ampm": "AM"
        },
        "worker": {
            "hostname": "worker-linux-docker-1bb6ca34.prod.travis-ci.org",
            "os": "travis-linux-5"
        },
        "repo": "ruleant/buildtime-trend",
        "job": "252.1",
        "build": "252",
        "branch": "master",
        "duration": 145,
        "ci_platform": "travis"
        "stages": [
            {
                "duration": 136,
                "finished_at": {
                    "timestamp": "2014-08-22T10:20:03",
                    "day_of_week_short_en": "Fri",
                    "day_of_month": "22",
                    "day_of_week": "5",
                    "month": "08",
                    "month_full_en": "August",
                    "second": "03",
                    "month_short_en": "Aug",
                    "year": "2014",
                    "hour_24": "10",
                    "hour_12": "10",
                    "day_of_week_full_en": "Friday",
                    "minute": "20",
                    "hour_ampm": "AM"
                },
                "started_at": {
                    "timestamp": "2014-08-22T10:17:47",
                    "day_of_week_short_en": "Fri",
                    "day_of_month": "22",
                    "day_of_week": "5",
                    "month": "08",
                    "month_full_en": "August",
                    "second": "47",
                    "month_short_en": "Aug",
                    "year": "2014",
                    "hour_24": "10",
                    "hour_12": "10",
                    "day_of_week_full_en": "Friday",
                    "minute": "17",
                    "hour_ampm": "AM"
                },
                "name": "install"
           },
           {
                "duration": 5,
                "finished_at": {
                     "timestamp": "2014-08-22T10:20:08",
                    "day_of_week_short_en": "Fri",
                    "day_of_month": "22",
                    "day_of_week": "5",
                    "month": "08",
                    "month_full_en": "August",
                    "second": "08",
                    "month_short_en": "Aug",
                    "year": "2014",
                    "hour_24": "10",
                    "hour_12": "10",
                    "day_of_week_full_en": "Friday",
                    "minute": "20",
                    "hour_ampm": "AM"
               },
               "started_at": {
                    "timestamp": "2014-08-22T10:20:03",
                    "day_of_week_short_en": "Fri",
                    "day_of_month": "22",
                    "day_of_week": "5",
                    "month": "08",
                    "month_full_en": "August",
                    "second": "03",
                    "month_short_en": "Aug",
                    "year": "2014",
                    "hour_24": "10",
                    "hour_12": "10",
                    "day_of_week_full_en": "Friday",
                    "minute": "20",
                    "hour_ampm": "AM"
                },
                "name": "tests"
            }
        ],
    }
}