Added support for metadata for job runs #208

maxses · 2024-03-27T21:42:49Z

With the laminarc tag command the build scripts can augment runs with metadata. This can be SCM-oriented information like the used branch or source-version.
The metadata is a list of key-value-pairs. The metadata is automatically shown in the run-view of a job in the web UI.

Each call of laminarc tag can assign one key/value pair.

Example:

laminarc tag $JOBNAME $NUMBER` \
		Branch "$(git -C $WORKSPACE branch --show-current)"

With the `laminarc tag` command the build scripts can augment runs with metadata. This can be SCM-oriented information like the used branch or source-version. The metadata is a list of key/value pairs. The metadata is automatically shown in the run-view of a job in the web UI. Each call of `laminarc tag` can assign one key/value pair. Example: ``` laminarc tag $JOBNAME $NUMBER` \ Branch "$(git -C $WORKSPACE branch --show-current)" ```

All keys from metainfo of all shown runs is concatenated. So only relevant colums for metadata is shown.

pkg-config is needed in docker image. "laminar.conf" has to be copied from "usr/etc/laminar.conf" to "etc/laminar.conf" in installation directory.

ohwgiles · 2024-03-28T22:25:53Z

CMakeLists.txt

@@ -60,6 +60,7 @@ set(CMAKE_INCLUDE_CURRENT_DIR ON)
 set(CMAKE_CXX_STANDARD 17)

 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare")
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=deprecated-declarations" )


What caused this? If it is not related to this change it should be dealt with separately. Also consider that only Debug builds get -Werror, so if this is really needed then it should be put in CMAKE_CXX_FLAGS_DEBUG

ohwgiles · 2024-03-28T22:26:37Z

CMakeLists.txt

@@ -142,6 +143,7 @@ set(LAMINARD_CORE_SOURCES
    src/run.cpp
    src/server.cpp
    src/version.cpp
+    src/json.cpp


I don't think it is necessary to break this out into a separate file - see later comments for explanation

ohwgiles · 2024-03-28T22:29:01Z

UserManual.md

+
+Example:
+```
+laminarc tag $JOBNAME $NUMBER Branch "$(git -C $WORKSPACE branch --show-current)"


Since we only intend this to be called from within the run, laminarc can handle reading $JOBNAME and $NUMBER, see for example what set does

ohwgiles · 2024-03-28T22:33:50Z

src/laminar.cpp

+       return run->tag(key, value);
+   else
+   {
+      LLOG(WARNING, "No active run with ", job, buildNum);


return false (propagates error to laminarc)

ohwgiles · 2024-03-28T22:39:01Z

src/run.h

+    // store metadata into run
+    bool tag(const std::string& key, const std::string& value);
+    // Return the meta as JSON text
+    std::string getMetaDataJsonString();


I don't think Run should know about JSON at all. Just do

const std::map<std::string, std::string>& metadata() const { return metadata; }

and do the conversion to JSON in the Laminar class with a local method there.

ohwgiles · 2024-03-28T22:42:50Z

src/run.h

@@ -101,6 +109,8 @@ class Run {
    kj::ForkedPromise<void> startedFork;
    kj::PromiseFulfillerPair<RunState> finished;
    kj::ForkedPromise<RunState> finishedFork;
+
+    std::map<std::string, std::string> metaDataMap;


There is a minor consideration here: what order should the tags show up in the frontend? Using std::map means they will show up alphabetically, but I wonder if it would be preferable for them to show up in the order that they were created? Unfortunately I think that would require using vector<pair<string,string>> and manually checking for duplicates, or using boost::multi_index_container

Alphabetically is also fine

ohwgiles · 2024-03-28T22:43:46Z

src/run.cpp

+   LLOG(INFO, "Setting tag ", key, value );
+   metaDataMap[key]=value;
+   return true;
+}


make it void if it cannot fail

ohwgiles · 2024-03-28T22:48:20Z

src/resources/index.html

+     <template v-show="Object.keys(job.metadata).length" v-for="(value, key) in job.metadata">
+       <dt>{{key}}</dt>
+       <dd>{{value}}</dd>
+     </template>


I think this would look better before / on top of the artifacts

ohwgiles · 2024-03-28T22:59:09Z

src/resources/index.html

@@ -131,20 +131,25 @@ <h2>{{route.params.name}}</h2>
   <canvas id="chartBt"></canvas>
  </div>
  <div style="grid-column: 1/-1">
+   <div v-show="false">{{metaDataKeys=["XBranch", "XVersion"]}}</div>
+   <div v-show="false">{{allJobs=jobsQueued.concat(jobsRunning).concat(jobsRecent)}}</div>
+   <div v-show="false">{{metaDataKeys=uniteMetadataKeys(allJobs)}}</div>


I think metadataKeys is best generated in the JS at the time the run list is received, and updated when a new run joins the list

ohwgiles · 2024-03-28T23:04:46Z

src/resources/index.html

@@ -131,20 +131,25 @@ <h2>{{route.params.name}}</h2>
   <canvas id="chartBt"></canvas>
  </div>
  <div style="grid-column: 1/-1">
+   <div v-show="false">{{metaDataKeys=["XBranch", "XVersion"]}}</div>


ohwgiles · 2024-03-28T23:06:27Z

src/resources/index.html

   <table class="striped">
    <thead><tr>
     <th><a class="sort" :class="(sort.field=='result'?sort.order:'')" v-on:click="do_sort('result')">&nbsp;</a></th>
     <th>Run <a class="sort" :class="(sort.field=='number'?sort.order:'')" v-on:click="do_sort('number')">&nbsp;</a></th>
     <th class="text-center">Started <a class="sort" :class="(sort.field=='started'?sort.order:'')" v-on:click="do_sort('started')">&nbsp;</a></th>
     <th class="text-center">Duration <a class="sort" :class="(sort.field=='duration'?sort.order:'')" v-on:click="do_sort('duration')">&nbsp;</a></th>
     <th class="text-center vp-sm-hide">Reason <a class="sort" :class="(sort.field=='reason'?sort.order:'')" v-on:click="do_sort('reason')">&nbsp;</a></th>
+     <th class="text-center vp-sm-hide" v-for="(value, key) in metaDataKeys">{{key}}<a class="sort" :class="(sort.field=='reason'?sort.order:'')" v-on:click="do_sort('reason')">&nbsp;</a></th>


Do not sort by reason. It should be pretty simple to add sorting by metadata fields, using something like 'meta_' + key for sort.field and in the backend using ORDER BY metadata -> $key, but also happy just to not implement sorting for metadata

ohwgiles · 2024-03-28T23:10:03Z

pkg/debian11-amd64.sh

@@ -34,6 +34,8 @@ Depends: libcapnp-0.7.0, libsqlite3-0, zlib1g
 Description: Lightweight Continuous Integration Service
 EOF
 echo /etc/laminar.conf > laminar/DEBIAN/conffiles
+mkdir -p laminar/etc
+mv laminar/usr/etc/laminar.conf laminar/etc/laminar.conf


I think this is a bug introduced elsewhere and this is not the right place to fix it

Fixed here 8c3d7f6

ohwgiles · 2024-03-28T23:10:13Z

pkg/debian11-amd64.sh

@@ -8,7 +8,7 @@ VERSION=$(cd "$SOURCE_DIR" && git describe --tags --abbrev=8 --dirty)-1~upstream

 DOCKER_TAG=$(docker build -q - <<EOS
 FROM debian:11-slim
-RUN apt-get update && apt-get install -y wget cmake g++ capnproto libcapnp-dev rapidjson-dev libsqlite3-dev libboost-dev zlib1g-dev
+RUN apt-get update && apt-get install -y wget cmake g++ capnproto libcapnp-dev rapidjson-dev libsqlite3-dev libboost-dev zlib1g-dev pkg-config


maxses force-pushed the feature/metadata branch from bba5a48 to 9c1cfad Compare March 27, 2024 22:43

maxses mentioned this pull request Mar 27, 2024

added support for build.env #207

Closed

maxses force-pushed the feature/metadata branch from 9c1cfad to accdff2 Compare March 27, 2024 23:20

maxses added 2 commits March 28, 2024 20:04

Show metainfo in the jobs run view

f6fe08b

All keys from metainfo of all shown runs is concatenated. So only relevant colums for metadata is shown.

Fixed building deb package for x86

4e01d14

pkg-config is needed in docker image. "laminar.conf" has to be copied from "usr/etc/laminar.conf" to "etc/laminar.conf" in installation directory.

ohwgiles reviewed Mar 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for metadata for job runs #208

Added support for metadata for job runs #208

maxses commented Mar 27, 2024 •

edited

Loading

ohwgiles Mar 28, 2024

ohwgiles Mar 28, 2024

ohwgiles Mar 28, 2024

ohwgiles Mar 28, 2024

ohwgiles Mar 28, 2024

ohwgiles Mar 28, 2024

ohwgiles Apr 1, 2024

ohwgiles Mar 28, 2024

ohwgiles Mar 28, 2024

ohwgiles Mar 28, 2024

ohwgiles Mar 28, 2024

ohwgiles Mar 28, 2024

ohwgiles Mar 28, 2024

ohwgiles Apr 1, 2024

ohwgiles Mar 28, 2024

Added support for metadata for job runs #208

Are you sure you want to change the base?

Added support for metadata for job runs #208

Conversation

maxses commented Mar 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxses commented Mar 27, 2024 •

edited

Loading