[Enhancement?] Build description or similar

[jakimfett]

Is there currently a way to add a "build description", eg some text that goes with the build (perhaps in the .run file?) that allows comment(s), instructions, or warnings to be communicated via the web frontend?

As the build hierarchy gets more convoluted, half the time I need to add notes for myself, and if I'm having trouble keeping the builds straight as the one setting them up, the end user(s) are definitely going to need something to nudge them towards the intended purpose.

[ohwgiles]

This is a nice idea. Could be done lots of ways, off the top of my head:

• a new setting in the job's .conf file
• a special comment format in the .run file that laminar could parse
• a separate file JOBNAME.html which might allow for larger/more complex modifications to the frontend

[jakimfett]

Is the JOBNAME.conf file format/usage documented somewhere?

I'm a fan of reducing the number of files necessary for a job to 'just work as intended', so would very much prefer an option that uses the first two ways mentioned (or something adjacent), rather than the third option.

If you want large and complex, imo, go install Jenkins or something similarly heavyweight.
I just need a way to say "this requires library X at location Y" or "compiled with BINDATA flag", etc.

[ohwgiles]

Is the JOBNAME.conf file format/usage documented somewhere?

It's mentioned twice in the docs, but there isn't a single reference point, there probably should be.

I would also like to reduce the number of necessary files. I've considered moving the .conf file to parseable script comments, but that's no good for symlinks or binaries; or using extended attributes, but that's a bit esoteric and doesn't survive certain file systems.

[jakimfett]

Thoughts on a template .run file that does the entire thing?

Something like:
(haven't actually tested the code)

#!/bin/bash -ex
# A template file for Laminar Continuous Integration

function description {
    cat << EOF
        Add a description for your job.
        Will be displayed at <url>/jobs/<jobname> next to the graph.
    EOF
}

function initialize {
    git clone <repo-name>
}

function runJob {
    make tests
    make DESTDIR=some/location install
}

if [ ! -d "${WORKSPACE}" ]; then initialize; fi

The one-liner at the end checks for the existence of the workspace, and runs the initialize function if-not-exists, which would in theory allow the .init file to be merged into the JOBNAME.run file, and avoids creating a special paradigm for the description?

[ohwgiles]

No. This brings a hard dependency on bash. Laminar .run files can be anything executable: Bash, Python, Perl, compiled binary, symlink to executable etc.

[jakimfett]

That makes sense, thanks for the clarification.

In that paradigm, it does make sense (in my opinion) to use a parse-able comment in the .run file more than the other options, therein keeping the extra files to a minimum.

The simplicity of setting up jobs is still my favourite part of the laminar system, and a huge part of why it's a massive step up from systems like drone/Jenkins/BuildBot/etc.

---

Tangentially, is there a way to trigger some sort of 'purge-and-re-initialize-workspace' function?
Right now, I've just created a job that does an rm across all the job folders, targeting the workspace, but I'm running into concurrency problems. <-- moved to #100 per request.

[ohwgiles]

a parse-able comment in the .run file more than the other options, therein keeping the extra files to a minimum

Yes, but that won't work for binaries, and introduces some parsing complexity. I'm not sure that it's a better approach than the other two options.

Tangentially, is there a way

Not sure I understand exactly what you mean. Could you please open a new issue for this question?

[jakimfett]

@ohwgiles commented:
...that won't work for binaries...

Can you help me understand what you mean by this?
Are you saying that if the <job>.run file is a binary, the comment cannot be pulled out by the web UI?
Are you saying that, if the comment is in the <job>.run file, the binary cannot pull the comment?

ohwgiles also commented:
...options...

Would it make more sense to have a folder-per-job, or is the cfg/jobs/ directory intended to be a flat hierarchy?

[ohwgiles]

Are you saying that if the .run file is a binary, the comment cannot be pulled out by the web UI?

Yes. Probably no-one will ever do this, but laminar promises to run anything executable. Requiring config options to be set in comments would restrict this promise. That's an option on the table, I was just explaining the downside to this particular method.

Are you saying that, if the comment is in the .run file, the binary cannot pull the comment?

No, this implementation could fairly easily be added to laminard.

Would it make more sense to have a folder-per-job

It would if there ends up being lots of files per job, but I think this is not the case. Typically there is only the .run file. Of course this might change if setting a build description required another file...

is the cfg/jobs/ directory intended to be a flat hierarchy?

Yes....as long as this stays manageable. I would like to keep it that way if possible.

[jakimfett]

@ohwgiles commented:
...laminar promises to run anything executable...config options...in comments...restrict this promise...

That's glorious, and I'd drop my vote in the 'keep this promise intact' hat, every time.
I can already think of ways my team can use this (executing binaries) in the upcoming months.

@ohwgiles also commented:
...could fairly easily be added...

...but, you've convinced me that a comment in the runfile is not an ideal implementation.
Thanks for taking the time to explain.

@ohwgiles also commented:
...lots of files per job...

What's your threshold for 'lots of files'?
All of my jobs have a .init, as well as the .run, and having a <jobname>.desc would round it out to three, which is kinda my soft upper limit on 'manageable'.

I can't currently think of a reason to add more than this (as the .desc file could contain more than just the job description, in theory), but I'm also curious where you are on this.

---

Flat hierarchy adjacent:
...how hard would it be to check if <jobname>.run is a directory, and if so, treat the entire contents as the .run directive, similar to how e.g. the /etc/<whatever>.d/ nomenclature for system config files works?

[ohwgiles]

What's your threshold for 'lots of files'? [...] three, which is kinda my soft upper limit on 'manageable'.

I agree that any more than approximately three is not sustainable

as the .desc file could contain more than just the job description, in theory

There is already an optional <jobname>.conf, which can be used to add tags to a job. This is currently my preferred place to add a description field, which I think is equivalent to your suggestion.

check if .run is a directory,

You mean .conf and .init would remain outside, but all scripts in this directory would be run as part of the job? In alphabetical order? Interesting idea, could you elaborate on your use case? My initial reaction is that I'm not sure it should go into laminar core because it can be done fairly trivially with a script; say if you created a directory job.run.d full of scripts, and job.run with for script in ${BASH_SOURCE[0]}.d/*; do if ! $script; then exit $?; fi; done

[jakimfett]

@ohwgiles the followup on my laminar queries has been exceptional, and I'm looking forward to getting description pages rolled out for all my builds!

Your commitment and actions towards building efficient, scaleable software has inspired and motivated my own efforts towards the same, and I've been recommending laminar whenever the topic of build pipeline automation is discussed.

Thank you again for making an awesome tool, and for the attention paid to cohesive design that this project demonstrates.

[ohwgiles]

@jakimfett my pleasure. Thank you for your support and your kind words :)