Add package builds #1965

ingwinlu · 2018-05-08T09:48:12Z

Purpose

Enable debian package generation on the build server to replace the existing one.

Checklist

commit messages are fine ("module: short statement" syntax and refer to issues)
affected documentation is fixed
release notes are updated (doc/news/_preparation_next_release.md)

resolves #1911
resolves #1752

ingwinlu · 2018-05-09T14:25:59Z

jenkins build jenkinsfile[package/stretch] please

ingwinlu · 2018-05-09T15:18:28Z

@markus2330 after struggling with jenkins way of handling git checkouts (so much automagic going on) I got the build going. However the test failures do not appear when I run it locally so I will have to look into it.

ingwinlu · 2018-05-09T16:27:55Z

jenkins build jenkinsfile please

ingwinlu · 2018-05-09T18:12:35Z

@petermax2 any idea why https://github.com/ElektraInitiative/libelektra/blob/master/src/plugins/crypto/test_internals.h#L263 might throw an error here? When I run the command locally (in the same docker image) it passes...

petermax2 · 2018-05-09T18:20:47Z

any idea why https://github.com/ElektraInitiative/libelektra/blob/master/src/plugins/crypto/test_internals.h#L263 might throw an error here? When I run the command locally (in the same docker image) it passes...

Maybe the GPG key is already installed on the build server, so that GPG does not exit with 0? But this is just a wild guess.

ingwinlu · 2018-05-09T18:31:20Z

jenkins build jenkinsfile[package/stretch] please

ingwinlu · 2018-05-09T18:56:44Z

Maybe the GPG key is already installed on the build server, so that GPG does not exit with 0? But this is just a wild guess.

Should not be the case as the docker container should be 'fresh' on the build server as well.

It seems like errorKey is empty. Or am I using elektra wrong?

petermax2 · 2018-05-09T19:05:53Z

What content does msg have? Maybe the stderr output of GPG is empty, but there might be a response in stdout (which goes to msg if I am not mistaking).

ingwinlu · 2018-05-09T19:21:23Z

@petermax2 msg only holds the private key sent to gpg

petermax2 · 2018-05-09T19:24:11Z

msg only holds the private key sent to gpg

Hm, no errorMsg and no msg. That's weird.

ingwinlu · 2018-05-09T20:01:25Z

Since it is a docker only problem I assume it is related to tty issues. error code returned is 512 but I can not get it to print an error message

ingwinlu · 2018-05-09T20:14:01Z

jenkins build jenkinsfile[package/stretch] please

ingwinlu · 2018-05-09T20:27:10Z

@markus2330 any insight on why testmod_blockresolver is failing? maybe we can apply that to the crypto stuff

markus2330 · 2018-05-09T20:35:46Z

No, I can only guess: The genTempFilename could be non-unique: It uses tv.tv_usec but maybe docker does not give precise time? @tom-wa do you have any insight?

My idea was that both fcrypt and blockresolver work with temp files. But it easily could be a wrong track.

markus2330 · 2018-05-10T06:40:33Z

I now get similar errors when building the package:

 42/118 Test  #48: testmod_blockresolver ............***Failed    0.00 sec
BLOCKRESOLVER     TESTS
==================

/home/markus/Projekte/Elektra/libelektra/tmpbuilddir/libelektra/src/plugins/blockresolver/testmod_blockresolver.c:70: error in test_BlockresolverWrite: blockresolver->kdbSet failed
Compare <key = only the inside has changed
>, with <key = inside block
>
in file /home/markus/Projekte/Elektra/libelektra/tmpbuilddir/libelektra/src/plugins/blockresolver/blockresolver/compare.block, line 5.
/home/markus/Projekte/Elektra/libelektra/tmpbuilddir/libelektra/tests/cframework/tests.c:157: error in compare_line_files: comparing lines failed
/home/markus/Projekte/Elektra/libelektra/tmpbuilddir/libelektra/src/plugins/blockresolver/testmod_blockresolver.c:72: error in test_BlockresolverWrite: files do not match as expected

testmod_blockresolver Results: 15 Tests done — 3 errors

        Start  73: testmod_fcrypt
 57/118 Test  #73: testmod_fcrypt ...................***Failed    0.01 sec
FCRYPT       TESTS
==================

/home/markus/Projekte/Elektra/libelektra/tmpbuilddir/libelektra/src/plugins/fcrypt/testmod_fcrypt.c:187: error in test_file_crypto_operations: kdb set failed
/home/markus/Projekte/Elektra/libelektra/tmpbuilddir/libelektra/src/plugins/fcrypt/testmod_fcrypt.c:188: error in test_file_crypto_operations: file content did not change during encryption
/home/markus/Projekte/Elektra/libelektra/tmpbuilddir/libelektra/src/plugins/fcrypt/testmod_fcrypt.c:191: error in test_file_crypto_operations: kdb get (pregetstorage) failed
/home/markus/Projekte/Elektra/libelektra/tmpbuilddir/libelektra/src/plugins/fcrypt/testmod_fcrypt.c:195: error in test_file_crypto_operations: kdb get (postgetstorage) failed
/home/markus/Projekte/Elektra/libelektra/tmpbuilddir/libelektra/src/plugins/fcrypt/testmod_fcrypt.c:196: error in test_file_crypto_operations: postgetstorage did not encrypt the file again
/home/markus/Projekte/Elektra/libelektra/tmpbuilddir/libelektra/src/plugins/fcrypt/testmod_fcrypt.c:231: error in test_file_signature_operations: kdb set failed
/home/markus/Projekte/Elektra/libelektra/tmpbuilddir/libelektra/src/plugins/fcrypt/testmod_fcrypt.c:232: error in test_file_signature_operations: file content did not change during encryption
/home/markus/Projekte/Elektra/libelektra/tmpbuilddir/libelektra/src/plugins/fcrypt/testmod_fcrypt.c:235: error in test_file_signature_operations: kdb get failed

So at least these two errors are not related to docker.

markus2330 · 2018-05-10T06:41:16Z

Btw. if building the packages fails because of some python files (not installed), please revert b0e54fc

ingwinlu · 2018-05-10T06:44:13Z

But it works locally for me, something is effed up bad.
I will rebuild my docker image locally with the latest debian stretch image (mine was a few weeks old). maybe that was it (that would also explain it failing outside of docker if it is debian release related).

markus2330 · 2018-05-10T07:03:01Z

Weird, I get these two errors reproducible on two Debian stretch machines, regardless of if I cleaned up /tmp files.

ingwinlu · 2018-05-10T07:45:30Z

still can not reproduce it and I have run out of ideas on what I could try

ingwinlu · 2018-05-10T09:24:54Z

@markus2330 can you run a printenv | sort on the machines where you could reproduce it? I can not even get it to reproduce with the same docker image on a7...

markus2330 · 2018-05-10T15:54:40Z

env -i PATH=$PATH scripts/build-debian-package and env -i PATH=/usr/bin:/bin scripts/build-debian-package also fails with the same two test cases, so the env is irrelevant.

I am afraid we will have to bisect which change introduced the problem. Seems like the build job was long enough not working to let new errors creep in.

Thanks for your work on the build server. There is evidence again and again that it is really important to detect faulty commits quickly. In the situation we are now, it is really hard to find out why this fails.

ingwinlu · 2018-05-10T16:33:40Z

strangely enough the build on stretch mr still passes the tests and only fails when signing so I don't think there is a faulty commit that we can pinpoint the behavior to. More likely something in the environment is inconsistent that would explain why it works on some machines and some not.

however i have no idea why i can not reproduce it with the docker container when running the steps independently...

ingwinlu · 2018-05-12T14:24:04Z

jenkins build jenkinsfile please

markus2330 · 2018-05-12T16:05:07Z

Thank you for your great last-minute-commitment to finally fix the stretch repos and the nice release notes!

A small remark: The convention is to set "ready to merge" not before the build server finished.

markus2330

Thank you, looks great. As always, more docu welcomed!

markus2330 · 2018-05-12T16:07:48Z

doc/news/_preparation_next_release.md

+responsivness.
+It focuses heavily on containerisation to improve hardware utilization.
+If you are interested in `#devops` have a look at our
+[Jenkinsfile](https://github.com/ElektraInitiative/libelektra/blob/master/scripts/jenkins/Jenkinsfile).


Can you write a line or two about how to get docker images locally?

Currently the setup does not allow unauthenticated pulls from our repository.

markus2330 · 2018-05-12T16:08:44Z

doc/news/_preparation_next_release.md

+Add it to your `sources.list`:
+```
+deb     [trusted=yes] https://debian-stretch-repo.libelektra.org/ stretch main
+deb-src [trusted=yes] https://debian-stretch-repo.libelektra.org/ stretch main


Please also add to doc/INSTALL.md

Already did.

markus2330 · 2018-05-12T16:09:52Z

scripts/jenkins/Jenkinsfile

+        withEnv(["DEBSIGN_PROGRAM=gpg",
+                 "DEBFULLNAME=Jenkins (User for Elektra automated build system)",
+                 "DEBEMAIL=autobuilder@libelektra.org"]) {
+            sh "rm -R ./*"


sounds dangerous. Maybe put into a function "initialCleanup"?

It isn't, it is guaranteed to be in a jenkins workspace in a docker container.

A function seems to be useful nevertheless, even though it is trivial.

Btw. Why not delete files starting with .?

The function might already exist:

https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#deletedir-recursively-delete-the-current-directory-from-the-workspace

I know I used it in the very same file. But it behaves differently as it deletes the current directory, not just the content of the current directory.

markus2330 · 2018-05-12T16:10:29Z

scripts/jenkins/Jenkinsfile

+    return tasks
+}
+
+def build_package_debian_stretch() {


why are some functions camel case and others use _ ? (Not to be changed in this PR, though)

I wanted to make it clear that some are to be used likeSteps (and hence need a node) while all_the_other_need_not.

//EDIT I have not settled on if this is a good idea hence it is not documented anywhere.

This would be useful to have written somewhere at the top!

again i am not even sure if i followed it consequently hence it is not final to stay that way yet.

Okay, as said this does not have to be resolved within this PR.

markus2330 · 2018-05-12T16:11:00Z

scripts/jenkins/Jenkinsfile

+                $class: 'GitSCM',
+                branches: scm.branches,
+                extensions: scm.extensions + [
+                    [$class: 'PerBuildTag'],


What do all these extensions do? Can you point to the docu of where you found all these knobs?

all this can be found here https://build.libelektra.org/jenkins/job/elektra-jenkinsfile/pipeline-syntax

I got 404. Please link it in the beginning of the file.

you need to login. it is just a collection of available commands on the server. you can use any other jenkins dsl documentation as well.

googling jenkinsfile reference brings the full list available via https://jenkins.io/doc/pipeline/steps/

Any reason why the file is not visible without login?

Yes, such things should be in the intro of the file.

Any reason why the file is not visible without login?

No Idea.

we don't link c tutorials in c files, we don't link manpages in our scripts, what makes the Jenkinsfile so different?

we don't link c tutorials in c files, we don't link manpages in our scripts,

Because it is not one of the development languages and nevertheless nearly everybody will have to do something with it (with minimal knowledge of the language).

And the Jenkinsfile had little docu even compared to C code. But it slowly improves. 😉

what makes the Jenkinsfile so different?

If I understood you correctly, Jenkinsfile might be written in either groovy or in a DSL. So you need to specify which one we use now.

it will always be 'groovyish', the choice is between a declarative and a 'scripted' version of the DSL. we currenlty use the scripted one because the declarative has been not complex enough.

However they are to some degree interchangeable.

Because it is not one of the development languages and nevertheless nearly everybody will have to do something with it (with minimal knowledge of the language).

The problem with documenting essential language features is that it is very much a moving target. Patches come out at least once a week.

Linking the documentation that can be found via simple google queries is not providing any benefit, else you would add c tutorials to every c header, or cmake documentation to every cmakelists file.

I would prefer to stop having those documentation discussions till I have finished the accompanying doc/*.md file.

it will always be 'groovyish', the choice is between a declarative and a 'scripted' version of the DSL. we currenlty use the scripted one because the declarative has been not complex enough.

Seems like I misunderstood. Please add this information.

Linking the documentation that can be found via simple google queries is not providing any benefit, else you would add c tutorials to every c header, or cmake documentation to every cmakelists file.

No, not every file, only at the entry points. I added them now for cmake 9a7dcbe

you would add c tutorials

There is a huge difference about how well known C and groovy are. What applies to groovy might not apply to C. CMake is similar, though.

I would prefer to stop having those documentation discussions till I have finished the accompanying doc/*.md file.

Ok. But please makes notes of these discussion because the next time I review I will most likely overlook things. And newcomers (speak everyone from the current dev team because afaik nobody else knows these DSLs) will be happy for every pointer that is necessary for them.

markus2330 · 2018-05-12T16:11:11Z

scripts/jenkins/Jenkinsfile

+                    [$class: 'RelativeTargetDirectory',
+                     relativeTargetDir: target_dir]
+                ],
+                userRemoteConfigs: scm.userRemoteConfigs


again, what does this do?

markus2330 · 2018-05-12T16:11:58Z

scripts/jenkins/Jenkinsfile

+                sh "git merge --no-ff -m 'merge $VERSION' temp"
+
+                sh "dch -l '.$BUILD_NUMBER' 'auto build'"
+                sh "git commit -am 'auto build $VERSION'"


does this work? In shell a single tick would take $VERSION literally.

sure. It gets expanded before it is passed to the shell by groovy.

markus2330 · 2018-05-12T16:13:09Z

scripts/jenkins/Jenkinsfile

@@ -610,13 +664,13 @@ def checksum(file) {

 /* Generate a Stage


Why is it called maybe?

because it maybe executed. depending on the parameters passed.

I am referring that this should be documented. And "Generate a Stage" is misleading if it maybe generates a stage.

it is always generating a stage, the stage might just be skipped

markus2330 · 2018-05-12T16:13:35Z

scripts/jenkins/Jenkinsfile

                            checkout scm
-                            docker.image(imageFullName(image)).inside { cl() }
+                            docker.image(imageFullName(image))
+                                  .inside("-v ${env.HOME}/git_mirrors:/home/jenkins/git_mirrors") { cl() }


Please document this line.

not sure what you want documented there. how .inside works is documented in the docker dsl plugin. what the -v switch does to docker run is documented in the docker docs.
and the rest is groovy syntax?

Yes, exactly. There are so many DSLs involved that you cannot expect people to know what this does. Please translate it into English.

I am also curious what twas wrong with the previous statement.

And please link to all these documentations in the very beginning of the file.

there was nothing wrong with the old statement, it is just that we now want the mirror repositories available inside docker containers because we manipulate them for the releases (and we do not want full checkouts).

sanssecours · 2018-05-12T17:17:43Z

scripts/jenkins/Jenkinsfile

+ * General Information about Jenkinsfiles can be found at
+ * https://jenkins.io/doc/book/pipeline/jenkinsfile/.
+ *
+ * A Snipped generators is available to the public at


Typo: Snipped → Snippet

and singular "generator"?

markus2330 · 2018-05-12T19:40:34Z

@ingwinlu What happened with the jenkinsfile buildjob? "Build failed. Might be due to partial build. Please check status in jenkins". In Jenkins it is yellow.

ingwinlu · 2018-05-12T19:45:13Z

Either due to me trying to get the jenkinsfile to build for pushes to master as well (ghpbr apparently has some problems there). Or the homepage build jobs lags out the master so hard that it can not properly run build jobs.

ingwinlu · 2018-05-12T19:45:29Z

jenkins build jenkinsfile please

markus2330 · 2018-05-12T19:47:30Z

Ok, sorry. I needed a testrun for the homepage (it would be a release stopper, too. But everything worked!)

ingwinlu · 2018-05-12T19:51:20Z

I have the dockerfiles for the front + backend ready to go just waiting to get this merged first

markus2330 · 2018-05-12T20:26:38Z

Thank you, great job!

markus2330 · 2018-05-12T20:31:42Z

Can we disable https://build.libelektra.org/jenkins/job/elektra-git-buildpackage-stretch/ now?

ingwinlu added work in progress build labels May 8, 2018

ingwinlu self-assigned this May 8, 2018

ingwinlu force-pushed the add_jenkinsfile_package branch from 307d8b2 to 371f3d4 Compare May 8, 2018 14:04

ingwinlu force-pushed the add_jenkinsfile_package branch from 59bd707 to 88df6ca Compare May 9, 2018 14:43

ingwinlu force-pushed the add_jenkinsfile_package branch from 8bb8973 to 7d3428a Compare May 10, 2018 05:54

ingwinlu mentioned this pull request May 10, 2018

travis build jobs fail #1968

Closed

ingwinlu force-pushed the add_jenkinsfile_package branch from 7d3428a to 1212c95 Compare May 10, 2018 06:52

ingwinlu added 4 commits May 12, 2018 16:23

build: add new TODO

52437d4

build: maybeRegexStage -> maybeStage

d83666c

build: fix spelling

760c6a4

build: link git_mirrors into docker container

32a9916

ingwinlu force-pushed the add_jenkinsfile_package branch from 7fcc7d3 to cbd59f1 Compare May 12, 2018 14:23

build: implement debian stretch package build

5727649

ingwinlu force-pushed the add_jenkinsfile_package branch from cbd59f1 to 5727649 Compare May 12, 2018 15:09

build: always build packages

33ec85b

ingwinlu added ready to merge and removed work in progress labels May 12, 2018

ingwinlu added 2 commits May 12, 2018 17:56

docs: Add news sections for debian packages and CI

44841a0

docs: Add section for stretch repository

02d0f61

ingwinlu requested a review from markus2330 May 12, 2018 16:01

markus2330 requested changes May 12, 2018

View reviewed changes

build: expand header documentation

4feace3

sanssecours reviewed May 12, 2018

View reviewed changes

ingwinlu added 2 commits May 12, 2018 19:20

fixup: typo

9efc703

fixup: typo

1645f54

markus2330 approved these changes May 12, 2018

View reviewed changes

markus2330 merged commit bc3b7b5 into ElektraInitiative:master May 12, 2018

ingwinlu deleted the add_jenkinsfile_package branch May 13, 2018 04:23

		@@ -610,13 +664,13 @@ def checksum(file) {

		/* Generate a Stage

Add package builds #1965

Add package builds #1965

Conversation

ingwinlu commented May 8, 2018 • edited Loading

Purpose

Checklist

ingwinlu commented May 9, 2018

ingwinlu commented May 9, 2018

ingwinlu commented May 9, 2018

ingwinlu commented May 9, 2018

petermax2 commented May 9, 2018

ingwinlu commented May 9, 2018

ingwinlu commented May 9, 2018

petermax2 commented May 9, 2018

ingwinlu commented May 9, 2018

petermax2 commented May 9, 2018

ingwinlu commented May 9, 2018

ingwinlu commented May 9, 2018

ingwinlu commented May 9, 2018

markus2330 commented May 9, 2018

markus2330 commented May 10, 2018

markus2330 commented May 10, 2018 • edited Loading

ingwinlu commented May 10, 2018 • edited Loading

markus2330 commented May 10, 2018 • edited Loading

ingwinlu commented May 10, 2018

ingwinlu commented May 10, 2018

markus2330 commented May 10, 2018

ingwinlu commented May 10, 2018

ingwinlu commented May 12, 2018

markus2330 commented May 12, 2018

markus2330 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ingwinlu May 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markus2330 commented May 12, 2018

ingwinlu commented May 12, 2018

ingwinlu commented May 12, 2018

markus2330 commented May 12, 2018

ingwinlu commented May 12, 2018

markus2330 commented May 12, 2018

markus2330 commented May 12, 2018

ingwinlu commented May 8, 2018 •

edited

Loading

markus2330 commented May 10, 2018 •

edited

Loading

ingwinlu commented May 10, 2018 •

edited

Loading

markus2330 commented May 10, 2018 •

edited

Loading

ingwinlu May 12, 2018 •

edited

Loading