Initial work

[theoweiss]

I've opened a PR for the initial checkin #1. I want to use this issue to discussing the further development. Here is a to-do list with the next things which have to be done. For sure the list is not complete, just add further to-dos if they are necessary.

Todo list for openhab b5

• test debian packages from gradle build
  • fresh installation
  • upgrades from current cloudbees snapshot
  • upgrades from gradle snapshots
  • for openhab2-offline and openhab2-online builds
• fix bugs in gradle build
• remove deb and apt-repo build from openhab-distro project
• add release task for b5
• add release task for snapshots
• release snapshots to bintray (replacement for apt-repo plugin)
• integrate snapshot build and release into cloudbees
• update projects Readme.md
• update docs
• test test test debian packages

Todo list for openhab 2.0.0

• stabilize deb build
• build and release rpm packages

[BClark09]

Fantastic, you've been busy! Are the changes in the PR ready for an initial test? Am I right in thinking ./gradlew download-openhab2-online-snapshot && ./gradlew distro-openhab2-online-snapshot is the correct order to get a .deb file?

update docs

From a user's point of view, what will be different to how installation usually works which requires the documentation to change?

[theoweiss]

I've just updated the README: ./gradlew buildSnapshot is your friend ;-).

From a user's point of view, what will be different to how installation usually works which requires the documentation to change?

The snapshot installation will probably change because I'll try to publish the snapshots using the bintray apt-repo.

[theoweiss]

@BClark09 I've merged the initial checkin. It would be great if you could do some testing.

[BClark09]

Sure thing! I've got a spare Debian laptop ready for testing, but will get round to doing it as soon as I can.

[ThomDietrich]

Hey @theoweiss these points are of concern to me:

• release snapshots to bintray (replacement for apt-repo plugin)
• integrate snapshot build and release into cloudbees
• update docs

Will identical snapshots be available on both bintray and cloudbee? If so, how long? If I understand you correctly, the idea is to provide the latest snapshot on both to ease the transition for users?

What I take from that: I'll have to switch the repository link to bintray in both the linux article and the openHABian scripts!?

[BClark09]

Testing went well:

1. I used ./gradlew buildSnapshot to build both the OFFLINE and ONLINE .deb packages of snapshot #624. No apparent error in building.

2. I installed openHAB on a fresh debian laptop using the offline .deb package first, no errors were logged in dpkg, or the service daemon on first run. openHAB configured itself successfully without error.

3. I installed the NTP, Yahoo Weather, and Exec1.x bindings, added some sample things and items and used mapDB persistence to restoreOnStartup. Worked flawlessly.

4. Forced an upgrade using the same deb package, after "upgrade" bindings reinstalled themselves and persisted items displayed the expected state.

5. Repeated Steps 1-4 with the online package.

6. Repeated Steps 1-5 with the cloudbees packages, making sure to upgrade to the packages built earlier. Once again, nothing unexpected popped up during testing. Only comment here is that builds from gradle have -1 appended to the timestamp.

I'm going to build #625 tomorrow so that I can test the above with specifically different snapshot files but I don't expect any differences in behaviour so from what I can see so far:

Fresh installation works.
Cloudbees snapshot > gradle snapshot upgrade works.
gradle snapshot > gradle snapshot upgrade works.

[theoweiss]

@BClark09 you(!) have been busy, thanks a lot. Sounds very good.
We also have to check the issues for the deb packaging opened in the openhab-distro project.
I found :
openhab/openhab-distro#328

[theoweiss]

@ThomDietrich

Will identical snapshots be available on both bintray and cloudbee? If so, how long? If I understand you correctly, the idea is to provide the latest snapshot on both to ease the transition for users?
What I take from that: I'll have to switch the repository link to bintray in both the linux article and the openHABian scripts!?

I hope to get the bintray deployment ready before the b5 release. If I get this working the snapshots will be available for a short time on both, bintray and cloudbees to ease the transition. Together with the b5 release we will remove the snapshot build from the openhab-distro project. Hopefully. I've read the available docs how to release from gradle to bintray but found no docs or examples how to handle debian repos. As you can see there is some uncertainty if all these things are in place until the b5 release.

[BClark09]

@theoweiss

First major problem I'm coming up with. It doesn't look like there's any handling of configuration files even though I see they're set in gradle. For example my changes to /etc/openhab2/html/index.html are lost on an upgrade when using dpkg.

[BClark09]

It looks like this issue is related to: nebula-plugins/gradle-ospackage-plugin#56

EDIT 1: Adding the line "configurationFile("/etc/openhab2/html/index.html")" in ospackage {.. stops replacing the html file, but obviously this is going to be a problem for us with the number of configuration files we have. Wildcards did not work.

EDIT 2: Adding "configurationFile(details.path)" in eachFile might work, will test and submit a PR if true.

EDIT 3: Seems good, submitted PR #3

[BClark09]

Alright! So Debian looks good to go,

We also have to check the issues for the deb packaging opened in the openhab-distro project.
I found :
openhab/openhab-distro#328

The build doesn't create a /var/run/openhab.pid so I'm guessing this is Ubuntu specific, I'll install Ubuntu on a VM and get back to you on this. openhab/openhab-distro#327 is also relevant.

[ThomDietrich]

I believe the pid file problem is related to the systemd service file openhab2.service.
http://www.dsm.fordham.edu/cgi-bin/man-cgi.pl?topic=systemd.service -> PIDFile and GuessMainPID
These options are not in the files, which may be part of the problem and part of the solution...

[BClark09]

Oh right, fair enough thanks. Then it'll be a mixture of that assigning start.sh as it's main program and init.d/openhab2 using start-stop-daemon and --exec

[BClark09]

Whilst looking into the above, I notice that /etc/init.d/openhab2 isn't included in the gradle build. I've got a branch I'm working on to address these two issues.

[BClark09]

@theoweiss I've tested this on multiple Linux distributions now (init.d systems need PR #6 included) across multiple updated and I'm pretty confident in the Initial work. What are the next steps here? A release task for bintray?

[theoweiss]

@BClark09 I've just merged your PR, thanks again.

Yes we will need two release tasks for bintray: one for the snapshots and one for the releases. As soon as the linux-pgk build is integrated into the cloudbees build the snapshot build must be triggered automatically each time when the openhab-distro build finished successfully.

I think we should try to use the bintray gradle plugin (https://github.com/bintray/gradle-bintray-plugin). Unfortunately I found no examples for publishing deb files. Here you will find the script that I've used for publishing openHAB2 beta releases: https://github.com/openhab/openhab-linuxpkg/blob/master/resources/bin/bintray-upload-debs.sh . This must be transcribed to gradle-bintray and for snapshot releases.

Furthermore I've started to port the apt-repo maven plugin to gradle (https://github.com/theoweiss/apt-repo). Until now we used it for publishing the snapshot debs. You will find my work here https://github.com/theoweiss/apt-repo-lib/tree/initialDev and here https://github.com/theoweiss/apt-repo-gradle. The idea is to have this available in case we can't get the gradle-bintray plugin running.

If you don't mind it would be great if you could care for the gradle-bintray task and I'll try to finish the apt-repo gradle plugin as a fallback solution?

[BClark09]

Not sure when I'll be able to look at it, but I did previously have a look and think that this will get us pretty close:

bintray {
  user = System.env.BINTRAY_USER
  key = System.env.BINTRAY_API_KEY

  dryRun = false
  publish = true    
  override = true

  filesSpec {
    from ('build/distributions') { include '*.deb' }
       into '/'
  }

  pkg {
    repo = 'apt-repo2'
    name = 'openhab2'
    userOrg = 'openhab'
    desc = 'Packages for openHAB 2.x'
    websiteUrl = 'http://www.openhab.org/'
    issueTrackerUrl = 'https://github.com/openhab/openhab-linuxpkg/issues'
    licenses = ['EPL-1.0']
    vcsUrl = 'https://github.com/openhab/openhab-linuxpkg'

    attributes = ['deb_distribution': 'development', 'deb_component': 'main', 'deb_architecture': 'all']
   }
}

Although I'm a little unsure how it will actually handle the different deb files. It'll probably need a version parameter with a variable name parameter that can be set to [online|offline]-[snapshot|beta] but not sure how best to do that yet.

[BClark09]

Hi @theoweiss, I've edited the above, but am very unsure on if this would generate anything useful. Would you mind testing this? (Probably best into the test name instead of openhab2).

.gradlew bintrayUpload 

Initial thoughts:

• the .deb packages may need to be separated into four different folders in the build/distributions folder to distinguish between versions.
• If not then each package needs a specific configuration.

EDIT: Nope, got a trial account and can't get this working at all. Any iteration of it and I'm stuck with the error:

message: Unable to upload files: Debian distribution. component and architecture must be specified.

As far as I know, the attributes should be set according to https://bintray.com/docs/api/

[theoweiss]

I had a quick look into the sources of gradle-bintray-plugin. As far as I understand the attributes will be used to create the request body: https://github.com/bintray/gradle-bintray-plugin/blob/58a2c1813ebadc8ad3242a6ca73a4fa1471823c5/src/main/groovy/com/jfrog/bintray/gradle/BintrayUploadTask.groovy#L231 . But the documentation indicates that debian information must be specified using HTTP matrix parameters or HTTP request headers (https://bintray.com/docs/api/#_debian_upload). May be using the gradle-bintray-plugin is a dead end approach :-(

[theoweiss]

May be there are two other options:

1. gradle Exec tasks https://docs.gradle.org/current/dsl/org.gradle.api.tasks.Exec.html and make a curl call similar to that one in the upload script https://github.com/openhab/openhab-linuxpkg/blob/master/resources/bin/bintray-upload-debs.sh .

2. the gradle rest plugin https://github.com/noamt/rest-gradle-plugin

We would also need a task to add a new version for each upload.

[BClark09]

I'd also seen the gradle Exec task as this seems to be the one used in most deb builds (for example). I like the idea of using the REST plugin though. I might have the chance to play with both tomorrow!

[BClark09]

@theoweiss , What do you think of https://bintray.com/benclark as a way of managing these packages. Check out the apt-snapshot repo where I've managed to get version uploads working, we may also need to add another task to clear old versions. I can successfully install and upgrade using apt when I add the repo to the sources list too, it now recognises when you're already on a version!

My own branch now does this through gradle as part of the buildSnapshot command, or manually start an upload with upload-openhab2-online-snapshot etc.

[theoweiss]

@BClark09 I'm preparing some feedback until tomorrow.

[BClark09]

@theoweiss, did you have any quick points of feedback? I have time to work on it this weekend.

[theoweiss]

Hi @BClark09,

sorry for the delay!
First of all I think I should explain the apt-repo setup I had in my mind.
I think we should have only one repo, which is the still existing apt-repo2 for openHAB2 (https://bintray.com/openhab/apt-repo2/openhab2).
All released packages should be available from this repo. Released packages are snapshots, betas and stable versions.
Having more than one repo until now is just a workaround for technical reasons.

• The snapshots should have deb_distribution=unstable, betas have deb_distribution=testing and releases have deb_distribution=stable.
  Releases have a second deb_distribution=$version.
• Furthermore the signing support is missing X-GPG-PASSPHRASE header (just look at bintray-upload-debs.sh).
• The debian package version must not be hardcoded (def filename="${packageName}_${version}-1_all.deb").
• deb_architecture=all is not sufficient use: deb_architecture=all,armhf,armel,mips,arm64
• I think the bintray version must be created?
• calc_metadata is missing
• use expressions like this for variables (not only System.env)

user = project.hasProperty('bintrayUser') ? project.property('bintrayUser') : System.getenv('BINTRAY_USER')

Regards,
Theo

[BClark09]

Thanks @theoweiss I've started updating it now, does https://bintray.com/benclark/apt-repo2 look a little more like what you were expecting (distribution needs to be changed from "null" to "testing")? Or would you rather everything be placed inside an openHAB2 package?

[ThomDietrich]

I'm not sure about the last part (compare Firefox or other add-ons based software) but I get your argument "recommendation per packaging/installation option". I thought with the online version mostly stable this would be a good time to think about abandoning the offline version for openHAB 2.0 GA all together (or provide it as a less favorable version) but I do not want to argue about this in length here. Just wanted to point it out as this Issue / repo would be influenced by such a decision. ;)

[theoweiss]

@kaikreuzer it seems like OJO ( https://oss.jfrog.org) isn't a valid option because https://www.jfrog.com/confluence/display/RTF/Deploying+Snapshots+to+oss.jfrog.org states
"hosting your maven-compatible build snapshots". Debian repos which are not maven-compatible don't seem to be supported :-(
Some time ago you suggested to use bintray for the nightly builds because the version string doesn't contain "SNAPSHOT" anymore: openhab/openhab-distro#270 (comment)
From a technical perspective this would work.

[theoweiss]

As a fallback we could also stick with cloudbees and use the gradle port of the maven apt-repo plugin. It would be still better than the current solution because we could solve the lastSuccessfulBuild/WS problem which is also mentioned here openhab/openhab-distro#270 (comment)

[kaikreuzer]

it seems like OJO ( https://oss.jfrog.org) isn't a valid option

Hm, good point. I didn't think about the fact that we are not talking about Maven artifacts.

Some time ago you suggested to use bintray for the nightly builds because the version string doesn't contain "SNAPSHOT" anymore

Let's go for this. After all, Debian packages do not know the notion of a "snapshot", so it should be fine to upload all of them to Bintray.

[BClark09]

Alright! That should be fine but I think we will need a task to remove old versions.

[theoweiss]

Yes automatically removing old versions would be right solution, but for now I think it would be OK to remove them by hand using the web ui of bintray.

[BClark09]

To achieve that I'd assume it would suffice to run a script to perform the following steps:

1. Get a list of versions currently in the repo by parsing the response of:
   CURL -X GET "-u${BINTRAY_USER}:${BINTRAY_KEY}" "https://api.bintray.com/packages/${BINTRAY_ORG}/${BINTRAY_REPO}/${BINTRAY_PACKAGE}/
2. List the bottom [total number of versions] - [desired number of versions] that do not match "2.0.0~b*"
3. For each match run:
   CURL -X DELETE "-u${BINTRAY_USER}:${BINTRAY_KEY}" "https://api.bintray.com/packages/${BINTRAY_ORG}/versions/${BINTRAY_REPO}/${BINTRAY_PACKAGE}/${versionToDelete}

[theoweiss]

These curl calls from gradle are a bit ugly especially the error handling and even more if some complex logic is needed. Having these calls in an "external" bash script is a way to go, but I would prefer to have these things in the gradle build script. I want to give this rest-gradle-plugin a try and if I don't succeed we could take the bash script approach. What do you think?

[ThomDietrich]

Everything seems to work flawlessly till now. Great ;) Would it be possible to release a stable dummy package (e.g. the beta5 re-branded, needs to be "newer") so I can already test switching around between branches?

[theoweiss]

For testing the best way to go is:

• create your own Bintray account
• create an empty bintray Debian repository (maybe called "apt-repo2")
• within this repo create a package (maybe called "openhab2")
• create a GPG key and upload it to Bintray
• clone openhab-linuxpkg
• adapt openhab-linuxpkg to your needs (maybe with assistance from maintainers ;-) )
• release openHAB to your private repo

  ./gradlew buildSnapshot -PbintrayOrg=<thomDietrichAccount> -PbintrayRepo=apt-repo2 -PbintrayPackage=openhab2 -PbintrayUser=<thomDietrichUser> -PbintrayKey=<thomDietrichsApiKey> -PbintrayGpg=<thomDietrichsGPGKeyPassword>
  

• use this repo for your tests

[ThomDietrich]

Thanks for the instructions Theo but I do not intent to get active in the repository development. I've already too much on my plate, not only related to openHAB. I thought this would be a no-brainer for you guys and it wouldn't harm anyone. But it's okay, I'll just wait for the stable to be released.

[BClark09]

I want to give this rest-gradle-plugin a try and if I don't succeed we could take the bash script approach. What do you think?

I would also like to have everything running from within the gradle build. I particularly like the responseHandler.

A quick example:

task "upload-${dist}"(type: org._10ne.gradle.rest.RestTask) {

  def fileName = "${gPackageName}_${gVersion}-${gRelease}_${OSPACKAGE_ARCH}.deb"

  httpMethod         = 'PUT'
  uri                = 'https://api.bintray.com/content/${BINTRAY_ORG}/${BINTRAY_REPO}/${BINTRAY_PACKAGE}/${gVersion}/pool/main/${gVersion}/${fileName}'
  username           = ${BINTRAY_USER}
  password           = ${BINTRAY_KEY}
  requestBody        = new File("build/distributions/${fileName}")
  requestContentType = groovyx.net.http.ContentType.BINARY
  contentType        = groovyx.net.http.ContentType.JSON
  
  requestHeaders     = [X-GPG-PASSPHRASE:              ${BINTRAY_GPG},
                        X-Bintray-Debian-Distribution: ${gDebDist},
                        X-Bintray-Debian-Component:    'main',
                        X-Bintray-Debian-Architecture: ${BINTRAY_ARCH},
                        X-Bintray-Publish:             1]

  responseHandler = {
    assert it.message == 'success'
  }
}

But how to add the file to the body? I couldn't be sure.

[theoweiss]

You could try:

 requestBody = new File(fileName)

for the body.

And:

contentType = BINARY

I googled for "groovy RESTClient PUT file" because noamt uses "groovyx.net.http.RESTClient". Just a wild guess.

[BClark09]

Yes, you're absolutely right! I've edited the above to contain a full example. Unfortunately, my trial expired so am not able to test what it does to Bintray.

[theoweiss]

Which trial? You don't have to pay for Bintray "Community - Open Source".

[BClark09]

Whoops, my mistake. Created an account.

I've been struggling with this for a while, this is what I have so far (including a little tidy up): https://github.com/BClark09/openhab-linuxpkg/blob/add-rest-plugin/build.gradle

The request body needs to be converted to bytes to be used in requestBody (There's an issue for support of File), Since task configuration is done at the evaluation stage in gradle, I can't for the life of me work out how to use .bytes...

[theoweiss]

IMHO using .bytes may be a bad idea because I think the whole contents of the deb file would be stored in the requestBody variable, which means in RAM. Could you point me to the issue you mentioned?

[BClark09]

Certainly - noamt/rest-gradle-plugin#26

[BClark09]

I've come to the conclusion that this won't accept a file stream for a long time. I had a play around with the underlining HTTP and REST clients and the resulting task was:

import groovyx.net.http.HTTPBuilder
import org.apache.http.entity.FileEntity
import org.apache.http.HttpHeaders

import static groovyx.net.http.ContentType.JSON
import static groovyx.net.http.Method.*
import static java.io.File.separatorChar
import static java.lang.System.err
import static java.lang.System.exit

buildscript {
    repositories {
        mavenCentral()
        jcenter()
    }   
    dependencies {
        classpath 'org.codehaus.groovy.modules.http-builder:http-builder:0.5.2'
    }   
}

task "upload-${dist}"(dependsOn: "build-${dist}") {

    def fileName = "${gPackageName}_${gVersion}-${gRelease}_${OSPACKAGE_ARCH}.deb"
    def fileToUpload = new File("build/distributions/${fileName}")

    HTTPBuilder bintray = new HTTPBuilder('https://api.bintray.com', JSON)
    bintray.parser.'application/json' = { '' }
    bintray.auth.basic "$BINTRAY_USER", "$BINTRAY_KEY"

    def requestHeaders = [
        'X-GPG-PASSPHRASE':              "${BINTRAY_GPG}",
        'X-Bintray-Debian-Distribution': "${gDebDist}",
        'X-Bintray-Debian-Component':    'main',
        'X-Bintray-Debian-Architecture': "${BINTRAY_ARCH}",
        'X-Bintray-Publish':             '1'
     ]

    doLast{
        bintray.request(PUT, JSON) {
            uri.path = "/content/${BINTRAY_ORG}/${BINTRAY_REPO}/${BINTRAY_PACKAGE}/${gVersion}/pool/main/${gVersion}/${fileName}"
            headers[HttpHeaders.AUTHORIZATION] = 'Basic ' + ("$BINTRAY_USER:$BINTRAY_KEY".toString().bytes.encodeBase64())
            requestHeaders.each { key, value ->
                headers."${key}" = "${value}"
            }
            contentType = groovyx.net.http.ContentType.JSON
            requestContentType = groovyx.net.http.ContentType.BINARY
            body = fileToUpload.bytes
            response.success = {
                println 'Done.'
            }
            response.'409' = {
                err.println "WARNING: File already uploaded, new file ignored."
            }
            response.failure = {
                err.println "Failed to upload file: $it.statusLine.reasonPhrase."
                exit it.status
            }
        }    
    }
}

I think i'd prefer to use the curl executable task within gradle considering the memory usage in this task and the extra faff that needs to be performed to get it to function correctly. I'd also prefer to use the existing task over a bash script, wdyt?

[theoweiss]

Your right. I think the bug is in groovyx.net.http library and we would have to use the apache hc lib to get a proper solution. For now we should stick with the current curl way. What we still need is a task for removing outdated versions, where we don't have the body problem. We could use groovyx.net.http for this or a well crafted bash script.

But on the top of the todo list is the adaption of the gradle script to release 2.0.0.RC1 see here: https://community.openhab.org/t/upcoming-releases/18503/17

[theoweiss]

@BClark09 I've started to work on the adaptions needed for releasing RC1.

[BClark09]

Right, is RC1 going in Stable or Testing? Apologies, I've not had any notification on that message chain until today so was not privy to the release dates.

I was in the process of cleaning up the build.gradle (see branch here), but I can merge this after RC1 and also work on a way of cleaning up the old snapshot versions. I think we should be fine using the REST plugin here.

@theoweiss Would you be able to add 'userdata/etc/branding-ssh.properties' to the list of files that we dis-include from the conffile list? This came up as a problem for some people using apt.

[kaikreuzer]

Right, is RC1 going in Stable or Testing?

I'd say Testing, just like the beta releases.

[theoweiss]

@BClark09 I'm just making the RC1 release. I've merged the necessary changes to this project. I'll do a test installation. The distribution name is "testing". If you have time tomorrow could you also make some basic tests?

[theoweiss]

@theoweiss Would you be able to add 'userdata/etc/branding-ssh.properties' to the list of files that we dis-include from the conffile list? This came up as a problem for some people using apt.

@BClark09 now I've not been notified about your comment. We could add it to the next release or even increment deb "release" number and make another deb release for RC1.

I'm sure you were not aware of this community thread (https://community.openhab.org/t/upcoming-releases/18503/17m) because you haven't been in the maintainer list at the time this thread was opened.

With a next step we have to add a parameterised task for releasing 2.x.x., shouldn't be to hard to implement.

[BClark09]

That's fine, upgrade on a debian 8 machine is working fine. Also tested a fresh install on Ubuntu 14. The packages are doing their job fine and addons seems to be working fine also.

However, @kaikreuzer, after the install of legacy, 1.x bindings show up on paper-ui, but when trying to install them they stall and on refresh they no longer show, even on a restart, this behaviour is seen even on a fresh install, can you and @theoweiss can confirm this behaviour?

I'm sure you were not aware of this community thread (https://community.openhab.org/t/upcoming-releases/18503/17m) because you haven't been in the maintainer list at the time this thread was opened.

Initially it read "Oops! That page doesn’t exist or is private." but I can read this on refresh. Thanks.

[kaikreuzer]

can you and @theoweiss can confirm this behaviour?

I tried the "normal" install and all seems to work well:

08:02:06.310 [INFO ] [smarthome.event.ExtensionEvent      ] - Extension 'binding-astro1' has been installed.
08:02:20.295 [INFO ] [b.core.service.AbstractActiveService] - Exec Refresh Service has been started
08:02:20.298 [INFO ] [smarthome.event.ExtensionEvent      ] - Extension 'binding-exec1' has been installed.

So I assume it is either some local problem on your end or some issue with the apt package.

[BClark09]

@kaikreuzer You're right, it was a local problem and works as expected now. But I am still getting some minor behaviour:

When the legacy switch is off in paper UI and then I add the legacy .kar file, (through apt, or normal installation) all the legacy bindings show up only once in the add-ons menu. If you try to install these, they will disappear until you actually turn on the legacy switch, and installation of these bindings works as expected.

[kaikreuzer]

Ok, I can live with that ;-)