Migration to a Maven + Bnd based build system #5005

maggu2810 · 2019-02-27T18:21:49Z

We are in the process of changing the build system. We are not accepting new pull requests that are still using the old way from here on.

For further information see: Migration to a Maven + Bnd based build system

maggu2810 · 2019-02-27T18:22:17Z

Migration to a Maven + Bnd based build system

The build system has been migrated to a Mave + Bnd based build system.
We are not accepting new pull requests that are still using the old way from here on.

Background

openHAB is a java project that is build around the OSGi framework. openHAB add-ons are OSGi bundles.

Over the years the bnd software ("bnd is the engine behind a number of popular software development tools that support OSGi") has established itself as the defacto standard for creating OSGi bundles.

The openHAB maintainers have decided to move from the very Eclipse centric buildsystem to maven/bnd. A combination that not only allows to develop bindings seamlessly in other IDEs than Eclipse, but Java beginners and intermediates find a commonly known maven buildsystem.

Advantages of the new buildsystem

Builds faster
Offline mode builds
Less buildsystem file overhead
External dependencies per extension via the standard maven dependency system
Unit tests are now part of the same project (Eclipse allows to create a unit test for a class via GUI now)

How to migrate

The easiest way to migrate is to use the create_openhab_binding_skeleton script in the bundles folder and create the basis for your migrated binding and than copy your binding into the new generated folder:

Generate your binding with create_openhab_binding_skeleton <binding name> "<your full name>" <your github name>. (This script also updates the CODEOWNERS and bom addons pom. So if you have already updated those files you need to clean that up).
Move your source from your src/main to the new binding folder in binding in bundles.
Move your README.md to the new binding folder. If you have any non standard additional files. Move them also (like images used in your readme)

Add maven dependencies to the pom.xml file if your add-on requires dependencies like so:

<dependencies>
  <dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient-osgi</artifactId>
    <version>4.5.2</version>
    <scope>provided</scope>
  </dependency>
</dependencies>

Remove those included jar files. ONLY IF you can't find a maven central bundle you can keep the jar file in the lib/ directory.

Move ESH-INF to src/main/resources/ESH-INF
Move unit tests from a potential separate .test project into src/test.
Move integration tests into a new bundle, see the itests/ directory for examples.

Adapt src/main/feature/feature.xml to add any external dependencies, you must list them here, too. An example:

   <feature name="openhab-binding-lgwebos" description="LG webOS Binding" version="${project.version}">
        ...
       <bundle dependency="true">mvn:org.apache.httpcomponents/httpclient-osgi/4.5.2</bundle>
       <bundle start-level="80">mvn:org.openhab.binding/org.openhab.binding.lgwebos/${project.version}</bundle>
   </feature>

There are two types of external libraries. Those that are ready for OSGi (like the one above) and those that aren't. The latter ones are listed with a wrap prefix keyword like this:

 <feature prerequisite="true">wrap</feature>
 <bundle dependency="true">wrap:mvn:io.netty/netty-common/4.1.34.Final$Bundle-Name=Netty%20Common&amp;Bundle-SymbolicName=io.netty.netty-common&amp;Bundle-Version=4.1.34</bundle>

Please ask for details and review if in doubt.

Examples

for an add-on with lib directory: https://github.com/openhab/openhab2-addons/tree/master/bundles/org.openhab.binding.allplay

How to build on the command line

Enter mvn clean install -DskipChecks in the project directory or itest project directory.

Subsequent calls can include the -o for offline.

For integration tests you might need to run: mvn clean install -DwithResolver.

Skip checks and tests if you just want to test the compilation process and perform the command from within the directory of the bundle that you want to build:

mvn clean install -DskipChecks -DskipTests -o

How to develop in the Eclipse IDE

Install Bndtools in your Eclipse IDE. (Is automatically installed if you have used the Eclipse Installer for openHAB development.)
Checkout the bnd based openHAB demo application: git clone --depth=1 https://github.com/maggu2810/openhab-demo.
Open the directory in Eclipse. Wait for the download and build to finish (about 3-5 minutes).
Checkout openhab2-addons: git clone --depth=1 https://github.com/openhab/openhab2-addons
In Eclipse File, Import, "Existing maven projects": Add your migrated project (not all projects!) to the workspace.

In Eclipse Package Explorer: Search for pom.xml in the demo-app project.

Add your add-on as maven dependency like so:

<project ...>
  ...
  <dependencies>
     <dependency>
         <groupId>org.openhab.addons.bundles</groupId>
         <artifactId>org.openhab.binding.astro</artifactId>
         <version>${project.version}</version>
         <scope>runtime</scope>
     </dependency>
  </dependencies>
</project>

In Eclipse Package Explorer: Search for app.bndrun in the demo-app project. Double click (takes a few seconds).
Add your project to "Run requirements" via drag&drop from the package explorer.

NOTE: If the binding you want to run/debug cannot be found in the Browse Repos list, or the list is empty, then follow these steps to rebuild the list of bindings in the BOM (Bill Of Material) and make your new binding visible from the IDE launch configuration:
- using Git bash console go to git\openhab2-addons\bom\openhab-addons
- recreate the BOM with: mvn -DskipChecks -DskipTests clean install
- go back to Eclipse, close and re-open the app.bndrun file and now you should be able to find the binding in the list
Save and click "Resolve": Execute with "Run OSGi"

The demo application runs a slim set of openHAB core bundles including automations (next gen rules) and PaperUI (http://127.0.0.1:8080/paperui/). The startup should only take about 5 seconds and you are greeted by the openHAB console where you can type in console commands.

wborn · 2019-02-27T18:24:02Z

But it should be simple for Bintray.

It's sure a lot easier to create your own Bintray repo compared to uploading to Maven Central. Isn't it also easier for them being removed one day causing builds to fail? As fallback we'll probably still have such JAR artifacts in stable/milestone/snapshot releases.

But IMHO this would be just as problematic as when someone decides to delete their GitHub repo containing all the source code of a library. Since we don't store such sources or source JARs.

To mitigate these issues we could also setup an openHAB repo proxy to such 3rd party repos.

why are we trying to use local JAR files at all

I'd like to manage them using dependencies in POMs as well. It would be a good improvement to be able to update libraries and all their dependencies by just updating a version number. :-)

maggu2810 · 2019-02-27T18:29:03Z

So, my offer:
You (who) tells me an addon project name that also offers an OSGi based test project.
I assume it should be one with no -- in the best case -- PR opened.
Perhaps for the very beginning it should be one without a 3rd party JAR that is not available on a Maven repo.

I try to start the base work:

split into one Maven + Tycho reactor and one Maven + Bnd reactor
migrate the project to a Maven + Bnd one (as a direct child of bundles)
migrate the OSGi based test project to a Maven + Bnd one (as a direct child of itests)

That's just an offer to help. If another one volunteers to do that job, I am happy, too.

davidgraeff · 2019-02-27T18:34:31Z

astro is a perfect match, isn't it?
I would then migrate MQTT as soon as the remaining PRs are in, to test how well the migration process works.

wborn · 2019-02-27T18:44:23Z

I'm OK with Astro as well. The binding itself doesn't really matter as long as it's a bit representative of most projects. If we run into issues while migrating other add-ons we can always ask for more help. 😉

wborn · 2019-03-19T07:08:16Z

Thanks for adding a migration guide @davidgraeff !
Didn't notice the first post was updated. :-)

J-N-K · 2019-03-19T07:25:38Z

Can we add that to the developer documentation (esp. the Part about eclipse)?

J-N-K · 2019-03-19T07:38:25Z

One thing regarding the miration guide: "remove '/lib'" is a good idea but will not work out-of-the-box for all bindings, since not all libs are availabe on bintray. And not all of those that are available can easily be used. So this should be checked before migration.

davidgraeff · 2019-03-19T07:46:53Z

@J-N-K There was a discussion about this between Markus and Wouter (I think?) and the agreement afaik is to move everything to (if necessary private) bintray/maven repos. For new contributions I also tend to say to either include the source code completely or only allow "established" libraries on maven central.

J-N-K · 2019-03-19T08:00:14Z

The end of this discussion was, that a decision by oh2-maintainers should be made about that. AFAIK this is still pending.

Including the source code maybe difficult. For example the Allplay binding includes a lib (Tchaikovsky) that is build with gradle and includes other libs that are not available on bintray (source is available on GitHub, though). Integrating that into our own build would greatly increase build time.

davidgraeff · 2019-03-19T08:03:17Z

My long time dream for openHab is Gradle as well, but Markus will slowly poke me to death with a spoon if I suggest that :D (But the speed advantage is real)

J-N-K · 2019-03-19T08:06:18Z

What I was trying to say: other project might use other build systems and integrating those with our own is quite some effort and may not be worth the time (compared to building them and uploading a binary to our own 3rd-party-libs repo).

davidgraeff · 2019-03-19T08:09:38Z

What I was trying to say: other project might use other build systems and integrating those with our own is quite some effort and may not be worth the time

The idea is to go to the maintainer of such a library and ask for a maven publish. There is even a maven plugin that uses github pages as a static repository site. We do not depend on their build system.

lolodomo · 2019-03-19T11:54:05Z

I am locked on the last step, that is launching the demo app. When I double-click on the file app.bndrun, it simply opens the file in the Eclipse editor. With what app should I open this file to start the demo app ?

davidgraeff · 2019-03-19T12:05:04Z

it simply opens the file in the Eclipse editor

It's like with pom.xml files: You can edit the source file or have the eclipse UI. The same for bndrun files: The default view might be text for you, but there should be the tab to show the UI (you see the glimpse of this selection in the screenshot above).

If not, your eclipse installation is missing bndtools (which is fatal). Please either manually install them or use the eclipse installer for openhab-core development. Let it do whatever it does, remove everything from the workspace again and then perform the steps from above.

lolodomo · 2019-03-19T12:35:45Z

Bndtools is now (manually) installed and I can see the dialog box with run requirements.
But I can't add my binding from openhab2-addons. In the Browser repos at left, I see apparently only bundles from org.openhab.core.bom.openhab-core-index. No binding here, mainly core stuff.

If I click on the green +, it opens the "Add Bundle Requirement" dialog box but this is empty, I only see at left "Workspace default-ws" with nothing inside.

Same for Run Bundles. Project run path dialog box shows an empty workspace.

If I drag my project from the project explorer into the run requirements, I get this error:
Error adding one or more requirements
Error generating requirement
Unable to derive identity from an object of type Project

lolodomo · 2019-03-19T13:37:53Z

I can go one step later, that is I can now add my binding and start OH. The problem is that it fails to start.

cweitkamp · 2019-03-19T13:39:41Z

Great tutorial. This will help others a lot. I walked through it step by step and want to add the following things:

Remove META-INF and OSGI-INF

We have to remove the build.properties file too.

Move unit tests from a potential separate .test project into src/main/test.

Move unit tests from a potential separate .test project into src/test.

Additional changes need to be done in the files CODEOWNERS, the BOM (bom/openhab-addons/pom.xml, groupId) and features/karaf/openhab-addons/src/main/feature/feature.xml (bundle).

lolodomo · 2019-03-19T13:45:08Z

But I get a 404 error if I try to open the dashboard or Basic UI !
Is the demo app fully prepared/packaged to run UIs with the demo sitemap ?

davidgraeff · 2019-03-19T13:46:28Z

The demo only runs paperUI. You see that in the "Run requirements" section. There is only paperUI referenced. Add whatever you need there ~~(you don't need to adapt pom.xml, because bnd knowns about all openhab-core and openhab-webui bundles)~~

Btw paperUI can be reached directly via http://openhab-ip/paperui without the dashboard (which is deprecated... next to PaperUI but that's another story.)

davidgraeff · 2019-05-05T18:48:28Z

I think this guide is quite universal: http://www.joern-karthaus.de/blog/karaf.html

lolodomo · 2019-05-06T09:47:15Z

I can no more compile a binding in development.

[INFO] Resolving dependencies of MavenProject: org.openhab.binding:org.openhab.binding.rotel:2.5.0-SNAPSHOT @ D:\dev\openhab2\git\openhab2-addons\addons\binding\org.openhab.binding.rotel\pom.xml
[INFO] {osgi.os=linux, osgi.ws=gtk, org.eclipse.update.install.features=true, osgi.arch=x86}
[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: org.openhab.binding.rotel 2.5.0.qualifier
[ERROR]   Missing requirement: org.openhab.core 2.5.0.201905060301 requires 'osgi.service; (objectClass=org.osgi.service.event.EventAdmin)' but it could not be found
[ERROR]   Cannot satisfy dependency: org.openhab.binding.rotel 2.5.0.qualifier depends on: java.package; org.eclipse.smarthome.core.library.types 0.0.0
[ERROR]
[ERROR] See http://wiki.eclipse.org/Tycho/Dependency_Resolution_Troubleshooting for help.
[ERROR] Cannot resolve dependencies of MavenProject: org.openhab.binding:org.openhab.binding.rotel:2.5.0-SNAPSHOT @ D:\dev\openhab2\git\openhab2-addons\addons\binding\org.openhab.binding.rotel\pom.xml: See log for details -> [Help 1]

Is it now impossible to compile with the old mechanism ?

If I move my binding to the new build system, I assume I will not be able to run it in Eclipse ?

davidgraeff · 2019-05-06T09:57:11Z

It will be harder and harder to use the old development workflow as we more and more change all bits to the new system. At the moment for your particular problem you only need to edit your launch configuration and add whatever appears when you enter "EventAdmin" (or "events" not sure).

lolodomo · 2019-05-06T10:11:42Z

I am moving the binding to the new build system !
But now that the old build system is broken, it is more than urgent to have a clear and easy way to develop with the new build system. I need a way to load my binding in Eclipse and run OH with my binding in Eclipse to test it.

BobRak · 2019-05-06T16:49:35Z

I am also now trying to build my Smartthings binding since I am done making the code review changes but I am getting the same error as @lolodomo is getting:

[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: org.openhab.binding.smartthings 2.5.0.qualifier
[ERROR]   Missing requirement: org.openhab.core 2.5.0.201905060301 requires 'osgi.service; (objectClass=org.osgi.service.event.EventAdmin)' but it could not be found
[ERROR]   Cannot satisfy dependency: org.openhab.binding.smartthings 2.5.0.qualifier depends on: java.package; org.eclipse.smarthome.core.events 0.0.0

It seems like the path forward is to convert my binding to use the new build system. Is that the advised solution? If so, do I follow the Setup the Development Environment instructions at https://www.openhab.org/docs/developer/. Or, the instructions above. I'm confused because this issue is closed and the instructions at https://www.openhab.org/docs/developer/ say the instructions will be updated when the the migration is complete.

Questions about Eclipse setup

Do I need to do anything special to be openhab-core included in the Eclipse setup. Under the old smarthome instruction it talked about including the Smarthome Runtime. Is that still necessary.
How do I get the Infrastructure project?

davidgraeff · 2019-05-06T18:01:34Z

The migration is not yet complete, it is only finished for this repository, not for the openhab-distro one.

kaikreuzer · 2019-05-06T20:51:19Z

I've merged openhab/openhab-distro#905 now, so we should be getting closer again to a working IDE - would be great if everyone could test and provide feedback, which issues are still left (NOTE: It might take a few hours before the Eclipse Installer picks up the new setup file).

kaikreuzer · 2019-05-07T20:29:09Z

@openhab/2-x-add-ons-maintainers For all bundles with a lib folder, I am seeing errors like

in the official IDE setup and in consequence, they fail to compile. What's wrong here?

davidgraeff · 2019-05-07T20:34:17Z

Hm. Could be related to my move of the pom.xml file from poms/bndrun/poms.xml to root. I needed to adapt relative paths. Travis and Jenkins were happy though. Try to build with maven on your Mac (which might behave different to Linux?)

kaikreuzer · 2019-05-07T20:43:34Z

Try to build with maven on your Mac

I am trying to get our IDE setup working again for our developers, so the goal here is to have no errors in the IDE. Yes, Maven headless seems to work as Jenkins confirms.

J-N-K · 2019-05-07T20:44:21Z

@kaikreuzer, please Check l. 704 in the base pom. I assume that will fix your problem. Could you also check if l. 662 needs to be adapted, too?

kaikreuzer · 2019-05-07T20:46:07Z

@davidgraeff Having moved the pom sounds like a plausible explanation, though. And it seems to hint that using lib folders isn't a good idea anymore...

kaikreuzer · 2019-05-07T20:50:05Z

@J-N-K Thanks, changing l-704 indeed does the trick!

It would be cool if all of you could also try out the new IDE setup as I am really struggling with it a bit... If we want to have it ready for the community again, we should make sure that it works reliable - I think I have now created PRs to have at least all compilation and buildpath errors resolved, so that's a start.

J-N-K · 2019-05-07T20:50:18Z

I agree we should get rid of the lib folders as soon as possible. We could simply upload all those files to bintray and add them as dependencies.
Most of need wrapping, though. And I didn‘t manage to wrap libs that are used in itests because bnd can‘t do that.

davidgraeff · 2019-05-07T20:54:04Z

#5503 is quite important for that reason. Could you create an openhab team on jcenter and share the credentials on the oh2 maintainer GitHub team, so that we can start to OSGi bundle those libs?

For new bindings with external libs we should push contributors to ask upstream for OSGi bundled variants.

kaikreuzer · 2019-05-07T20:59:41Z

I think I have now created PRs to have at least all compilation and buildpath errors resolved

All, but marytts as it required a Maven build to be executed. We might be able to add that as a setup task in the Oomph file...

kaikreuzer · 2019-05-07T21:02:24Z

Could you create an openhab team on jcenter

You cannot directly upload stuff to jcenter, you upload it to Bintray and we have a Maven repo there: https://bintray.com/openhab/mvn

I can provide access to the addon maintainers, if you provide me your Bintray account name - but let's first clarify on #5503, where to upload it to exactly.

mgeramb · 2019-05-11T16:13:47Z

I have now changed my development environment to the new build system followed the instructions from here https://www.openhab.org/docs/developer/

Everything works so far, but I can not export the binding as plugin.
The dialog is empty:

Any ideas?

maggu2810 · 2019-05-11T16:20:01Z

That dialog is for PDE stuff.
Just use the JAR file in your target directory.

amosk · 2019-05-15T15:34:18Z

Looking for guidance from the OH core devs before making an IDE setup guide PR.

As of 15May19 bndtools does not support latest Eclipse available to download (2019-06) in the Eclipse installer. Latest support is 2019-03. See "Additional Details" in https://marketplace.eclipse.org/content/bndtools

Eclipse version compatibility has been a sore point for new devs for a while.
I recommend changing https://www.openhab.org/docs/developer/#eclipse-ide-setup step #2 to be more specific, be that a specific Eclipse version or specify downloading the latest 'released' version. Knowing that both ways have their drawbacks, what are your thoughts?

maggu2810 · 2019-05-15T15:45:55Z

As of 15May19 bndtools does not support latest Eclipse available to download (2019-06) in the Eclipse installer. Latest support is 2019-03. See "Additional Details" in https://marketplace.eclipse.org/content/bndtools

Is 2019-06 already released? It is not listed in https://projects.eclipse.org/releases
I assume it is not stated on that page because it is not released yet.
I did not realize problems using 2019-06.

Hilbrand · 2019-08-15T20:08:45Z

I've updated the migration steps in the second comment to use the skeleton script.

maggu2810 mentioned this issue Feb 27, 2019

Added maven archetype project to generate binding skeletons openhab/openhab-core#589

Merged

wborn mentioned this issue Mar 2, 2019

add base to migrate to bnd and migrate astro #5018

Merged

wborn assigned davidgraeff, maggu2810 and wborn Mar 2, 2019

wborn added the infrastructure Build system and Karaf related issues and PRs label Mar 10, 2019

wborn mentioned this issue Mar 15, 2019

Add-on documentation missing due to repo layout changes openhab/openhab-docs#918

Closed

davidgraeff pinned this issue Mar 18, 2019

davidgraeff mentioned this issue Mar 18, 2019

[mqtt][WIP] Make Homeassistant discovery work for Tasmota #4927

Closed

wborn mentioned this issue Mar 19, 2019

[buienradar] Buienradar Binding initial contribution #5131

Merged

kaikreuzer mentioned this issue May 7, 2019

fixed issues that led to compilation errors in the IDE #5566

Merged

digitaldan mentioned this issue May 11, 2019

[venstarthermostat] WIP for submission hww3/openhab2-addons#4

Merged

This was referenced Aug 4, 2019

[sinope] Sinopé Binding initial contribution #3099

Merged

[heos] Denon HEOS Binding initial contribution #2886

Merged

kaikreuzer mentioned this issue Aug 15, 2019

[mysensors] MySensors Binding initial contribution #2066

Closed

kaikreuzer mentioned this issue Aug 16, 2019

[folderwatcher] Folder Watcher Binding initial contribution #4940

Closed

jannegpriv mentioned this issue Sep 13, 2019

[verisure] Verisure Binding initial contribution #4789

Merged

pfink mentioned this issue Aug 6, 2020

Maven Unleash Plugin removes explicit POM version on every release #4523

Closed

Migration to a Maven + Bnd based build system #5005

Migration to a Maven + Bnd based build system #5005

Comments

maggu2810 commented Feb 27, 2019 • edited Loading

maggu2810 commented Feb 27, 2019 • edited by cweitkamp Loading

Migration to a Maven + Bnd based build system

Background

Advantages of the new buildsystem

How to migrate

Examples

How to build on the command line

How to develop in the Eclipse IDE

wborn commented Feb 27, 2019

maggu2810 commented Feb 27, 2019 • edited Loading

davidgraeff commented Feb 27, 2019

wborn commented Feb 27, 2019

wborn commented Mar 19, 2019

J-N-K commented Mar 19, 2019

J-N-K commented Mar 19, 2019

davidgraeff commented Mar 19, 2019 • edited Loading

J-N-K commented Mar 19, 2019 • edited Loading

davidgraeff commented Mar 19, 2019

J-N-K commented Mar 19, 2019

davidgraeff commented Mar 19, 2019

lolodomo commented Mar 19, 2019 • edited Loading

davidgraeff commented Mar 19, 2019 • edited Loading

lolodomo commented Mar 19, 2019 • edited Loading

lolodomo commented Mar 19, 2019

cweitkamp commented Mar 19, 2019 • edited Loading

lolodomo commented Mar 19, 2019

davidgraeff commented Mar 19, 2019 • edited Loading

davidgraeff commented May 5, 2019

lolodomo commented May 6, 2019

davidgraeff commented May 6, 2019 • edited Loading

lolodomo commented May 6, 2019

BobRak commented May 6, 2019

davidgraeff commented May 6, 2019

kaikreuzer commented May 6, 2019

kaikreuzer commented May 7, 2019

davidgraeff commented May 7, 2019 • edited Loading

kaikreuzer commented May 7, 2019

J-N-K commented May 7, 2019

kaikreuzer commented May 7, 2019

kaikreuzer commented May 7, 2019

J-N-K commented May 7, 2019

davidgraeff commented May 7, 2019

kaikreuzer commented May 7, 2019

kaikreuzer commented May 7, 2019

mgeramb commented May 11, 2019

maggu2810 commented May 11, 2019

amosk commented May 15, 2019

maggu2810 commented May 15, 2019

Hilbrand commented Aug 15, 2019

maggu2810 commented Feb 27, 2019 •

edited

Loading

maggu2810 commented Feb 27, 2019 •

edited by cweitkamp

Loading

maggu2810 commented Feb 27, 2019 •

edited

Loading

davidgraeff commented Mar 19, 2019 •

edited

Loading

J-N-K commented Mar 19, 2019 •

edited

Loading

lolodomo commented Mar 19, 2019 •

edited

Loading

davidgraeff commented Mar 19, 2019 •

edited

Loading

lolodomo commented Mar 19, 2019 •

edited

Loading

cweitkamp commented Mar 19, 2019 •

edited

Loading

davidgraeff commented Mar 19, 2019 •

edited

Loading

davidgraeff commented May 6, 2019 •

edited

Loading

davidgraeff commented May 7, 2019 •

edited

Loading