Urgent migration needed to a newer Vaadin version

[strailov]

As you may know hawkBit uses Vaadin 8 for its user interface. Vaadin 8 reached its end of life (EOL) at 21/02/2022. Currently hawkBit depends on the latest released under Apache 2.0 license Vaadin 8 version - which is still ok. However, if a security vulnerability of that version is found - there won't be an open source fix. In that case the Vaadin based UI shall be removed from hawkBit (if still using Vaadin 8).
So the hawkBit UI shall urgently be migrated to newer Vaadin versions (e.g. 23/24), which are supported and under open source license.
Are there volunteers to implement that issue ?

[avgustinmm]

Current state

hawkBit UI uses Vaadin as a web UI framework. The currently used version is Vaadin 8 (8.14.3). This version, according Vaadin Roadmap, has no free support since 21st Feb 2022. There are some version release after that date (8.15.0 - 8.16.0) that seem to be Apache 2.0 licensed and probably could be used in hawkbBit. However, since 8.16.0 the license is Commercial Vaadin Developer License 4.0 , so they could not be used in hawkBit.

Problem

At the time being all is OK but if a security vulnerability is discovered in free versions the hawkBit will have to just drop the UI unless it is migrated to newer version of Vaadin.

There is another problem. hawkBit uses, at the moment, Spring Boot 2.7.12. According to Spring Boot EOL Spring boot 2.7 stream will reach end of support 18th Nov 2023. So, up to that date hawkBit shall be migrated to Spring Boot 3.0 / 3.1. Since Vaadin 8 seem to be incompatible with Spring Boot 3 (they added support for Spring Boot 3 in Vaadin 24 (Vaasin 24 pre release) we shall drop UI unless it is migrated to Vaadin 23.

Options

A. Migrate Vaadin to version 24

Since both, migration to Spring Boot 3.0/3.1 and Vaadin 24 are big change and challenge they shall be made as separate as possible. So the proper plan could be:

1. Migrate to Vaadin 23 (supporting Spring Boot 2.7) - huge efforts, community support is needed here.
2. Migrate in parallel to Spring Boot 3.0/3.1 (big efforts expected) and to Vaadin 24 (with Spring Boot 3 support, hopefully insignificant efforts)

All this shall be implemented by the 18th Nov 2023, so the step 1 shall be ready, let say, until 1st Nov 2023.

B. Drop hawkBit UI

If there is no community for Vaadin migration in the needed time frame we'll have to just migrate the hawkBit update server to Spring Boot 3 and drop the hawkBit UI.

It is possible that in some future time someone from the community will return back the Vaadin based UI by migrating to 24+ version or add completely new UI. But for some time, at least, hawkBit will remain without UI.

[siepkes]

I'll just leave my "2 cents" here. It might contain some useful insights.

I think the main "difficultly" is that Vaadin >= 10 (There is no Vaadin 9) is completely different from Vaadin 8. Vaadin moved away from GWT (i.e. from Java to Javascript transpilation). At the time Google created a lot of uncertainty around GWT's future (which has since been resolved). I guess Vaadin got scared and moved away from GWT. This was before it's successor, J2CL was open sourced by Google.

Depending on how Hawkbit uses Vaadin there might be no easy "upgrade" from Vaadin 8. It's almost as if you are migrating to a different framework. If you want something similar to Vaadin 8 (i.e. Java to Javascript transpilation) you are probably better of using something like DominoUI which is based on Google's GWT successor, J2CL. Or something like TeaVM.

[truedat101]

Curious what the direction is? It sounds painful. It looks like there is an industry around handling migrations between Vaadin UI versions. I am new to the HawkBit community but I have years of work in Java and Mobile Java.

• Has the team considered a fork of Vaadin UI at the point of the last OSS release? I assume the Vaadin community was either thrilled or not thrilled about the licensing change. Does an active fork actually exist? https://github.com/vaadin/framework/network ... seems not.
• Has the team looked at writing a migration tool to generate the migration to something else? I note, there exists a 7 to 8 tool. Java is really good for introspection and code generation, it could be possible. My brief search didn't turn up any tools for Vaadin UI 8 to framework ___ .
• I did find someone who went down the rabbit hole of the analysis of options: https://medium.com/@fezu54/vaadin-8-and-vue-js-a-smooth-migration-b0ef248962d and a move to vue.js

This last approach seems pragmatic. As an outside to the group, and someone who does generally hand off the web part to the "web people", the focus and energy around web UI and front end design is in JS and Typescript. Like it or hate it. It's much easier to find a volunteer to carry forward vue.js or similar. I realize it may sound like sacrilege to develop components outside of Java. It looks like there is a way to use spring boot plus vue.js anyways.

Interested in the outcome and plan.

[siepkes]

@truedat101 Interesting read.

I realize it may sound like sacrilege to develop components outside of Java.

I think the main issue with such a thing would be that the Hawkbit community consists mostly of people comfortable with languages which are often used in the back-end. Such as Java, Kotlin, Rust, etc. Meaning people comfortable with for example Typescript and Vue would need to be brought in to the project.

Personally I'm currently working on projects with htmx as an alternative to Vaadin and as a way of not doing a Typescript based SPA. The way htmx works might look weird when you first see it, but once you start using htmx it actually makes a lot of sense. One big advantage of htmx is that the community is very large because it works with any backend language (Java, Python, Ruby, etc.). Especially on the Python side there are a lot of people who would rather use Django + htmx then doing a full Typescript based SPA.

[truedat101]

I'll take a look at htmx. I've seen the pattern of "stackism" in many startups (where the leader thinks everything has to be done in only one language for simplicity or elegance or simply for the reason that they don't like the languages they don't understand deeply). The only problem with this way of thinking is that it limits who can work on the front end if it isn't html5/js/css/etc. Professional designers typically aren't comfortable with Java or other compiled languages. But it totally makes sense from an efficiency thing for a smaller team. My take though is use the best tool for the job. Be impartial about that and you get the best tools in the stack.

Thanks for the explanation. If there was a front end framework that fits well into eclipse, maybe that's the first criteria, and then maybe consider recruiting some support for front end development outside of the Vaadin UI world.

I also didn't note the Spring Boot EOL question, which is another angle to consider.

[avgustinmm]

Hi all,
the current direction on Vaadin 8 UI is clear - discontinuation - as announced here. This was the only option since we have not found any volunteers to do that migration. That UI was part of the last 0.4.1 and then removed from the master code - meaning it won't be part of the next 0.5.0 release.
This issue still stays in case anyone volunteers to migrate. In that case, we might return this UI.
Unfortunately, this leaves the hawkBit without UI. Some developers are interested in writing a UI from scratch using web technologies, but it is still unclear if it could happen.
Was implemented a demo Simple UI with very limited functionality for just "play around" purposes but nothing more.

[truedat101]

Thanks for the update @avgustinmm . I had planned to write a thin (limited feature set) DM/OTA management server in open source for a project. I stumbled across this project in trying to do some due diligence before deciding to build anything. Since this project exists, I would like to use it, however, the lack of a UI going forward is a concern. I need a UI that is ops friendly for customers that will have to maintain devices out in deployment. I can't give anyone a backend DM system without some GUI, although yes, I guess I could give some ops people scripts. Or do the approach of something that is just slurping release artifacts out of Github.

So, if I understand correctly, #1510 and #1554 remove the Vaadin UI. I will consider then putting a UI on this with a framework that I am familiar with as a strawman concept to scope the effort. What would help, do there exist UI blueprints, a UI specification, or some other form of design documents associated with the UI as it existed in releases <= v0.41? I was impressed by the scope of the existing UI in looking through the demo. This isn't a small project. I think taking a design blueprint approach starting from a UI specification is the way to attract web designers in any case. I am assuming the authors of the current UI are still involved?

[avgustinmm]

There is no UI specification I know about. I suppose the old UI could be used as a reference though probably some aspects could have been done better in it.
Unfortunately, the authors of the legacy UI are not involved in the project anymore. This was something that made the legacy UI migration harder for us which contributed to the decision to discontinue the legacy U.

[truedat101]

Thanks @avgustinmm , I'll take a look through the online demo and see if I can roughly map out what's there in terms of scope, how many screens and complex interactions, features packed in each screen (table sorting and search are always a tricky thing to work out).

On the scale front, what is the max number of devices you guys have seen deployed under management through this system? I am thinking ahead on how a UI needs to be loaded up to test the responsiveness of the UI under those conditions.

For my own project, it won't be more than 300 devices initially but will potentially ratchet up a lot after about a year.

[truedat101]

In terms of RFEs for the UI, is there any single laundry list of UI changes or new enhancements that people have been clamoring for?

[truedat101]

For reference, looks like #880 and #980 are the previous issues related to migrating Vaadin UI versions, dating back to 2019, 2020. There might be earlier versions. And there is #955 #1038 , related to Java versions.

[truedat101]

Ok, so without being too presumptuous, I put together a UI Wireframes (just exact screengrabs of the existing Demo UI ... no details of functionality etc, no annotations, no design theme notes, no user role notes etc.).

PDF is here on onedrive: https://1drv.ms/b/s!AuIL2jCvMCqRh-I_V1fw7UsMUxXBuA?e=dxvf0W

I will try to do the simple one, which is a UI for the login. I have no idea about the backend so I will look into that. On my other projects, we have normally got some separation from the backend code and the frontend WEB UI, which creates some challenges but on the other hand, in theory, it can be a completely different type of developer working on stuff. I will take a look at HTMX recommended by @siepkes , and also Vue.js based approach. If that goes smoothly, maybe I can just keep going. Either way, no expectations yet, no intention of a pull request or anything. I can just look at what is the real effort, and see where that fits in to my own goals to get a OTA / DM server spun up for a few other projects I have (in both cases it needs a UI for non-technical humans potentially to manage releases). Rather than writing a new system from scratch which was the original ridiculous goal because I didn't see anything in my first investigations, this would hopefully save me time to use hawkbit master/0.5x or whatever the latest is.

I welcome feedback. I'm not a pure designer by any stretch (I used to be involved in OMA Standards bodies and Sun J2ME), but I will try to work in a design friendly way and by all means, if there are real web designers lurking, please step in and say hi. The complixity of DM concepts is real enough that the web UX shouldn't interfere with rolling out new releases successfully.

[avgustinmm]

Regarding scale - since this is in the IoT domain I'd expect 100k - 1mil - 10mil... For UI I think there shall be no big difference between 100k and 10mil

[diegorondini]

Hi David,

What would help, do there exist UI blueprints, a UI specification, or some other form of design documents associated with the UI as it existed in releases <= v0.41? I was impressed by the scope of the existing UI in looking through the demo. This isn't a small project. I think taking a design blueprint approach starting from a UI specification is the way to attract web designers in any case. I am assuming the authors of the current UI are still involved?

Just to add some context, Bosch initially used the Vaadin UI as the UI in their Bosch IoT Rollouts product (Eclipse hawkBit is currently an upstream of IoT Rollouts). You can see that in their "Management UI (Classic)" documentation:
https://docs.bosch-iot-suite.com/rollouts/Management-UI-%28Classic%29.html

Now they're using their own "Management UI", which I guess better integrates with "Bosch IoT Suite", but is not part of hawkBit:
https://docs.bosch-iot-suite.com/rollouts/Management-UI.html

[siepkes]

@truedat101 FYI: There are lots of screenshots in that documentation like: https://docs.bosch-iot-suite.com/rollouts/Create-a-software-module.html . Might be useful for wireframes?

[truedat101]

@siepkes and @diegorondini awesome information. Thanks. I think this will help me understand concepts around this.

[truedat101]

Regarding scale - since this is in the IoT domain I'd expect 100k - 1mil - 10mil... For UI I think there shall be no big difference between 100k and 10mil

That's a significant size. Have you guys loaded up the system with test data in that scale (in terms of the 0.41 UI)? Makes sense the iot numbers are large. I generally get concerned with UI performance when there are large lists of data items to view or returned in some query, and how effectively that gets chunked up into pages (or not. My use case probably won't exceed 100K so this is good to hear.

[cybrox]

@truedat101 As someone who has used the existing UI a lot, I don't think the large number of devices registered is a big issue for the frontend side.

The current UI uses infinite scrolling (e.g. loading more entries when scrolling down) on all list views. If something similar would be implemented in whatever Framework the new UI would be written (I assume Vue, React, etc. have support/addons for this out of the box), it wouldn't matter if there are 100k, 1M, 10M devices in total, since they are loaded in batches of e.g. 50.

---

@avgustinmm @diegorondini Since you mentioned the Bosch IoT Rollouts product, are there any plans to completely fade out HawkBit over time or is HawkBit development becoming more active because of this and it is intended to keep it as the backend for that product?

We have been using HawkBit for a few of our products and right now stand before the issues that we'd ideally move it to a k8s cluster, which would require the S3 integration to work reliably (which I think it doesn't) and a UI to exist as well. So far, we have been unable to contribute much, since we don't have any Java developers but if a new UI in web technologies is written, that could change.

For me, it's hard to decide if we continue to build on top of HawkBit or roll our own solution in the future because the direction the project is going has been pretty unknown for the last year.

[avgustinmm]

@cybrox, I'm not sure what you mean by "unknown direction for the last year".
Last year, the contribution and activity increased. We have 2 official releases 0.3.0 and 0.4.1 (the previous, 0.2.0, was released 2018!). The project migrated to Spring Boot 2.7 last year and now (in master) is already migrated to Spring Boot 3.2.2 which was a big step and lots of development (started last year). The removal of in-app UI also led to an extension of the REST API in order to allow external an application to do real management.
If you are about the removal of UI - it's a consequence of long time postponing of migration to a new Vaadin version or different web framework in the previous years.
Anyway, regarding hawkBit, in my opinion it's current state is - library / framework that could be used to build a production read software update product. It's not something you could use directly in production. I could not say if it will be in near future.

[cybrox]

@avgustinmm Thanks for the reply! Yes, I am aware - and very thankful to you and all the other contributor - for these improvements!

What I meant is, while I saw activity increase, I still saw a lot of people who were considering using HawkBit but not (yet) actively helping to develop it being confused as to where the software would go, how addons could be used, if they would even still work, etc. The last releases have started clearing this up more and more.

For people using the software in production already, large-scale rewrites are often concerning, even though I don't doubt you guys will do a good job at it and I of course understand decisions such as deprecating the UI that's based on a product that's EOL.

I am looking forward to what's to come in the future. I think maybe more people would keep HawkBit on their radar, if there was a clear statement of the current state and rough roadmap readily accessible (e.g. as part of the README).

[truedat101]

I've just gotten my dev environment staged here in the lab and on the laptop. I think the UI facelift to something like a react or vue opens some new doors, though it may limit ability of the existing Java oriented team to want to contribute on that front. I have been behind the scenes making sure my project is interested in committing to some kind of OTA solution. Writing a tiny DM/OTA server from scratch was the other alternative. I guess if there was infinite time/resources I'd dive into that. After I have this other project set up with a simple OTA service, I will start the experiment of going through the APIs and the UI spec to see how long it takes to get a few screens going. If it works well I will just carry on and see what a first PoC turn into. I will need it for my project. In my case, I need an app to act as proxy to nRF devices to manage the upgrade of the nRF and it's associated connected MCUs (some PIC). So it is a bit of a hybrid and small numbers, but the notion of managing groups is critical for staging rollouts.

[truedat101]

@cybrox what is the nature of the S3 reliability issue?

[avgustinmm]

Since no volunteers for Vaadin 8 UI migration, no such are expected and have already been removed I'd close this issue.
If there is development on new UI (with new approach and technologies) this could be ins scope of different issues

[cybrox]

Thanks!

---

Just for completion @truedat101, it is notorously hard to set up (see this issue) and there are multiple critical issues (namely this and this) that are not currently addressed.

We have a partner company that tried setting it up in a proof-of-concept and decided it was not viable/stable. However, they haven't provided me any more details aside from "it was not possible to get working properly in the allocated time".

There have also been comments here that some internal refactoring might break the S3 extension but I don't remember where, unfortunately.