Working on Apache PLC4X is a huge amount of work and it’s expensive, as it requires purchasing expensive hard- and software. The industry seems to like using PLC4X and open-source in general, but doesn’t seem to be willing to support the people working on it. So, I will stop providing free community support for PLC4X.
You will be able to get commercial consulting, training and support from me via my website at https://c-ware.de and beyond that I have added a crowd-funding-like system for PLC4X feature development. Feel free to have a look at the currently listed initiatives here: https://c-ware.de/en/cf-listing-page/
This is my final attempt. If this also doesn’t help with getting at least some form of financial attribution for my hard work, I will close down my business and there will be no further form of support from my side.
|
Note
|
Nobody is talking about giving up the project. The Apache PLC4X project is more than just me, even if I’m the one with the most commits, we still have a large number of other contributors. Also will the project continue to provide free community support and I will continue to work on it … just no longer for free. |
I started working on what was to become Apache PLC4X about 10 years ago (way before Industry 4.0 was a thing), but I gave up on it after a few months of working really hard on it. At that time I wanted to provide Software for the automation industry, but my plans were to implement my own proprietary software stack for industrial communication, which only I could use to build custom solutions (This being my USP). However, I noticed quickly that this is a huge undertaking - nothing a single person could possibly implement all on his own. I finally decided to take a full employment position and give up on this idea.
About 5 years later all of this "Industry 4.0" was popping up everywhere. I was seeing a big opportunity for making production more efficient, wasting less resources and even making products live longer.
Unfortunately, I was only seeing super expensive solutions provided by the major automation vendors. The price tags of all of these were usually somewhere around 10k€ per user per year. I knew smaller companies wouldn’t be able to participate in the quest for digitizing their production as the license costs for these products would simply be eating up what the improvements could possibly bring them.
However, I knew open-source had all the tools needed for digitizing automation processes - If there only was a way to access the data on the production machinery.
Industrial automation was and still is using a huge number of specialized protocols for which only for some there were at least a few badly licensed open-source libraries (from a business perspective) or commercial libraries available. The commercial ones having average price-tags of 4k€ per node running a driver. This was when I remembered what I had been working on 5 years before. However, this time I didn’t want to do it all on my own, I wanted to do it the open-source way.
Luckily my former employer: codecentric believed in my vision of enabling companies to digitize their processes using open-source, and I was allowed to work fulltime on Apache PLC4X for about 3,5 years. It was the general idea to earn money with training and consulting in this area, based on the open-source expertise that was available in the company (Industrial connectivity, big-data, machine-learning, augmented-/virtual-reality, digitization in general).
In Christian Glomb’s talk (Siemens) at 2021s EclipseCon, he stated that over 50% of each industrial IoT projects costs were data-integration costs. This huge block is what Apache PLC4X aims at reducing drastically. So generally this hope was justified.
Like with typical Machine-Learning and AI projects, the job of the data-engineer is the least fun part and everyone wants to be a data-scientist, that works with a magically curated data-lake. This is the same with industrial data-integration. Most focus goes onto the projects that allow you to do great things with industrial data, however almost nobody is willing to work on the part of actually getting the data.
Unfortunately, we couldn’t have imagined how hard it is to grab hold in the automation industry market. Every POC we did was a huge success. Here I’m not talking about simply achieving the customer’s goal, it was more: drastically exceeding these expectations: Like being able to save 20.000.000€ in license costs compared to the commercial solution, being 130 times more performant than the commercial solution, or providing extremely secure unlimited real-time data for machine learning where the fastest industrial solution cost 100.000€ and more per machine in license costs and allowed only 10k data-points with a latency of a few minutes. (See: the success-stories post for details)
Believe it, or not. We still failed at getting customers to go beyond our incredibly successful POCs. I’ll write a separate article on this. What we were told off the record was, in every case things didn’t proceed due to politics and not our performance.
To make things worse, due to the NDA’s we had to sign everywhere, we couldn’t even openly talk about our success. Everything in the automation industry is considered top-secret, even simply telling the world that you’re using a given product seems to be impossible. But on the other side, "Who uses your stuff?" was the most important question we were asked during every pitch we did. Answering with: "Quite a lot, but I’m not allowed to tell you who and how" doesn’t really help.
I took every opportunity to talk at conferences, publish articles in magazines or write blog posts. I think in the pre-pandemic times I had an average of 26 talks per year at different national and international conferences and meetups. During pandemic times, the talks were less frequent, but I took every chance at talking at meetups and conferences I wouldn’t have been able to talk at otherwise. IT folks were definitely picking up on the stuff I was talking about.
We still failed miserably: On the one side there’s no point in convincing the IT folks that what you’ve got to offer is awesome, if the OT folks don’t know about it. The OT folks are the ones doing the decisions. Everything in the OT world is pay-to-play. You’ll never get an article posted in an OT magazine, if you don’t pay for it. In the OT world there are also almost no conferences, like in the IT world. Everything is about industrial fairs with extremely expensive booths. The budgets the big players have at their disposal are simply unbelievable. As an open-source project you have no chance of being noticed at all.
Ironically most requests for help on our mailing-list were like: “Thank you for providing this awesome PLC4X. It’s making our lives so much easier. So we’re using it in one of our countries biggest {steel melting plants}/{aerospace companies}/{car manufacturers}/{pharmazeutical companies}/… and have this problem … “ sent from email addresses like: john.doe@aol.com. They all wanted help. Help for free and didn’t even have the decency of mentioning what company they were working for.
So, in 2020 (I know … this was a pretty “sub-optimal” year all together) I had to decide. Either give up on PLC4X or give up my Job at codecentric. I didn’t really have to think and gave up the job and was planning on living from consulting gigs for industrial open-source, which I still strongly believed in. Things didn’t look that bad in Q2 and Q3 of 2020. Luckily in that year, I was also able to get financing from NLNet from European research funds to continue working on PLC4X from March till Oktober 2020.
Unfortunately, Covid didn’t go away and after a few months of positive numbers the next wave hit the world and all companies that were starting to open their purses, instantly closed them again.
Luckily for the both of us, at that time Shaun from Mapped approached me, as he was looking for folks able to help communicate with building automation systems. I was happy to join, however I insisted on being able to continue running my business as a side-thing (as long as it has no impact on my work for Mapped). Because I knew: If there’s no place a company could get commercial support or consulting from, I could probably just bury the project.
So since then, I have been working hard on the parts of PLC4X, which we needed for our work at Mapped. These were mainly porting PLC4X to Go and working on drivers and auto-discovery used in building automation systems, but not really much on things considered baseline project-work.
At Apache, we know community is the hardest but also the most important part. So, after working times, on weekends and even on most of my holidays, I kept on pushing the open-source project, writing reports, doing releases, mentoring people do their first contributions, submitting and performing talks, providing fee community support, addressing security issues, doing normal project housekeeping. In general: Keeping the lights on.
To make things worse, in 2020 another problem came to the pre-existing ones: In Germany a company needs to be profitable. If you run your business at a deficit for too long, you get your business closed, and then you usually even have to pay back a lot of taxes.
As I have been focusing on PLC4X without getting many paid gigs, but I needed to pay a lot for purchasing automation hard- and software, I have been running my (side-) business at a deficit for way too long now. I managed to convince the tax officials to give me a bit more time to become profitable (after all that’s my goal).
I think this year I even managed to at least stay out of the red numbers, solely thanks to the Spanish company Zylk and the only paid gig I did in 2021. They hired me for implementing an open-source Profinet driver as part of the Apache PLC4X project. In addition, I kept investments at an absolute minimum at the same time.
However, at the end of this year, one thing is different: I’m sick of fighting. I’m sick of investing my precious free time till I feel I’m burning up and not really getting anything back. Just as with the folks currently working in healthcare: Even if you think kind words and clapping is enough … believe me: It’s not. I’m sick of how everything in the OT is pay-to-play and especially how everything in the OT is clearly communicating on a post-fact way. While in the past it was considered good to back your claims by facts, now it’s just about spreading what the author wants to be true. The one being loudest, with the biggest marketing budget, is going to be the winner. I’m not willing to play that game.
So if things don’t change in 2022 I have planned to shut down my business myself. Sadly this is also the year this business would have reached its 25-year anniversary.
As a last attempt to keep the lights on, I am switching the mode I am providing support for PLC4X: I am no longer implementing features users might need, I am no longer instantly fixing bugs for free. Especially I will not invest my private money to buy expensive hardware in order to implement or fix stuff I am then giving away for free.
I will continue help others fix and build stuff themselves and if I personally need something for my work at Mapped or just for fun or want to try something out, I’ll continue doing this. However, I will no longer port PLC4X to new languages, port drivers to new languages, implement new drivers or integration modules and most definitely not help companies with free consulting, training and support.
For this, I have always offered commercial support, consulting and training and will continue to do so. Beyond that, I have even set up a new section on my website, in which I list various PLC4X features that have been asked for a lot of times in the past. This list can be extended on request. Each of these tasks has a price-tag assigned to it. Companies liking to see one of these initiatives become a reality are free to back the campaigns in a crowdfunding-like fashion. As soon as the funding goal is reached, I will start working on it, not before. You can also contact me if you want to fund a campaign differently.
If the industry doesn’t support the people behind open-source, I at least will from now on stop blindly supporting them for free.
Even if you might not be able to directly help, you still could help me. You would do me a great favor, if you could share this with people you know who you think should read it.
Thank you for having taken the time to read this.