HIP19: Approval Process For Third-Party Manufacturers

[jamiew]

Authors: @jamiew (jamiedubs), @georgica, @philltran
Initial PR: #86
Start Date: 2020-11-14
Category: Governance (Meta)

Rendered view:
https://github.com/helium/HIP/blob/master/0019-third-party-manufacturers.md

Summary:

This proposal seeks to lay out requirements for new third-party manufacturers to be approved by the community, and a process by which onboarding keys can be issued by Helium, Inc to those manufacturers.

[shawaj]

Would be very interested in this for use with balenaCloud and Pi Supply / Nebra gateways 👍

[PaulVMo]

I would really like to see more detail in the HIP on what minimum security requirements will be. I don't think this goes far enough and could lead to more, easily exploitable hotspots similar to RAK. Put another way, this does not seem to preclude RAKspots which are currently leading to significant growth in gaming. The HIP talks about favoring security but does not actually spell out any requirements as to what that means (e.g., hardware keys, non-removable storage, secure firmware).

Down the line if and when there is better anti-gaming and/or different classes of gateways (e.g., "light gateways") this may be an acceptable set of requirements. However, those things do not current exist. At this point, I would prefer not to approve more manufacturers that are as insecure as the current generation of hotspots. To me, this does not provide enough clarity on those requirement and thus leaves the door open to more untrustworthy hotspots which are detrimental to the growth of the Helium network.

[georgica]

The way i see it is:

1. lock down the firmware as much as you can
2. Better to encrypt the key folder on the SD card (you can't really copy encrypted folders)
3. Make sure the decryption key/passphrase is hard to obtain
4. firmware should authenticate hardware to ensure it is running on specific hardware.

[philltran]

I agree with @georgica. We should only require due diligence in securing the hardware.

[georgica]

to add to the above:

• Root user should not be accessible
• Malicious devices cannot be plugged in, USB or otherwise
• If you do allow HID devices make sure user/root password cannot be reset easily.
  (One example being using a flavor of linux booting into single user mode and resetting root password)

[jamiew]

Baking some of the requirements mentioned here into the HIP in #90 (WIP)

[shawaj]

@ryanteck and I were discussing this earlier today.

Given that an ATECC608B is only $0.65 ish and is presumably already supported in software from the original product would it not, in the long term at least, be a reasonable suggestion to include this secure element chip in all designs?

For the price of a chocolate bar it adds a lot. Even a TPM would be only around $2 but for the application is potentially overkill and would require building support for which would not be trivial.

In theory you could retroactively add these to existing devices although perhaps it would be a logistical nightmare to do so.

The other suggestions by @georgica and others are also definitely worth doing though.

[georgica]

@shawaj Problem with ATECC608B is that it provides "fake" security as you can still "game" with it easily, so not much difference.

[shawaj]

@georgica do you mean that you could move it from one system to another? Or that you could fake the radio traffic?

I thought the idea behind using the atecc on the original design was just to store the swarm keys?

[georgica]

@shawaj i mean you could convert it into a DIY quite easy and turn attach it to virtual cluster

[ryanteck]

@georgica Just looking over some of the points that you're talking about above.

Better to encrypt the key folder on the SD card (you can't really copy encrypted folders) I'm fairly sure this is incorrect, encrypted folders can be easily copied in their encrypted state.

Make sure the decryption key/passphrase is hard to obtain But wouldn't the decryption passphrase have to be known by the owner to essentially unlock the folder for the miner to operate?

Firmware should authenticate hardware to ensure it is running on specific hardware. How extreme do you mean by specific hardware? Do you mean the hardware combination (So likely in our case it'd be RPi Compute Module, Ethernet chip is a certain brand) or per device combination (So then checking the S/erial number and mac address is the same).

Malicious devices cannot be plugged in, USB or otherwise Quite literally to what extent, our current gateway has a USB port to be able let the user connect a USB Wi-Fi adaptor for connectivity. However while we could remove that a malicious party could potentially solder onto the pins where the port was, and even if we remove it entirely they could just solder directly onto the pins of the modules directly.

Overall some of the suggestions of requirements almost would require a purpose built gateway where the device is encased in resin and possibly contain chips that would self destruct if they sense light and other methods. However the issue with this and some of the requests above is that these would make the device irreparable and possibly e-waste if they went faulty which wouldn't be environmentally friendly and potentially against newer regulations coming into force encouraging repair ability.

Essentially the above would also be a concern as if the software recommendation changes to using the gateway-rs method (https://github.com/helium/gateway-rs) then the owner would want to be able to convert their gateway to use this and some of the above security wishes would prevent the owner from doing this. So unless there was guarantees that the software that would run for now is supported for X years then it would be an issue.

Secondly it seems almost as if you want an "unhackable" box, but also from the owner themselves which would require much more significant time and investment to do as originally when I was reading I thought it was securing against the rare chance of somebody having physical access to the miner that's not the owner.

The thing with it being "unhackable" is that nothing really is unhackable and it's just a matter of time before it can be hacked, a favourite example of mine to refer to is games consoles as many of these attempt to be unhackable but always eventually get hacked. (The most notable being the PS3).

Third, essentially I would question why new gateways have to be so much more secure when there's essentially already over 13,000 gateways which are now being deemed unsecure? Small things such as adding a chip to secure keys is quite easy to do but some of the above proposed requests are talking about specific hardware to be designed for new miners.

[ryanteck]

Also two things I've thought of since the above:

This is also assuming that hardware security is the only issue and isn't really accounting in software security. All of the software is open source which is both considered a security benefit (anyone can look at it and improve it) and also a negative (It makes it easier to find flaws in software). There's also consideration of open source licenses as much of it requires it to be shared due to open source licenses.

Also if the above security is introduced for new manufacturers, will existing manufacturers of unsecure hotspots be told they can no longer sell those units?

[georgica]

@ryanteck
Better to encrypt the key folder on the SD card (you can't really copy encrypted folders) I'm fairly sure this is incorrect, encrypted folders can be easily copied in their encrypted state.
Depends which method you employ.

Make sure the decryption key/passphrase is hard to obtain But wouldn't the decryption passphrase have to be known by the owner to essentially unlock the folder for the miner to operate?
No you can do auto decryption on boot Eg I for my firmware i have built and LKM that takes a few identifiers of the board scrambles them together and that is the passphrase, LKM code is heavily obfuscated and it has a few checks that prevent it from being loaded if anything changed.

Firmware should authenticate hardware to ensure it is running on specific hardware. How extreme do you mean by specific hardware? Do you mean the hardware combination (So likely in our case it'd be RPi Compute Module, Ethernet chip is a certain brand) or per device combination (So then checking the S/erial number and mac address is the same).
Make sure you have the hardware you intended before boot, for example SX1302 has a Unique id, RPI has a serial, eth has mac address, I have this check in the kernel directly and that makes it harder for them to be spoofed. In other words make sure the method you are using is not easily spoofable

Malicious devices cannot be plugged in, USB or otherwise Quite literally to what extent, our current gateway has a USB port to be able let the user connect a USB Wi-Fi adaptor for connectivity. However while we could remove that a malicious party could potentially solder onto the pins where the port was, and even if we remove it entirely they could just solder directly onto the pins of the modules directly.
Again you can leave that in there but make sure your Kernel only recognizes wi-fi cards. Or remove it from your PCB design completely....

Overall some of the suggestions of requirements almost would require a purpose built gateway where the device is encased in resin and possibly contain chips that would self destruct if they sense light and other methods. However the issue with this and some of the requests above is that these would make the device irreparable and possibly e-waste if they went faulty which wouldn't be environmentally friendly and potentially against newer regulations coming into force encouraging repair ability.
I am all for mitigating e-waste and i am an advocate of the "right to repair movement" that being said same thing would happen with ECC version if that dies then the whole thing becomes "e-waste" for some people because you cannot put it on the network and the key is lost. Thats why i am proposing a "software method". It's not required for the whole GW to be encased in resin just sensitive parts.

Essentially the above would also be a concern as if the software recommendation changes to using the gateway-rs method (https://github.com/helium/gateway-rs) then the owner would want to be able to convert their gateway to use this and some of the above security wishes would prevent the owner from doing this. So unless there was guarantees that the software that would run for now is supported for X years then it would be an issue.
gateway-rs is a lite version where it allows you to connect any gateway to the network because it does not contain the miner on the gateway it's self. the way i see it if you are using an RPI in there there is nothing preventing you from re-writing the image on the pi and installing the light gateway.

Secondly it seems almost as if you want an "unhackable" box, but also from the owner themselves which would require much more significant time and investment to do as originally when I was reading I thought it was securing against the rare chance of somebody having physical access to the miner that's not the owner.
Noting is unhackable but yes we do want to make it as difficult as possible.

Third, essentially I would question why new gateways have to be so much more secure when there's essentially already over 13,000 gateways which are now being deemed unsecure? Small things such as adding a chip to secure keys is quite easy to do but some of the above proposed requests are talking about specific hardware to be designed for new miners.
I would suggest you study the issues of the network and why we have arrived to the above. While there are now 13k "unsecure gw's" we do not wish to make it worse, otherwise we would of opened DIY and be done with it.
Let's put it this way you can connect 100's of miners to 1 concentrator making a lot of coins and not proving coverage then the network has failed and your coins become worthless. search in discord for Modesto, Port Hurdon and the issue of gaming.

This is also assuming that hardware security is the only issue and isn't really accounting in software security. All of the software is open source which is both considered a security benefit (anyone can look at it and improve it) and also a negative (It makes it easier to find flaws in software). There's also consideration of open source licenses as much of it requires it to be shared due to open source licenses.
While the helium source code is open source your gateway software and methods you use for security does not have to be.

Also if the above security is introduced for new manufacturers, will existing manufacturers of unsecure hotspots be told they can no longer sell those units?
Existing manufacturers are also taking these measures in improving the security in their gateways.

If you would like to discuss more i am available on discord "@georgica"

[ryanteck]

@georgica

As per the encryption, I'm fairly certain any method would be able to get copied. If somebody wants to copy encrypted data they can.

Firmware authenticating hardware: Making sure the serial numbers for each device matches is a not a good idea in my view. This requires much more significant work per unit and allows no repairability if theres any faults.

For USB: Really talking about modifying the linux kernel to not enumerate anything but the allowed devices doesn't seem to be a very good idea again. For an RPi based gateway (Unless it used the CM4) you couldn't remove USB without removing all connectivity methods. But even with this or any chip you could solder directly onto the SOC or Microcontroller so no level of hardware prevetion would achieve this.

For kernel level modifications: The issue I see with this is that manufacturers would have to maintain their own forks of the Linux Kernel with the modifications and ensure they're up to date. This would be a struggle for many to do and require significant investments in time to keep it inline with security of the mainline kernel. I could see if manufacturers don't keep it up to date that the miner would then be at significantly more risk of being hacked via exploits than via hardware. (And with software then at the risk of third parties gaining access and not even just the owner).

For the e-waste part: Essentially though you are saying that the entire device would be e waste? If you wouldn't allow any access to USB Ports then in our gateway hardware for example the entire board would require potting. Along with this is the serial numbers of say the SX1301/2 , RPi and MAC address have to match then would the user be allowed to update these on the firmware that checks the serial numbers if any had to be replaced (Or if the user wanted to upgrade any to better versions).

As for the gateway RS Point - yes the software could be re-written to support it which would be the plan, however in the same discussions when mentioning that no data should be accessible by the owner then unless it was OTA essentially it is being discussed as if the user couldn't then switch.

However if I understand right it then means that essentially then other devices (Computers in the cloud?) become the miners, so would these new miners then have the same security requirements?

Re the gaming it part: I admit I'm not that into Crypto so still trying to figure out the issues behind it, back when I was into crypto I could mine bitcoin with a Radeon HD 7870 and didn't require any special keys and such.

As for software: To be fully closed source you'd then be talking fully custom firmware which again is much more significant time and investment assuming no open source libraries are used.

It's something that we'll need to investigate more but unfortunately means hardware could take a lot longer to ship than if we were to ship hardware that meets the current requirements fine.

[georgica]

@ryanteck
Please bear in mind that goal here is to make it difficult. I am not saying you have to do what i am doing but ensure a level of difficulty

But let's address further:

1. There are various methods to encrypt that make it difficult to copy or shall i say piece together the file...
2. I don't see how that is hard? Just implement a boot check.
3. USB: Again you just literally need to delete the other USB drivers from the kernel and recompile....
4. LKM's are an option and just maintain that....I mean as a manufacturer you have to support and maintain your software...
   (Anybody can throw together some modules on a PCB and call our selfs a manufacturer).
   I have never seen a manufacturer that launched a product and said nope we are not making any updates....
5. What i am saying is that some people buy it for this specific purpose and yes for them once it breaks it serves no other purpose and they will throw it in the bin. But you could use say the RPI for other purposes if you are a maker right? So no not total e-waste....
6. Again nothing is stopping you from writing the EMMC or SD-card with other flavors of linux...you just do the checks on your firmware
7. Essentially you as the manufacturer can keep the miner in the cloud as long as you can keep them secure, so in your current model you could keep your miner on Balena and just authenticate the gateway with some method only known by you type of thing

I don't mean to be rude but all i am hearing is how you guys just threw together some off the shelf components and called it a gateway and have issues maintaining software for your product. Again anybody can do that we call it DIY...

[shawaj]

@georgica for the record, I don't think that what @ryanteck or I are saying is to just "throw together some off the shelf components and call it a gateway". We are simply discussing the actual requirements and intended outcomes and trying to weigh up the legalities and pros and cons of the various methods.

But it is quite clear that there is no consensus surrounding how to move forward with this - because on the one hand you're saying that there should be much more hardware, software and firmware based security, but on the other hand thousands of RAKspots (30k+?) have been sold in the last few months with none of these things considered (and they are also a bunch of off the shelf components). And they are continuing to be sold as we speak.

It feels like there is some disagreement on whether security or fast expansion of the network is the priority perhaps.

With regard to the closed source software side as well I think what Ryan is getting at is that in order to use a lot of the open source software for mining or hotspots you necessarily have to release any modifications under the same license. Yes you could write modules from scratch and supply them as closed source binary blobs or something like that but then it still doesn't really seem to solve the security piece - and means you're relying on trusted third parties with closed source code... This feels a lot like security by obscurity to me as well as not really being in keeping with the idea of decentralisation.

As a side point, balena doesn't "keep" machines in the way you suggest. It's just a way of managing IoT devices remotely. It's not a VM service like digital ocean or whatever.

I guess what all this discussion really comes down to is:

• what "level of difficulty" is sufficient?
• should the implementation of this be decided centrally by the community/helium Inc or on a manufacturer by manufacturer basis?
• if security is the number one priority then why are thousands of miners continuing to be sold without this?
• can the gaming of the system be removed in a way that doesn't require manufacturer specific closed source code?
• if you do auto decryption on boot then you are essentially implementing a secure element/ TPM in firmware based on device IDs. I don't see how this is any more secure than using a secure element for encryption and key storage?
• how to handle hardware failures or upgrades?
• it seems that gateway-rs is being lined up as the future of the network. If that is the case then is there any point in providing alternative hotspot miner manufacturers at this point in time then the ultimate goal is to move away from this approach?
• whilst we are not suggesting that we will not support or update our software, what happens if a device manufacturer does do that or goes bust for example? (This is inevitable over time to occur)

[philltran]

@jamiew What are the major issues holding up approval of this HIP? I think PaulM summed it up well in Discord:

A) securely issuing onboarding keys to manufactures including a process for potentially revoking the ability to onboard new gateways

B) how can swarm key be revoked for an individual gateway for gaming/abuse.

Personally, I think both of these points should be in a separate HIP. My understanding of this HIP 19 is more about approving the third-party manufacture and accepting the vendor applications.

The process of issuing onboarding codes, policing keys etc would better be served in another HIP as that conversation is more complex.

At any rate to keep moving forward, let's concretely define what is holding up approval and work on a solution to those issues.

[cvolkernick]

@philltran it sounded like capcom & hashc0de were on board with at least temporarily resuming the "status quo" method of vendors providing keys to them for the staking server. So that should cover A at least for the context of this HIP. I would possibly even say the 2nd part of that is another HIP, as it sounded like @jamiew was also in on board with. (actually seems like A part 2 and B are more or less the same thing?

[shawaj]

@cvolkernick @philltran I think @jamiew just wants to spread some festive cheer so is waiting until Christmas morning to drop the approval 😜😉

[jamiew]

Via Discord discussion, informal polls, the most recent community calls, and private discussions with folks not represented in those other forums, the Helium community have expressed overwhelming support for approving this proposal and moving forward w/ evaluating the two current applicants, Nebra Ltd & SyncroB.it

Addressing the remaining concerns:

• Helium Inc have indicated their support and willingness to accept onboarding keys from approved manufacturers, and developing a process by which the issuance of those keys can be made public.
• The community & manufacturers have both agreed that it's OK to approve this proposal this system is still being developed
• Lastly, it is agreed that adding the ability to revoke existing keys should be addressed in a separate proposal.

On behalf of the DeWi and the Helium community I'm calling this proposal ✅ approved

[philltran]

Awesome. HIP approval is my favorite.

[cvolkernick]

Congrats @philltran @georgica @jamiew 🥳

[Ernestoxx]

a dashboard should be required. Is 2021 for god sake.

[vincenzospaghetti]

This HIP has been deployed for 2 years! Congrats! We will be closing this issue.