Retrieve components from community sources

[glmnet]

The general idea is to simplify how people uses third party ESPHome components in their installations.

@jesserockz pointed me today about adding ESPHome into HACs, if I understand right, the idea will be that community made components can be retrieved from HACs, and HACs will take care of copying the repository files into the custom_components folder, this has the advantage of many people using more components easily. Here is his conversation with Ludeus https://github.com/hacs/integration/discussions/1935

There is also already an approach here #957 by @timsavage this one relies on PyPI to retrieve the components, and install them on the system, like normal python packages, this requires a modification on ESPHome to find components on the python package system.

--

The problem I see with HACs is that you need HACs!! That is, if you use esphome from command line, or without HomaAssistant "full" (i.e. only core) then this will not help you much.

The problem with packages, I see a bit complex on the user having to figure out how to get the package from git, where to download, where to run the command and let alone have him setup a virtualenv.

Both of these also suffer the problem that the yaml now compiles on a previously setup environment.

I have another idea a few month ago, what about adding this functionality to esphome itself?

components could be retrieved by a special component, e.g.

component_repository:
  - source: https://github.com/glmnet/custom_components/custom1@1.2
  - source: ---

This component will just download the stuff and keep the custom_components folder updated with components needed. There might be a few issues to resolve on the way but seems quite straightforward to use and share stuff

[jesserockz]

First thing that comes to mind is how would you solve different yamls using different component versions/git tags/commit shas?

[glmnet]

that was the issue to solve on the way part

[SenexCrenshaw]

I like your route and think this should not be tied to just HACs. Could not HACs create a yaml file to "automate" this for the end user?

- source: customer_component_source.yaml # Not sure the include syntax off hand

Let HACs create that file. Maybe even add this to the esphome dashboard.

[timsavage]

My intention with packages was more to decouple components from the main ESPHome package (to allow for different release frequency) and allow for third party libraries to be easily integrated.

Installation becomes a simple pip install name_or_package from PyPI or pip install https://github.com/glmnet/custom_components/custom1.git.

Was less about installing individual components although that would be a supported use-case.

Before I got swamped with other work I was looking at the following:

• Adding installation of packages into the UI, would work well in Home Assistant and takes advantage of the mature tooling available in Python for packaging.
• Creating an installer for windows that would create a virtualenv as part of the install process, would simplify the entire process for non-technical users, for OSX/Linux this could be accomplished with a shell script.

However, both the package approach and defining external dependencies in your YAML file that are downloaded as required could go hand in hand. Both solutions cover slightly different use-cases.

[corvis]

Hey @glmnet and @jesserockz check this pull request which is a significant extension to existing packages functionality.
I had to 2 main concepts in mind:

1. Make package to define own public interface so the user who imports 3rd party component shouldn't explore the code to understand how to configure it.
2. More advanced loaders so the package now is not just a single file but rather a real package which might include yaml + .cpp/.h files.

Quick usage example:

packages:
   base: 
     source: packages/heating_ctrl.pkg/base.yaml
     params:
       ssid: blahblah
       log_level: INFO

At the moment I implemented just local file system loader but there is an abstraction which will allows us to implement GitHub loader, maybe some generic http loader etc. So it could be like this:

packages:
   base: 
     source: github:corvis/esphome-packages/packages/jb_heating.pkg@66d019d
     params:
       ssid: blahblah
       log_level: INFO

This could be a good base for community registry and the way to exchange packages.
Also it will allow quickly build easy to use components based on existing generic sensors. Let's say we have generic sensor which allows to read modbus registers. So I could easly create a packages just with yaml to works with let's say SDM energy meter. So user shouldn't messup with complex configuration but instead they could just add a package with clean interface like:

packages:
   sdm120m: 
     source: github:corvis/esphome-packages/packages/sdmxxx.pkg@66d019d
     params:
       address: 0x12
       model: sdm12m

[OttoWinter]

I'm a bit confused, the package system is only for loading YAML file fragments (like some configuration section), right? The way I understood it this discussion is about loading external components (so the source that gets executed etc).

Still very interesting concept though!

[corvis]

Right @OttoWinter, this is a bit different but the purpose might be similar. E.g. in my current scenario I have yaml package + cpp with some custom logic for irrigation controller. The new version of packages allows to encapsulate yaml config + includes + C code so the end user shouldn't care about it and just have to follow the package contract.

The main reason I've mentioned it in this thread is the common part related to loading external resources I believe it should be unified and my PR contains some initial efforts in this direction.

[OttoWinter]

I implemented a version of this here and would like some feedback on the syntax.

For the system: I saw @timsavage's proposal for using PyPi/pip for this, which looks interesting. However, I think people having to publish to PyPi (and having to set up the setup.py file etc for packaging) is a problem. Also I heard that pkg_resources has a high overhead (at least it says so in the importlib docs). My PR implements it using the importlib MetaPathFinder system, which really is just a couple of lines of code in loader.py.

So the question for you guys is: What syntax for the YAML, and what repository setup for the custom component should we use?

The PR implements it like this:

external_components:
  - source: esphome/esphome@importlib-custom-components
    # Optional: specify which components to import, otherwise all
    components: [bh1750]

That says: import the bh1750 component from the esphome/esphome github repo on branch/tag importlib-custom-components.

The repo setup should then be like this:

for example:

.
├── components
│   └── my_custom_component
│       ├── __init__.py
│       ├── my_custom_component.cpp
│       ├── my_custom_component.h
│       └── sensor.py
└── README.md

This way, if the custom component needs multiple esphome components, those can easily be added.

So, is this setup good/would another one be better for custom component creators? Is there a better name for the external_components section?

[mmakaay]

Yes, it all sounds great and I recognized a lot of my mumbling while reading the pull request discussion after posting this.
I think the feature will be very much welcomed. I have some people who are willing to test out my component code, and how to download and incorporate the component code in the ESPHome config tree is the main recurring support issue. So really looking forward to this.

[glmnet]

@mmakaay I've done docs and fixed an issue with paths, I believe it will help if some testing is done (and review), Thanks

[mmakaay]

I tested the new feature and it worked flawlessly.
The only remark that I would have about the behavior during the esphome compile, is that there seems to be no feedback about the repository being included on the initial clone operation.
During the second compile, I do see that the repo is pulled (I set the refresh time to 0s).

[timsavage]

This looks good. Haven't needed to test yet, but the syntax is clear and supporting any GIT location (and local directories) covers any scenario I'd likely encounter.

[mmakaay]

I am testing the behavior of type: local , and I've got zero complaints.

I've pointed it to a folder in which I cloned my github component repo (which is in the layout as required by the external components), and now I'm able to work on the component's code and compile the firmware.