Implement external custom components installing from YAML

[OttoWinter]

What does this implement/fix?

A prototype for #1554

This implements a new system for downloading and including custom components straight from YAML. Note: early PoC - syntax will change.

Important: This also changes the python requirement to 3.7 (previously was 3.6). Now that the docker base image is running 3.7 and most distributions have 3.7+ already, that should not be a major problem.

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Configuration change (this will require users to update their yaml configuration files to keep working)

Related issue or feature (if applicable): fixes TODO

Pull request in esphome-docs with documentation (if applicable): esphome/esphome-docs#1107

Test Environment

• ESP32
• ESP8266
• Windows
• Mac OS
• Linux

Example entry for config.yaml:

# Set the component bh1750 to be imported from GitHub esphome/esphome repo on branch importlib-...
external_components:
  - source: esphome/esphome@importlib-custom-components
    components: [bh1750]

Explain your changes

This PR is split in two (large) commits:

• First, I refactored and rewrote the component loader to use importlib's MetaPathFinder system
• The second commit implements the external_components system.

The expected format for the repository is:

<REPO_ROOT>/components/my_custom_component/* (this directory must include at least a __init__.py file); this is how I expect custom component creators to present the code.

<REPO_ROOT>/esphome/components/my_custom_component/* (this directory must include at least a __init__.py file); this format is used mainly for being able to import certain components from a specific esphome branch/version

For now, only git and local sources are supported, others can be added later.

Future work includes refreshing the repository every so often.

Checklist:

• The code change is tested and works locally.
• Tests have been added to verify that the new code works (under tests/ folder).

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated in esphome-docs.

[glmnet]

This is definitely simplify component sharing on the community. 🎉🎉

[glmnet]

Beside this which mostly indicates an user entering wrong source url, I would add a check where it explicitly needs to indicate overriden ESPHome bundled components. So if you want a new fancy captive_portal then you must indicate it here, otherwise ESPHome's is used.

[OttoWinter]

I see. On the other hand, doing so would be a problem for some external integrations.

For example: I create an external repo for my fancy new display driver xyz9000 and publish that. Then I want to add another display like xyz9001, but at some point I want to refactor some of the shared code into a new module xyz_base.

In that case, if we force users to manually set the components, we would really limit external component repos - getting all users to change their config would be painful.

I don't think stuff like display drivers will overwrite some internal stuff like captive_portal - or if it does end up being a problem we can create a policy where certain "core" components have to explicitly opted in. We can do that later too because requiring that would not be a big breaking change (I don't suspect many external integrations will overwrite core stuff)

[glmnet]

is this a standard like the way platformio or GH actions names refers to github repos? its a bit magical as no mention to github is made on the string itself, like github:// etc. just an opinion

[OttoWinter]

Correct, that syntax is based on GH actions'.

The reason I don't want a full URL like https://github.com/esphome/esphome@dev is because then with the @dev part it wouldn't be an actual URL (as in if you type it in your browser you'll get an error page)

github:// could be a good alternative though

[glmnet]

I was thinking about this, one minor issue custom_components mechanism has now is that if you put a component there, it will be applied to all your components, that makes sense on HA as you only run one Home Assistant, hwoever all yaml files will share the included elements in custom_components

While this solves the issue, as you can use the file:// in source in a single yaml then you can have a custom component just for one yaml, I believe a mix of this component serving custom_components folder by default will fit better.

I propose a default (always on) configuration for external_components which will load custom_components folder:

external_components:
  - type: local
    path: './custom_components'
    components: 'all'

So that configuration is default, and it will act as current default behavior of ESPHome, but can be overriden so you can:

external_components:

Will disable loading of custom_components
and you can also override to have it load just one component on the custom_components folder:

external_components:
  - type: local
    path: './custom_components'
    components: [ 'rtttl' ]

This might also plain the path to make mandatory setting of the component to be overriden, allowing all as is might yield unexpected results, e.g. if I load a component from a repo, and then the a new component shows up which overrides one of my components I might not notice.

[OttoWinter]

@glmnet I see what you mean with the custom_components dir. One major thing to consider is that default configs for lists can be very unintuitive. For example: very early on ESPHome would automatically apply a smoothing sliding_window_moving_average filter for every sensor, but we'd get tons of reports of "why is the output suddenly so different when I add this other filter".

Maybe we can do something like HA instead: Print a warning message for every component loaded through the /custom_components directory. external_components clearly is the way even local experiments should be implemented with in the future, as it's a safer alternative where the user explicitly opts in. By having a good alternative, the warning message would also not be a big problem.

[glmnet]

Dashboard screenshot:

[OttoWinter]

@glmnet Thank you so much for finishing this PR and the docs! Really appreciated!