New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Library Dependency Graph does not show full version information #1274

Open
euphi opened this Issue Jan 4, 2018 · 7 comments

Comments

Projects
None yet
2 participants
@euphi

euphi commented Jan 4, 2018

The Library Dependency Graph does not show the installation method or beta-versions (or git revision) of used libraries. Also, if different revisions of a library are installed (because it is a "sub-dependency" of different dependencies with differing version requirements) they may be shown as same revision.

So I propose some improvements Library Dependency Graph to help tracking down problems that may arise due to complex version dependencies:

  • There should be a clear indication about the installed revision and the installation method (repository, git, etc..)
  • There should also be a warning message if a dependency is installed in different versions.
  • Furthermore, if exactly the same dependency is used several times, the sub-dependencies should not be shown again. (Maybe something like | |-- ..repeated.. could be shown instead).

Example 1: (taken from https://travis-ci.org/euphi/Homie_BareMinimum/builds/324773837)

--> The graph should indicate that the library is directly installed from git and with git-tag v2.0.0-beta.3.

Library Dependency Graph ( http://bit.ly/configure-pio-ldf )
|-- <Homie> v2.0.0
|   |-- <ArduinoJson> v5.12.0
|   |-- <AsyncMqttClient> v0.8.1
|   |   |-- <ESPAsyncTCP> v1.0.1
|   |-- <Bounce2> v2.3
|   |-- <ESP Async WebServer> v1.1.1
|   |   |-- <ESPAsyncTCP> v1.0.1
|   |   |-- <ESP8266WiFi> v1.0
|   |   |-- <Hash> v1.0
|   |   |-- <ArduinoJson> v5.12.0
|   |-- <DNSServer> v1.1.0
|   |   |-- <ESP8266WiFi> v1.0
|   |-- <Ticker> v1.0
|   |-- <ESPAsyncTCP> v1.0.1
|   |-- <ESP8266mDNS>
|   |   |-- <ESP8266WiFi> v1.0
|   |-- <ESP8266HTTPClient> v1.1
|   |   |-- <ESP8266WiFi> v1.0
|   |-- <ESP8266WiFi> v1.0
|-- <Hash> v1.0

Example 2: (taken from https://travis-ci.org/euphi/ESP-Touch/jobs/324666146, my by far most complex project)

  • Various dependencies (I know I can simplify the dependency tree by removing redundant dependencies form my own projects, but this shows the dependency tree as it has grown over time)
|-- <Homie> v2.0.0                            <------- https://github.com/marvinroger/homie-esp8266.git#v2.0.0-beta.3
|   |-- <ArduinoJson> v5.12.0
|   |-- <AsyncMqttClient> v0.8.1
|   |   |-- <ESPAsyncTCP> v1.0.1
|   |-- <Bounce2> v2.3
|   |-- <ESP Async WebServer> v1.1.1
|   |   |-- <ESPAsyncTCP> v1.0.1
|   |   |-- <ESP8266WiFi> v1.0
|   |   |-- <Hash> v1.0
|   |   |-- <ArduinoJson> v5.12.0
|   |-- <DNSServer> v1.1.0
|   |   |-- <ESP8266WiFi> v1.0
|   |-- <ESPAsyncTCP> v1.0.1
|   |-- <ESP8266WiFi> v1.0
|   |-- <ESP8266mDNS>
|   |   |-- <ESP8266WiFi> v1.0
|   |-- <ESP8266HTTPClient> v1.1
|   |   |-- <ESP8266WiFi> v1.0
|   |-- <Ticker> v1.0
|-- <HomieNodeCollection> v0.8.1
|   |-- <Homie> v2.0.0                            <------- https://github.com/marvinroger/homie-esp8266.git#v2.0.0-beta.3
|   |   |-- <ArduinoJson> v5.12.0
|   |   |-- <AsyncMqttClient> v0.8.1
|   |   |   |-- <ESPAsyncTCP> v1.0.1
|   |   |-- <Bounce2> v2.3
|   |   |-- <ESP Async WebServer> v1.1.1
|   |   |   |-- <ESPAsyncTCP> v1.0.1
|   |   |   |-- <ESP8266WiFi> v1.0
|   |   |   |-- <Hash> v1.0
|   |   |   |-- <ArduinoJson> v5.12.0
|   |   |-- <DNSServer> v1.1.0
|   |   |   |-- <ESP8266WiFi> v1.0
|   |   |-- <ESPAsyncTCP> v1.0.1
|   |   |-- <ESP8266WiFi> v1.0
|   |   |-- <ESP8266mDNS>
|   |   |   |-- <ESP8266WiFi> v1.0
|   |   |-- <ESP8266HTTPClient> v1.1
|   |   |   |-- <ESP8266WiFi> v1.0
|   |   |-- <Ticker> v1.0
|   |-- <ESP8266_SSD1306> v3.2.7
|   |   |-- <Wire> v1.0
|   |-- <Wire> v1.0
|   |-- <ESP8266WiFi> v1.0
|   |-- <HomieLoggerNode> v0.9.1
|   |   |-- <Homie> v2.0.0                            <------- https://github.com/marvinroger/homie-esp8266.git#develop
|   |   |   |-- <ArduinoJson> v5.12.0
|   |   |   |-- <AsyncMqttClient> v0.8.1
|   |   |   |   |-- <ESPAsyncTCP> v1.0.1
|   |   |   |-- <Bounce2> v2.3
|   |   |   |-- <ESP Async WebServer> v1.1.1
|   |   |   |   |-- <ESPAsyncTCP> v1.0.1
|   |   |   |   |-- <ESP8266WiFi> v1.0
|   |   |   |   |-- <Hash> v1.0
|   |   |   |   |-- <ArduinoJson> v5.12.0
|   |   |   |-- <DNSServer> v1.1.0
|   |   |   |   |-- <ESP8266WiFi> v1.0
|   |   |   |-- <ESPAsyncTCP> v1.0.1
|   |   |   |-- <ESP8266WiFi> v1.0
|   |   |   |-- <ESP8266mDNS>
|   |   |   |   |-- <ESP8266WiFi> v1.0
|   |   |   |-- <ESP8266HTTPClient> v1.1
|   |   |   |   |-- <ESP8266WiFi> v1.0
|   |   |   |-- <Ticker> v1.0
|   |   |-- <Hash> v1.0
|-- <HomieLoggerNode> v0.9.1
|   |-- <Homie> v2.0.0                            <------- repetition (#develop)
|   |   |-- <ArduinoJson> v5.12.0
|   |   |-- <AsyncMqttClient> v0.8.1
|   |   |   |-- <ESPAsyncTCP> v1.0.1
|   |   |-- <Bounce2> v2.3
|   |   |-- <ESP Async WebServer> v1.1.1
|   |   |   |-- <ESPAsyncTCP> v1.0.1
|   |   |   |-- <ESP8266WiFi> v1.0
|   |   |   |-- <Hash> v1.0
|   |   |   |-- <ArduinoJson> v5.12.0
|   |   |-- <DNSServer> v1.1.0
|   |   |   |-- <ESP8266WiFi> v1.0
|   |   |-- <ESPAsyncTCP> v1.0.1
|   |   |-- <ESP8266WiFi> v1.0
|   |   |-- <ESP8266mDNS>
|   |   |   |-- <ESP8266WiFi> v1.0
|   |   |-- <ESP8266HTTPClient> v1.1
|   |   |   |-- <ESP8266WiFi> v1.0
|   |   |-- <Ticker> v1.0
|   |-- <Hash> v1.0
|-- <Automaton> v1.0.2
|-- <Adafruit NeoMatrix> v1.1.2
|   |-- <Adafruit GFX Library> v1.2.2
|   |-- <Adafruit NeoPixel> v1.1.3
|-- <Adafruit GFX Library> v1.2.2
|-- <Adafruit NeoPixel> v1.1.3
|-- <MPR121> v1.0.0
|   |-- <Wire> v1.0
|-- <Hash> v1.0
|-- <ESP8266_SSD1306> v3.2.7
|   |-- <Wire> v1.0
|-- <Wire> v1.0
@ivankravets

This comment has been minimized.

Member

ivankravets commented Jan 4, 2018

We use library name and version from manifest. The full detailed version is available via pio run --verbose command.

@euphi

This comment has been minimized.

euphi commented Jan 4, 2018

The output with --verbose still does not show the origin of installed libraries:

Library Dependency Graph ( http://bit.ly/configure-pio-ldf )
|-- <Homie> v2.0.0 (/home/ian/Coding/ESP_Projects/Homie_BareMinimum/.piolibdeps/Homie)
|   |-- <ArduinoJson> v5.12.0 (/home/ian/Coding/ESP_Projects/Homie_BareMinimum/.piolibdeps/ArduinoJson_ID64)
|   |-- <AsyncMqttClient> v0.8.1 (/home/ian/Coding/ESP_Projects/Homie_BareMinimum/.piolibdeps/AsyncMqttClient_ID346)
|   |   |-- <ESPAsyncTCP> v1.1.3 (/home/ian/Coding/ESP_Projects/Homie_BareMinimum/.piolibdeps/ESPAsyncTCP_ID305)
|   |-- <Bounce2> v2.3 (/home/ian/Coding/ESP_Projects/Homie_BareMinimum/.piolibdeps/Bounce2_ID1106)
|   |-- <ESP Async WebServer> v1.1.1 (/home/ian/Coding/ESP_Projects/Homie_BareMinimum/.piolibdeps/ESP Async WebServer_ID306)
|   |   |-- <ESPAsyncTCP> v1.1.3 (/home/ian/Coding/ESP_Projects/Homie_BareMinimum/.piolibdeps/ESPAsyncTCP_ID305)
|   |   |-- <ESP8266WiFi> v1.0 (/home/ian/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|   |   |-- <Hash> v1.0 (/home/ian/.platformio/packages/framework-arduinoespressif8266/libraries/Hash)
|   |   |-- <ArduinoJson> v5.12.0 (/home/ian/Coding/ESP_Projects/Homie_BareMinimum/.piolibdeps/ArduinoJson_ID64)
|   |-- <ESP8266mDNS> (/home/ian/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266mDNS)
|   |   |-- <ESP8266WiFi> v1.0 (/home/ian/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|   |-- <ESPAsyncTCP> v1.1.3 (/home/ian/Coding/ESP_Projects/Homie_BareMinimum/.piolibdeps/ESPAsyncTCP_ID305)
|   |-- <DNSServer> v1.1.0 (/home/ian/.platformio/packages/framework-arduinoespressif8266/libraries/DNSServer)
|   |   |-- <ESP8266WiFi> v1.0 (/home/ian/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|   |-- <Ticker> v1.0 (/home/ian/.platformio/packages/framework-arduinoespressif8266/libraries/Ticker)
|   |-- <ESP8266HTTPClient> v1.1 (/home/ian/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266HTTPClient)
|   |   |-- <ESP8266WiFi> v1.0 (/home/ian/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|   |-- <ESP8266WiFi> v1.0 (/home/ian/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi)
|-- <Hash> v1.0 (/home/ian/.platformio/packages/framework-arduinoespressif8266/libraries/Hash)
|-- <Wire> v1.0 (/home/ian/.platformio/packages/framework-arduinoespressif8266/libraries/Wire)

@ivankravets ivankravets added this to the 3.5.1 milestone Jan 4, 2018

@ivankravets

This comment has been minimized.

Member

ivankravets commented Jan 4, 2018

Thanks, I'll try to add info about VSC repo.

@euphi

This comment has been minimized.

euphi commented Jan 4, 2018

As you know, there were many problems in the last weeks with some dependencies in development version in the Homie-ESP8266 project. It was quite complicated to see the cause of the problems, especially if it was introduced by an update of a library or even the framework.

I'm thinking how platformio can help to find such problems. I propose the following ideas (list includess the ideas from my original post):

  • clear indication about the installed version
  • clear indication about the origin of a library (repo, git, svc, ....)
  • Warning if a library is installed multiple times with different versions
  • Warning (or Notice) if a library is required again within the same subtree
    • Rationale for this: In this case the higher-layer does not need to depend on the library, because one (or more) of its dependencies already requires the library. In my original post, in example 2, the dependency on Homie from the project is redundant, because the dependency HomieNodeCollection also requires Homie. Furthermore, this dependency to Homie from HomieNodeCollection is also redundant, because its dependency HomieLoggerNode also depend on it. Having these dependencies is not wrong, but complicate things. So, if platformio warns (or informs) the user about this, this helps encouraging library developers to keep their dependencies clean.
  • Warning if version from manifest mismatches with the version from origin (e.g git tag)
  • Allow registration of different library branches (versions) at repository
  • adding a "lint"-target that only checks if a warning is present and if so, fails. (to be used with CI).
@ivankravets

This comment has been minimized.

Member

ivankravets commented Jan 4, 2018

Thanks a lot for ideas!

P.S:

Homie-ESP8266 project

ESP8266 Arduino Core 2.4.0 is finally released! No need to use staging branch. You can switch Homie.

@ivankravets ivankravets added the lib label Jan 10, 2018

@ivankravets ivankravets modified the milestones: 3.5.1, 3.x.x Jan 18, 2018

ivankravets added a commit that referenced this issue Jan 18, 2018

@ivankravets

This comment has been minimized.

Member

ivankravets commented Jan 18, 2018

Thanks! I did some improvements in 2c79de9

I don't close this issue and moved it to 3.x.x milestone. I don't have a clear picture how to resolve all ideas which you described above.

Let me post a few comments:

Warning if a library is installed multiple times with different versions

We can't do that because that is a gold feature of PIO Core where you can have multiple build environments where each of them depends on own library version. As result, different version could be installed in .piolibdeps folder. When LDF starts a work, it will pick up the best version following your requirements in lib_deps for the particular build environment ([env:..])

Warning (or Notice) if a library is required again within the same subtree

You don't need to bother with that. Linker will optimize all libraries and object. It does not mean that dependent library will be included in final ELF multiple times. We just show the FULL DEPENDENCY GRAPH. There is no link between LDF GRAPH and GCC LINKER.

LDF GRAPH shows you all dependencies starting from a project root and following all INCLUDE directives

GCC LINKER analyze all libraries in circle mode (we enable it by default for GCC) and pick up the first declared symbols from the libs/objects

Warning if version from manifest mismatches with the version from origin (e.g git tag)

This is a very difficult task. VCS does not have "version" information. We have COMMIT, BRANCH, and TAG. Version - this is developer's "thing". If we start looking in VCS log, list all tags, parse them to version, we will significantly decrease performance.

Allow registration of different library branches (versions) at repository

Please provide more information.

adding a "lint"-target that only checks if a warning is present and if so, fails. (to be used with CI).

We have some idea in our TODO about pio check command, but I can't guarantee any ETA.

A temporary solution could be

pio run --verbose | grep warning

ivankravets added a commit to ivankravets/homebrew-core that referenced this issue Jan 18, 2018

platformio 3.5.1
* New ``test_speed`` option to control a communication baudrate/speed between [PIO Unit Testing](http://docs.platformio.org/page/plus/unit-testing.html) engine and a target device ([issue Homebrew#1273](platformio/platformio-core#1273))
* Show full library version in "Library Dependency Graph" including VCS information ([issue Homebrew#1274](platformio/platformio-core#1274))
* Configure a custom firmware/program name in build directory ([example](http://docs.platformio.org/page/projectconf/advanced_scripting.html#custom-firmware-program-name))
* Renamed ``envs_dir`` option to ``build_dir`` in [Project Configuration File "platformio.ini"](http://docs.platformio.org/page/projectconf/section_platformio.html#build-dir)
* Refactored code without "arrow" dependency (resolve issue with "ImportError: No module named backports.functools_lru_cache")
* Improved support of PIO Unified Debugger for Eclipse Oxygen
* Improved a work in off-line mode
* Fixed project generator for CLion and Qt Creator IDE ([issue Homebrew#1299](platformio/platformio-core#1299))
* Fixed PIO Unified Debugger for mbed framework
* Fixed library updates when a version is declared in VCS format (not SemVer)

BrewTestBot added a commit to BrewTestBot/homebrew-core that referenced this issue Jan 18, 2018

platformio 3.5.1
* New ``test_speed`` option to control a communication baudrate/speed between [PIO Unit Testing](http://docs.platformio.org/page/plus/unit-testing.html) engine and a target device ([issue Homebrew#1273](platformio/platformio-core#1273))
* Show full library version in "Library Dependency Graph" including VCS information ([issue Homebrew#1274](platformio/platformio-core#1274))
* Configure a custom firmware/program name in build directory ([example](http://docs.platformio.org/page/projectconf/advanced_scripting.html#custom-firmware-program-name))
* Renamed ``envs_dir`` option to ``build_dir`` in [Project Configuration File "platformio.ini"](http://docs.platformio.org/page/projectconf/section_platformio.html#build-dir)
* Refactored code without "arrow" dependency (resolve issue with "ImportError: No module named backports.functools_lru_cache")
* Improved support of PIO Unified Debugger for Eclipse Oxygen
* Improved a work in off-line mode
* Fixed project generator for CLion and Qt Creator IDE ([issue Homebrew#1299](platformio/platformio-core#1299))
* Fixed PIO Unified Debugger for mbed framework
* Fixed library updates when a version is declared in VCS format (not SemVer)
@euphi

This comment has been minimized.

euphi commented Jan 22, 2018

Warning if a library is installed multiple times with different versions

We can't do that because that is a gold feature of PIO Core where you can have multiple build environments where each of them depends on own library version. As result, different version could be installed in .piolibdeps folder. When LDF starts a work, it will pick up the best version following your requirements in lib_deps for the particular build environment ([env:..])

Ah ok. I didn't think about the build environments. However, it happens that libraries within the same build environment depens on different versions of another libraries. In this case, both versions are installed, but the build process uses only one (the newer one?). In my original post this happend to the Homie-dependency.

Warning (or Notice) if a library is required again within the same subtree

You don't need to bother with that. Linker will optimize all libraries and object. It does not mean that dependent library will be included in final ELF multiple times. We just show the FULL DEPENDENCY GRAPH. There is no link between LDF GRAPH and GCC LINKER.

My idea behind this was to encourage developers to have clean dependencies, to avoid unnecessary complicated trees (that may break due to updates and then conflicting dependencies). Of course there is no need to fix the warning - or better just call it "notice".

Warning if version from manifest mismatches with the version from origin (e.g git tag)

This is a very difficult task. VCS does not have "version" information. We have COMMIT, BRANCH, and TAG. Version - this is developer's "thing". If we start looking in VCS log, list all tags, parse them to version, we will significantly decrease performance.

You only need to do this for libraries that are directly installed from git (or other VCS). For libraries from repository there is no need to do this at runtime, but you could send a warning message to the developer when importing a new version to the repository server.

The idea behind this is again to help library developers to get their versions and dependencies clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment