Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document module dependency matrix #386

Open
marcelstoer opened this issue Apr 25, 2015 · 18 comments
Open

Document module dependency matrix #386

marcelstoer opened this issue Apr 25, 2015 · 18 comments
Assignees

Comments

@marcelstoer
Copy link
Member

@marcelstoer marcelstoer commented Apr 25, 2015

It's helpful I can easily include/exclude particular modules for custom builds using user_modules.h. However, this is always trial-and-error I assume because there's no documented module dependency matrix. If I exclude module X will my NodeMCU binary be broken because module Y depends on X?

This request is to maintain (in wiki) or auto-generate a module dependency matrix.

@marcelstoer
Copy link
Member Author

@marcelstoer marcelstoer commented May 7, 2015

I was just wondering if that's even necessary. Could it be that all modules are, or have to be, independent of each other anyway?

@marcelstoer marcelstoer mentioned this issue May 7, 2015
@vowstar
Copy link
Member

@vowstar vowstar commented Jun 25, 2015

TODO:
Build firmware dependent tree and make all modules more dependent.
@MarsTechHAN @vowstar

@MarsTechHAN
Copy link
Contributor

@MarsTechHAN MarsTechHAN commented Jun 25, 2015

But like SSL, some modules wont use it, and SSL will consume a lot RAM...

How about make a trick?

#ifdef MQTT

#define SSL

#endif

And cross-test will consume some time...We will get this done in a week.

@marcelstoer
Copy link
Member Author

@marcelstoer marcelstoer commented Jun 25, 2015

How about make a trick?

MQTT is the only module that fails the build if SSL is disabled, see #432 (there's a note about this in my cloud build). Hence, once this is fixed the SSL setting is totally independent from other settings. IMO that's the way to go.

I'd welcome if you postulate a guideline which says that all non-core modules (e.g. MQTT, 1-wire, I²C, cJSON, etc.) must be standalone and not depend on any other non-core module. Running makedepend or the compiler with the -M option could provide some automatic dependency analysis.

@TerryE
Copy link
Collaborator

@TerryE TerryE commented Jun 25, 2015

Ladder logic tricks like this are pretty standard but typically you would have the primary defines as a block and the dependency defines as a second block so in this you'd have something like

#if defined(MQTT) && !defined(SSL)
#  define SSL
#endif

This is the simplest and tidiest way to add such dependences

@confile
Copy link

@confile confile commented Nov 21, 2015

+1

@marcelstoer
Copy link
Member Author

@marcelstoer marcelstoer commented Jul 4, 2016

@TerryE @devsaurus @jmattsson @pjsg This starts to bother me more and more and I'm quite convinced that it'll bite us medium-/long-term if we don't tackle this. When I created this issue nearly a year ago we had maybe 20 modules, now there are more than 40.

Initially I thought it would be enough if we had a simple dependency matrix. However, meanwhile we have different sorts of dependencies.

  • classical, module A requires module B: currently with #1379 it looks like you'd need the new tls module for net, http and mqtt if you want secure connections
  • exclusive shared resources: hw_timer used by PCM and others
  • individual modules intended to be used together: rtctime, rtcmem, SNTP
  • de-facto dependencies: #1321, "User is then free to choose how to generate this info. Preferably from rtctime.get()"

Do you think it'd be feasible to maintain a n*n matrix (n=number of modules) and give intersecting cells a specific color (see list above)? If further clarifications are required colorized cells may contain foot note references of course. Can you think of a more sensible approach?

@jmattsson
Copy link
Member

@jmattsson jmattsson commented Jul 4, 2016

Visualising dependencies is tricky. A colour-coded matrix is probably not a bad choice, but I am a bit concerned about its size when we get to e.g. 80 modules. Right now I can't think of anything smarter though, but I'll sleep on it and see if anything pops into my head.

@pjsg
Copy link
Member

@pjsg pjsg commented Jul 4, 2016

There are two aspects to this -- one is to make the build work with arbitrary combinations of modules and the other is to make sure all the required functionality is present.

For the first, I think that the code ought to get it right (i.e. have the dependencies in some header file).

For the latter, the situation is a little more complex - but I don't think that there are many cases here. The TLS situation is the most obvious. I don't mind this being explicit.

For the sntp/rtctime case, I'm in favor of just treating these as hard dependencies and using the header file approach.

For the hw_timer case, this is already handled by the build system -- it is always only included if some module that needs it is included (this is the linker magic).

Presumably it would be possible to write a script that, given a list of lua files, extracted all the names that need to be defined and thereby could extract the required module list.

@devsaurus
Copy link
Member

@devsaurus devsaurus commented Jul 4, 2016

A matrix is probably ok for an overview. Though I'm not a fan of encoding information exclusively with colours.
As a con: You'd have to iterate over the matrix again and again in case other modules need to be included due to recommended or must-have dependencies.

If you're looking for a way to guide the user wrt module selection, then the dependencies could also be defined as a set of statements per module. A structured format (JSON?) with dependency type, related module and a description:

  • u8g:
    • recommendation spi: some drivers require SPI interface
    • recommendation i2c: some drivers require I2C interface
  • pcm:
    • constraint pwm: no parallel operation
    • constraint perf: no parallel operation
  • ucg:
    • requirement spi: all drivers require SPI

A local script or nodemcu-build would run through the statements based on the selected modules and provided a summary about

  • missing/recommended modules
  • constraints for the user application
@marcelstoer
Copy link
Member Author

@marcelstoer marcelstoer commented Jul 4, 2016

I am a bit concerned about its size when we get to e.g. 80 modules.

Me too.

For the first, I think that the code ought to get it right (i.e. have the dependencies in some header file).

True but it'd still be nice if we could visualize this for our users. I anticipate questions like "Why is there module X in my firmware, I didn't enable it in user_modules.h?".

For the sntp/rtctime case, I'm in favor of just treating these as hard dependencies and using the header file approach.

AFAIU they're not, it's just that for the anticipated use cases you'll need all of them anyway. Correct Johny?

For the hw_timer case, this is already handled by the build system.

Yes, but I mean to document very clearly that you can't use pcm, perf and pwm at the same time.

...the dependencies could also be defined as a set of statements per module. A structured format (JSON?) with dependency type, related module and a description

I had been thinking about something like that as well. Haven't thought it through all the way, though. It'd most likely be easier to maintain than the matrix and it's got other pros:

  • could be injected into the respective module documentation page (similar to the current meta-info table) OR you do it the other way around: main in the docs and optionally generate an aggregated file
  • it scales better than the matrix
  • you could still generate a matrix from it, whereas the other way 'round would be a little more difficult
@stale
Copy link

@stale stale bot commented Jun 7, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Jun 7, 2019
@stale stale bot closed this Jun 21, 2019
@mk-pmb
Copy link
Contributor

@mk-pmb mk-pmb commented Dec 30, 2019

Could we maybe borrow the npm module manifest format (package.json) to use deps detection and visualisation tools from the JavaScript community? npm itself is very flexible about package contents, it's not limited to JavaScript modules. The manifest format even officially supports specifying your platforms like

{ "engines" : { "lua" : ">=0.1.0" } }

so no-one will misunderstand the modules as JavaScript modules.

@HHHartmann HHHartmann self-assigned this Dec 31, 2019
@HHHartmann
Copy link
Member

@HHHartmann HHHartmann commented Dec 31, 2019

I feel that this is still important. As we do not have individual packages in our C code the npm format seems a bit overdone for this though.
The categories that @devsaurus identified above seem to match our needs.

@czesla
Copy link

@czesla czesla commented Feb 22, 2020

Hello
I downloaded NodeMCU custom builds by adding 7 modules (file, GPIO, net, node, timer, UART, WIFI) without any additional options.
I launched the ESPFlasher application with the files "nodemcu-master-7-modules-2020-02-22-13-36-35-integer.bin" (434kb), address 0x000000 and "init_data_default.bin" address 0x3FC000.
After starting the board with ESPlorer I get the message:
"Can't autodetect firmware, because proper answer not received (may be unnown firmware)."
I repeated the firmware flashing at the address 0x000000 width "INTERNAL: // NODEMCU" instead of the downloaded "nodemcu-master-7-modules-2020-02-22-13-36-35-integer.bin" and then it was OK, so the ESP8266 system recognizes the firmware.
I tested Firware on two different types of ESP8266 boards (wemos D1 mini and LoLin new NodeMcu V3), each time with the same negative stroke.
I do not understand what the problem is? What i do wrong?

@czesla
Copy link

@czesla czesla commented Feb 22, 2020

Why the firmware is not working?

@HHHartmann
Copy link
Member

@HHHartmann HHHartmann commented Feb 22, 2020

@czesla Nice that you are willing to use this project.
I assume that you are using ESPlorer to connect to the esp8266? This program tries to detect some string in the output of the esp. Newer versions of the firmware have changed this output, so ESPlorer detection fails. Try to press the reset button on your esp8266 and you will (most likely) see the output inside ESP8266. If its only garbage try to change the serial speed.

That said, two more points:

@czesla
Copy link

@czesla czesla commented Feb 22, 2020

Thank you for your response. I changed the port speed to 115200 and now it's OK. Once again, thank you and sorry for bothering you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.