New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document module dependency matrix #386

Open
marcelstoer opened this Issue Apr 25, 2015 · 11 comments

Comments

Projects
None yet
8 participants
@marcelstoer
Collaborator

marcelstoer commented Apr 25, 2015

It's helpful I can easily include/exclude particular modules for custom builds using user_modules.h. However, this is always trial-and-error I assume because there's no documented module dependency matrix. If I exclude module X will my NodeMCU binary be broken because module Y depends on X?

This request is to maintain (in wiki) or auto-generate a module dependency matrix.

@marcelstoer

This comment has been minimized.

Show comment
Hide comment
@marcelstoer

marcelstoer May 7, 2015

Collaborator

I was just wondering if that's even necessary. Could it be that all modules are, or have to be, independent of each other anyway?

Collaborator

marcelstoer commented May 7, 2015

I was just wondering if that's even necessary. Could it be that all modules are, or have to be, independent of each other anyway?

@marcelstoer marcelstoer referenced this issue May 7, 2015

Closed

Low heap. #337

@vowstar

This comment has been minimized.

Show comment
Hide comment
@vowstar

vowstar Jun 25, 2015

Collaborator

TODO:
Build firmware dependent tree and make all modules more dependent.
@MarsTechHAN @vowstar

Collaborator

vowstar commented Jun 25, 2015

TODO:
Build firmware dependent tree and make all modules more dependent.
@MarsTechHAN @vowstar

@MarsTechHAN

This comment has been minimized.

Show comment
Hide comment
@MarsTechHAN

MarsTechHAN Jun 25, 2015

Contributor

But like SSL, some modules wont use it, and SSL will consume a lot RAM...

How about make a trick?

#ifdef MQTT

#define SSL

#endif

And cross-test will consume some time...We will get this done in a week.

Contributor

MarsTechHAN commented Jun 25, 2015

But like SSL, some modules wont use it, and SSL will consume a lot RAM...

How about make a trick?

#ifdef MQTT

#define SSL

#endif

And cross-test will consume some time...We will get this done in a week.

@marcelstoer

This comment has been minimized.

Show comment
Hide comment
@marcelstoer

marcelstoer Jun 25, 2015

Collaborator

How about make a trick?

MQTT is the only module that fails the build if SSL is disabled, see #432 (there's a note about this in my cloud build). Hence, once this is fixed the SSL setting is totally independent from other settings. IMO that's the way to go.

I'd welcome if you postulate a guideline which says that all non-core modules (e.g. MQTT, 1-wire, I²C, cJSON, etc.) must be standalone and not depend on any other non-core module. Running makedepend or the compiler with the -M option could provide some automatic dependency analysis.

Collaborator

marcelstoer commented Jun 25, 2015

How about make a trick?

MQTT is the only module that fails the build if SSL is disabled, see #432 (there's a note about this in my cloud build). Hence, once this is fixed the SSL setting is totally independent from other settings. IMO that's the way to go.

I'd welcome if you postulate a guideline which says that all non-core modules (e.g. MQTT, 1-wire, I²C, cJSON, etc.) must be standalone and not depend on any other non-core module. Running makedepend or the compiler with the -M option could provide some automatic dependency analysis.

@TerryE

This comment has been minimized.

Show comment
Hide comment
@TerryE

TerryE Jun 25, 2015

Collaborator

Ladder logic tricks like this are pretty standard but typically you would have the primary defines as a block and the dependency defines as a second block so in this you'd have something like

#if defined(MQTT) && !defined(SSL)
#  define SSL
#endif

This is the simplest and tidiest way to add such dependences

Collaborator

TerryE commented Jun 25, 2015

Ladder logic tricks like this are pretty standard but typically you would have the primary defines as a block and the dependency defines as a second block so in this you'd have something like

#if defined(MQTT) && !defined(SSL)
#  define SSL
#endif

This is the simplest and tidiest way to add such dependences

@confilemg

This comment has been minimized.

Show comment
Hide comment
@confilemg

confilemg commented Nov 21, 2015

+1

@marcelstoer

This comment has been minimized.

Show comment
Hide comment
@marcelstoer

marcelstoer Jul 4, 2016

Collaborator

@TerryE @devsaurus @jmattsson @pjsg This starts to bother me more and more and I'm quite convinced that it'll bite us medium-/long-term if we don't tackle this. When I created this issue nearly a year ago we had maybe 20 modules, now there are more than 40.

Initially I thought it would be enough if we had a simple dependency matrix. However, meanwhile we have different sorts of dependencies.

  • classical, module A requires module B: currently with #1379 it looks like you'd need the new tls module for net, http and mqtt if you want secure connections
  • exclusive shared resources: hw_timer used by PCM and others
  • individual modules intended to be used together: rtctime, rtcmem, SNTP
  • de-facto dependencies: #1321, "User is then free to choose how to generate this info. Preferably from rtctime.get()"

Do you think it'd be feasible to maintain a n*n matrix (n=number of modules) and give intersecting cells a specific color (see list above)? If further clarifications are required colorized cells may contain foot note references of course. Can you think of a more sensible approach?

Collaborator

marcelstoer commented Jul 4, 2016

@TerryE @devsaurus @jmattsson @pjsg This starts to bother me more and more and I'm quite convinced that it'll bite us medium-/long-term if we don't tackle this. When I created this issue nearly a year ago we had maybe 20 modules, now there are more than 40.

Initially I thought it would be enough if we had a simple dependency matrix. However, meanwhile we have different sorts of dependencies.

  • classical, module A requires module B: currently with #1379 it looks like you'd need the new tls module for net, http and mqtt if you want secure connections
  • exclusive shared resources: hw_timer used by PCM and others
  • individual modules intended to be used together: rtctime, rtcmem, SNTP
  • de-facto dependencies: #1321, "User is then free to choose how to generate this info. Preferably from rtctime.get()"

Do you think it'd be feasible to maintain a n*n matrix (n=number of modules) and give intersecting cells a specific color (see list above)? If further clarifications are required colorized cells may contain foot note references of course. Can you think of a more sensible approach?

@jmattsson

This comment has been minimized.

Show comment
Hide comment
@jmattsson

jmattsson Jul 4, 2016

Collaborator

Visualising dependencies is tricky. A colour-coded matrix is probably not a bad choice, but I am a bit concerned about its size when we get to e.g. 80 modules. Right now I can't think of anything smarter though, but I'll sleep on it and see if anything pops into my head.

Collaborator

jmattsson commented Jul 4, 2016

Visualising dependencies is tricky. A colour-coded matrix is probably not a bad choice, but I am a bit concerned about its size when we get to e.g. 80 modules. Right now I can't think of anything smarter though, but I'll sleep on it and see if anything pops into my head.

@pjsg

This comment has been minimized.

Show comment
Hide comment
@pjsg

pjsg Jul 4, 2016

Collaborator

There are two aspects to this -- one is to make the build work with arbitrary combinations of modules and the other is to make sure all the required functionality is present.

For the first, I think that the code ought to get it right (i.e. have the dependencies in some header file).

For the latter, the situation is a little more complex - but I don't think that there are many cases here. The TLS situation is the most obvious. I don't mind this being explicit.

For the sntp/rtctime case, I'm in favor of just treating these as hard dependencies and using the header file approach.

For the hw_timer case, this is already handled by the build system -- it is always only included if some module that needs it is included (this is the linker magic).

Presumably it would be possible to write a script that, given a list of lua files, extracted all the names that need to be defined and thereby could extract the required module list.

Collaborator

pjsg commented Jul 4, 2016

There are two aspects to this -- one is to make the build work with arbitrary combinations of modules and the other is to make sure all the required functionality is present.

For the first, I think that the code ought to get it right (i.e. have the dependencies in some header file).

For the latter, the situation is a little more complex - but I don't think that there are many cases here. The TLS situation is the most obvious. I don't mind this being explicit.

For the sntp/rtctime case, I'm in favor of just treating these as hard dependencies and using the header file approach.

For the hw_timer case, this is already handled by the build system -- it is always only included if some module that needs it is included (this is the linker magic).

Presumably it would be possible to write a script that, given a list of lua files, extracted all the names that need to be defined and thereby could extract the required module list.

@devsaurus

This comment has been minimized.

Show comment
Hide comment
@devsaurus

devsaurus Jul 4, 2016

Collaborator

A matrix is probably ok for an overview. Though I'm not a fan of encoding information exclusively with colours.
As a con: You'd have to iterate over the matrix again and again in case other modules need to be included due to recommended or must-have dependencies.

If you're looking for a way to guide the user wrt module selection, then the dependencies could also be defined as a set of statements per module. A structured format (JSON?) with dependency type, related module and a description:

  • u8g:
    • recommendation spi: some drivers require SPI interface
    • recommendation i2c: some drivers require I2C interface
  • pcm:
    • constraint pwm: no parallel operation
    • constraint perf: no parallel operation
  • ucg:
    • requirement spi: all drivers require SPI

A local script or nodemcu-build would run through the statements based on the selected modules and provided a summary about

  • missing/recommended modules
  • constraints for the user application
Collaborator

devsaurus commented Jul 4, 2016

A matrix is probably ok for an overview. Though I'm not a fan of encoding information exclusively with colours.
As a con: You'd have to iterate over the matrix again and again in case other modules need to be included due to recommended or must-have dependencies.

If you're looking for a way to guide the user wrt module selection, then the dependencies could also be defined as a set of statements per module. A structured format (JSON?) with dependency type, related module and a description:

  • u8g:
    • recommendation spi: some drivers require SPI interface
    • recommendation i2c: some drivers require I2C interface
  • pcm:
    • constraint pwm: no parallel operation
    • constraint perf: no parallel operation
  • ucg:
    • requirement spi: all drivers require SPI

A local script or nodemcu-build would run through the statements based on the selected modules and provided a summary about

  • missing/recommended modules
  • constraints for the user application
@marcelstoer

This comment has been minimized.

Show comment
Hide comment
@marcelstoer

marcelstoer Jul 4, 2016

Collaborator

I am a bit concerned about its size when we get to e.g. 80 modules.

Me too.

For the first, I think that the code ought to get it right (i.e. have the dependencies in some header file).

True but it'd still be nice if we could visualize this for our users. I anticipate questions like "Why is there module X in my firmware, I didn't enable it in user_modules.h?".

For the sntp/rtctime case, I'm in favor of just treating these as hard dependencies and using the header file approach.

AFAIU they're not, it's just that for the anticipated use cases you'll need all of them anyway. Correct Johny?

For the hw_timer case, this is already handled by the build system.

Yes, but I mean to document very clearly that you can't use pcm, perf and pwm at the same time.

...the dependencies could also be defined as a set of statements per module. A structured format (JSON?) with dependency type, related module and a description

I had been thinking about something like that as well. Haven't thought it through all the way, though. It'd most likely be easier to maintain than the matrix and it's got other pros:

  • could be injected into the respective module documentation page (similar to the current meta-info table) OR you do it the other way around: main in the docs and optionally generate an aggregated file
  • it scales better than the matrix
  • you could still generate a matrix from it, whereas the other way 'round would be a little more difficult
Collaborator

marcelstoer commented Jul 4, 2016

I am a bit concerned about its size when we get to e.g. 80 modules.

Me too.

For the first, I think that the code ought to get it right (i.e. have the dependencies in some header file).

True but it'd still be nice if we could visualize this for our users. I anticipate questions like "Why is there module X in my firmware, I didn't enable it in user_modules.h?".

For the sntp/rtctime case, I'm in favor of just treating these as hard dependencies and using the header file approach.

AFAIU they're not, it's just that for the anticipated use cases you'll need all of them anyway. Correct Johny?

For the hw_timer case, this is already handled by the build system.

Yes, but I mean to document very clearly that you can't use pcm, perf and pwm at the same time.

...the dependencies could also be defined as a set of statements per module. A structured format (JSON?) with dependency type, related module and a description

I had been thinking about something like that as well. Haven't thought it through all the way, though. It'd most likely be easier to maintain than the matrix and it's got other pros:

  • could be injected into the respective module documentation page (similar to the current meta-info table) OR you do it the other way around: main in the docs and optionally generate an aggregated file
  • it scales better than the matrix
  • you could still generate a matrix from it, whereas the other way 'round would be a little more difficult
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment