Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include global UserSettings.h file #15

Open
NicoHood opened this issue Sep 24, 2015 · 32 comments

Comments

@NicoHood
Copy link

commented Sep 24, 2015

If we could have a global UserSettings.h file which is included at the top of the Arduino.h file we could:

  • Let the user add custom settings to an .ino file for libraries and the core itself
  • This way we can easily overwrite things like RX_BUFFER_SIZE as well. (without a boards.txt mod)
  • We could also add options to disable the USB-Core and other components if not needed
  • We could add an option to disable the SerialEvent function. You can keep it by default for compatibility, but set a few parameters to tweak the Arduino code and save some flash
  • We could use this for libraries, such as PinChangeInterrupt to effectively make use of direct port access with pre defined pins. This also works for other libraries. A known pin at runtime is way faster and also saves a lot of time.
  • Global debug options could be enabled this way.
  • Many more things I can imagine of and you as well.

This requires every file to see the UserSettings.h file, it needs to be included in the Arduino.h file and it needs to be placed in the .ino folder, and if not it should be created as an empty file. Also make sure if you delete it, an empty dummy will be added again.

Imported from: arduino/Arduino#3757
Also fixes (and maybe more): arduino/Arduino#3717 arduino/Arduino#3635

NicoHood added a commit to NicoHood/Arduino that referenced this issue Sep 24, 2015

@stevemarple

This comment has been minimized.

Copy link

commented Sep 30, 2015

I think I have a commit which successfully allows macros (#defines) across all compilation units. My approach is to test for a build_props.txt file in the sketch folder. It has the same key=value format found in the board files. Global macros can be defined with

compiler.c.extra_flags=-D NDEBUG
compiler.cpp.extra_flags=-D NDEBUG -D TESTLIBRARY_BUFSIZE=100

Followers of the DRY principle can take advantage of the property expansion method in arduino-builder:

compiler.c.extra_flags=-D NDEBUG
compiler.cpp.extra_flags={compiler.c.extra_flags} -D TESTLIBRARY_BUFSIZE=100

A global UserSettings.h file couldcan be included across all compilation units:

compiler.c.extra_flags=-include {build.source.path}/UserSettings.h
compiler.cpp.extra_flags={compiler.c.extra_flags}

A property defining the sketch folder appears to be missing (#24) requiring the path to UserSettings.h to be hard-coded.
[Code snippet above updated now that build.source.path is correct.]

I'm not ready to submit a pull request since I have not added any unit tests but I would appreciate feedback and testing to know if this is useful. The commit can be found at stevemarple@a5fc12b

@cmaglie

This comment has been minimized.

Copy link
Member

commented Oct 5, 2015

The idea of having a specially-named .h file in the sketch be included during compilation of the core and libraries is a long-standing one: https://code.google.com/p/arduino/issues/detail?id=27

I've some concerns about opening this possibility to developers, I think that @damellis has very well explained the reasons here:

It may or may not be a good idea but it is, I think, a big change and one that should be considered carefully on the developers list. This would open up lots of potential tweaks to the core and libraries, greatly increasing the effective API of those files (i.e. previously-internal details of the files that people will now be able to rely on and therefore will become more difficult to change). It will also add a lot of potential complexity / features that people may see in sketches they get from others. In addition, by not having a way to provide pre-processor macros / compile-time constants to the core and libraries, I think we've forced people to find easier-to-use alternatives in the form of run-time configuration (which is more consistent with other parts of the Arduino API). With this option, I worry that people will instead take the path-of-least-resistant and use compile-time configuration, complicating things for users.

In 4 years I've seen cases where this rationale applies for true, the most significant are:

  • Problem: HardwareSerial1/2/3/4 objects always takes memory even if not used -> Path-of-least-resistant solution: Add defines to disable/enable HardwareSerial -> Current solution: Put objects in different compilation units, so they are automatically optimized by the compiler.
  • Problem: extending the core's USB-HID classes takes too much flash/ram if all the improvements are taken together (arduino/Arduino#1803) -> Path-of-least-resistant solution: add core's defines to selectively disable them -> Current solution (still work-in-progress): Pluggable USB/HID
  • Problem: some configuration values may be optimized saving some bytes -> Path-of-least-resistant solution: add defines for the configuration values -> Better solution: Sacrifice a bunch of bytes and make them runtime-values so the user doesn't have to remember yet another #define
  • Defines could be "abused" and used even when they are not needed, with the result to provide a more complex API for the end user. The following is an emblematic example: arduino/Arduino#1808

To conclude, is this feature really worth it? the things that are absolutely not possible without #defines and #ifdefs are really compelling for end users?

@ffissore

This comment has been minimized.

Copy link
Contributor

commented Oct 5, 2015

I agree with all these reasons and experience shows that better/easier to maintain solutions have been found without relying on #define.
However, there are cases in which a lib dev is trying to understand which problem one of her user is having: enabling -DDEBUG in one place could allow a non-dev to help a dev
What about yet-another property in library.properties: debug=true ? If present, -DDEBUG will be added to library compilation command lines

@NicoHood

This comment has been minimized.

Copy link
Author

commented Oct 5, 2015

Nah. I really do need this file inclusion, not only a library.properties. I need a setting which can be added to the sketch as additional .h file tab. Its not just about enabling debug. If i'd want this I can just put it in the librarys header itself.

Its about the time after you release stuff and the user wants to customize stuff. You dont need to tell him which source file he needs to find and what to edit. And the next time he needs a different setting and needs to search it again. This way we can have settings per sketch.

I suggest two options at the same time:

  • Add a UserSetting.h file which is included by Arduino.h. This way we can add #defines to disable different hardwareserial etc (see list above, thx christian)
  • Add the sketch folder to the general inclusion path. This way the library maintainer can force the user to add this file to its sketch. This is needed for more complex or improved libraries. This should normally solved with the option above, but this way you can force the user to use it and no other libraries/core will see this file if it has a unique name. This implementation is required to make the solution above work anyways.

You could also do a separate CoreSettings.h (which is included by the Arduino.h file) and a LibrarySettings.h file (which may be included by the library). A Wrapper for including both could be to just use #include UserSettings.h to just include both .h files.

Having 2 separate Settings files would be required for deactivating the USB-Core (how it is right now). In the CoreSettings.h you just #undef USBCON and in the LibrarySettings you do not do this, but maybe some other settings. This way one could write a whole new USB-Core, even without using the Pluggable stuff. Same goes for Serial deactivation etc.

@stevemarple

This comment has been minimized.

Copy link

commented Oct 5, 2015

I agree that in some circumstances #define's could be considered a lazy method, for instance to define internal buffer sizes. An alternative I use in my MicroNMEA and IniFile libraries is to make the
user pass a pointer to the buffer and its size. I'm not convinced it's easier to use, and I'm not sure that requiring novice users pass pointers is a good idea either. At least in this case an alternative
exists. There are things inside libraries which could be genuinely useful in some circumstances but consideration of memory requirements in common usage prevents it. This affects cases which would need additional member variables as I don't think that compiler/linker optimizations can recover
that memory. However conditional compilation would prevent its allocation.

There are many useful contributed libraries which require users to modify the library header files to comment/uncomment the appropriate #define. This is terrible practice and I suspect it has continued because the Arduino IDE lacks the ability to define sketch-specific macros. Adding a method to globally define macros would make using these libraries much less painful.

The effect of inserting a library.properties.debug property is limited and doesn't solve the problem I'm trying to address. Moreover it doesn't seem useful unless the option is exposed by the IDE - the developer would need to restart the IDE just to enable debugging. Changing a build_options.txt file would however be easy and it does correctly force a complete recompilation.

Whilst I accept you must consider the environment for novice users (that was me a few years ago) the Arduino ecosystem has grown. As the existence of the Arduino at Heart program shows some of that growth is towards more advanced/industrial users. Please consider their needs too.

Would a new pull request which only read build_options.txt when build.custom_sketch_properties=true be accepted? That would discourage novice use whilst enabling more experienced programmers greater customisation in the way every other common IDE allows.

@NicoHood

This comment has been minimized.

Copy link
Author

commented Oct 6, 2015

Am I think we are talking about two different features here.

  • Including a header file as global config
  • Passing custom compiler flags to the IDE.

I think both cases are really helpful. I really like your idea, but you should probably discuss this in the PR you opened. I can imagine things like: preserving some registers from being used for very special libraries etc. If this makes sense in all cases is out of question here.

However I'd rather put those compiler flags in the libraries.properties file. The dot_a_linkage setting is placed there as well, and why do we need another file? Also I think an option like build.custom_sketch_properties=true is not needed. Because if you dont want to use it, just dont (do not add this file). As simple as that.

@stevemarple

This comment has been minimized.

Copy link

commented Oct 6, 2015

@NicoHood Pull request #29 is a generalisation of this feature request. It enables a global config file to be included with gcc's -include command line option or by using the -D macro=definition option macros can be defined without an extra file. Your request is not the first for this feature but as far as I could see it is the first in relation to arduino-builder, hence the discussion here.
I took the approach I did since arduino-builder already accepts 'customisations' at a platform and board level, and from the command line. Tweaking the build in this way doesn't require any changes to platform or board recipes, or to https://github.com/arduino/Arduino.

@NicoHood

This comment has been minimized.

Copy link
Author

commented Oct 6, 2015

I was not aware that you can add -include as optional parameter. This sounds pretty nice and partly solves the issue here I think.

Is there a way to point to the sketch folder directly? Is there a way to ignore the include if no file was found? I am still not 100% if this solves the whole issue, but sure it enables us a wide range of new possibilities.

And finally how can I test this build (your PR)?

@stevemarple

This comment has been minimized.

Copy link

commented Oct 6, 2015

@NicoHood Instructions are in my first post in this thread, including how to reference the sketch source folder.

You need a very recent version of the Arduino IDE (nightly build or built from the latest sources). Replace the arduino-builder executable with one built from from dev-global-macros branch, https://github.com/stevemarple/arduino-builder/tree/dev-global-macros. Alternatively download a 64bit version from https://www.dropbox.com/sh/oypoojbqodwkc05/AACZwX4uSJdoH5AcMgFMq88ea?dl=0

@ffissore ffissore modified the milestones: 1.0.1, 1.0.2, 1.0.3, 1.0.4 Oct 27, 2015

@xcvista

This comment has been minimized.

Copy link

commented Jun 29, 2016

Or maybe in an less intrusive way, convert all simple #defines (simple names, the value is a literal of any kind or the default) in main sketch file into -Ds across all files? No new files introduced, same effect.

@spiderkeys

This comment has been minimized.

Copy link

commented Jun 30, 2016

I would also like to see a way to do this. Just a simple way to add -D options when I call arduino-builder would save me a lot of manual pre-build work.

@romansavrulin

This comment has been minimized.

Copy link

commented Feb 15, 2018

Hello, guys!

Is there any progress on adding global defines to make core or libraries configurable? What solution can be utilized to make it possible?

@ConnorRigby

This comment has been minimized.

Copy link

commented May 31, 2018

Also very interested in this. I have a repository that can build for multiple different targets. Having a -DMYTarget=1 would really be so much simpler than what i currently have to do.I have a hard time understanding why this is so frowned upon here, almost every other build system supports this to some extent.

UPDATE:

fwiw i was able to do this super cool hack:
-prefs="runtime.ide.version=10600 -D_MY_BOARD_ID=0"
This is so silly.

@stevemarple

This comment has been minimized.

Copy link

commented Jun 1, 2018

In the end I worked round this by defining my own boards.txt file with a menu option to include custom_include.h. I needed a boards file anyway to support my Calunium ATmega128P clone so it was the route of least resistance for me. I'd still like to see support added generally.

@robjen

This comment has been minimized.

Copy link

commented Aug 10, 2018

really could do with this feature, injecting #defines into a library at compile time allow me to write code that is stripped back to optimal, saving space and allowing for things like endianess without switch/if statments based on variables passed at 'begin' (extra code and kills performance on every access)

and this sooo nearly works .... I notice from looking at building other simple project that the environment is pulling ALL files into the temp area including the copies of the libraries and the sketch but there is a preprocessing step that fails on the project first build (it seems to build a library in place at the library path first) ....

so I did the following

wrote my own library put it in the proper place
wrote a stetch which in its directory contains mylib_config.h
library has a #include "../sketch/mylib_config.h"

  1. compile sketch and it fails missing file in library also note none of the libraries were copied ;-(
  2. comment out the #include in my library and build it once
    works fine (my library has defaults for anything not defined)
  3. now add back my #include line and build again
    now the build is using the cached files in the temp area (even copied over my mod from the library area) and the config files has its desired effect on the library now

it seems to me ... if the environment is going to cache the files in temp anyway ... why does it not do that step FIRST and pre-compile the library with that copy there ? since its going to be doing everything else in there until I close the envirnment. If the copy was made first then everything would work as people desire

@ricardojlrufino

This comment has been minimized.

Copy link

commented Aug 10, 2018

Since IDE does not support the concept of PROJECTS nor the definition of environment variables. That would be the least to do.

Most IDE has a configuration file to specify this type of configuration.

@xcvista

This comment has been minimized.

Copy link

commented Aug 10, 2018

@ricardojlrufino It might be a better idea to keep the name "Project" out of Arduino, and try to achieve the goals by tweaking the existing sketch format.

For the #define's, my idea would be, as stated above, convert all simple #define's in the main sketch (.ino) file into -D for all calls to the compiler.

For other IDE settings, I would prefer #pragma arduino in main sketch file: whatever comes after this in the line is treated as part of boards.txt for this sketch.

@PaulStoffregen

This comment has been minimized.

Copy link

commented Aug 13, 2018

Everyone recently commenting on this issue should re-read Cristian's comment posted on Oct 5, 2015.

@ConnorRigby

This comment has been minimized.

Copy link

commented Aug 13, 2018

here is that post. That solution would be fine had it been merged.

@sleemanj

This comment has been minimized.

Copy link

commented Aug 26, 2018

When avr-gcc 5.0 arrives in Arduino land then the __has_include macro will be available.

That would mean that a UserSettings.h optionally being provided and with no impact if not provided should be as simple as adding -DUSER_SETTINGS_FILE=\"/path/to/UserSettings.h\" and then library (or core) authors can use....

#if defined(__has_include) &&  __has_include(USER_SETTINGS_FILE)
    #include USER_SETTINGS_FILE
#endif

For alternative core developers it will be trivially simple to add this to their core, but without the Arduino core itself doing so there risks to be a proliferation of differing standards "UserSettings.h", "Configuration.h", USER_SETTINGS_FILE, LIBRARY_SETTINGS....

Indeed, if even the sketch source folder is available to the GCC preprocessor, it would probably be possible to construct include file paths in macros with some magic somehow, the quote marks necessary for #include are a bit of an issue it seems but I think a correct incantation of #define magic could be found that would allow constructing arbitrary candidate include files if the sketch folder is exposed and to include them if they exist, or not if they don't.

My interest is mostly in tiny processors where every byte counts. Currently I'm writing a library for sleep management which for some cases uses the WDT, but not all use-cases employ it, for example sleeping "forever" doesn't need the WDT, sleeping "deeply" does, sleeping "idle" doesn't necessarily.

The compiler has no way to optimize out an unused ISR like the WDT, so even if the library-user never calls thing_that_needs_to_use_the_wdt() the WDT ISR is still included. The only way (as far as I know) to not have that WDT ISR in, is to not compile it at all. On a chip with 1k flash even 50 bytes of unused ISR is hugely wasteful.

@matthijskooijman

This comment has been minimized.

Copy link
Collaborator

commented Aug 26, 2018

The compiler has no way to optimize out an unused ISR like the WDT

Slightly off-topic, but there is a trick with using a .a archive file for linking, which is used by HardwareSerial. See this comment (which only applies for files linked through .a files, but you can specify some property in library.properties that makes this happen). If you have additional questions, feel free to drop me a mail, rather than continuing this as an off-topic discussion here.

@stevemarple

This comment has been minimized.

Copy link

commented Aug 26, 2018

@sleemanj Core developers can add the desired functionality now, no need for avr-gcc 5.0. See #15 (comment) and https://github.com/stevemarple/Calunium/blob/master/software/arduino-1.6/calunium/avr/boards.txt

@sleemanj

This comment has been minimized.

Copy link

commented Aug 26, 2018

@stevemarple doesn't that (-include) cause an error if the file does not exist? It does in my avr-gcc (4.8.2)

$ avr-gcc -include foo.h test.c
cc1: fatal error: foo.h: No such file or directory

With __has_include no error would happen if the file doesn't exist.

Edit: I see you have set that up to only be used if you select the appropriate menu option, in which case, yeah you could do that certainly, accepting that if the user picks that option and attempts to compile without actually having the required file created then they will get an error.

@PaulStoffregen

This comment has been minimized.

Copy link

commented Aug 26, 2018

Again, everyone here should re-read Cristian's comment near the top of this issue, written Oct 5, 2015.

Arduino has historically rejected this idea (in a variety of forms) not based on technical issues with its implementation, but based on the massive effective expansion of the API it would have, and the anticipated long-term effect that could have on the Arduino ecosystem.

Focusing on the technical implementation details is missing the "big picture".

@sleemanj

This comment has been minimized.

Copy link

commented Aug 26, 2018

Focusing on the technical implementation details is missing the "big picture".

Which is, that this is probably going to happen anyway (and as above with @stevemarple 's core already is) with or without Arduino officially endorsing it with a one-true-way (of having compile-time configuration passed into cores and libraries).

I think it would be better with a one-true-way personally, rather than having each core doing their own thing.

This ticket is still open, so right now it has not been rejected flatly as WONTFIX, I think it's reasonable to continue discussion at least until it is, and part of that is technical detail.

sleemanj added a commit to sleemanj/SimpleSleep that referenced this issue Aug 28, 2018

Refactor so that the WDT can be optimised out if it's not required.
Thanks to @matthijskooijman who pointed out the possibility in
arduino/arduino-builder#15 (comment)

With LTO turned on the ATTiny13 will now do a simple forever sleep in about 30 bytes.
@EtienneGameSeed

This comment has been minimized.

Copy link

commented Nov 21, 2018

So finally is there any way to do this ?
I need this too, in order to customize the audio features of the Teensy Core.
I've tried to add -prefs="build.extra_flags=-DAUDIO_INPUT_NB_CHANNEL=2" to the arduino builder command line, but it doesn't work.
arduino_builder seems to understand the command and takes it into account, it adds it to the customBuildProperties in build.options.json, but the compiler still throws an error like "error: 'AUDIO_INPUT_NB_CHANNELS' was not declared in this scope"
Have I missed something ?

@facchinm

This comment has been minimized.

Copy link
Member

commented Nov 21, 2018

@EtienneGameSeed it looks like you are missing a minus sign.
The command should read -prefs="build.extra_flags=-DAUDIO_INPUT_NB_CHANNEL=2".

@EtienneGameSeed

This comment has been minimized.

Copy link

commented Nov 21, 2018

Thanks, sorry it's a typo error that was in my message (I corrected it).
What I have in the build.options.json file is :
"customBuildProperties": "build.extra_flags=-DAUDIO_INPUT_NB_CHANNEL=2"
But it doesn't work...

@facchinm

This comment has been minimized.

Copy link
Member

commented Nov 21, 2018

Which board/IDE version/builder version are you targeting? I just tested that flag and it works as intended, but it also overrides the existing field (eg, if build.extra_flags was already set at -Dsomething this disappears) so you must find an unused hook.

@EtienneGameSeed

This comment has been minimized.

Copy link

commented Nov 21, 2018

I'm using IDE 1.8.7 and TeensyDuino 1.44.
I found why it didn't work and corrected it.
It was because there was no "build.extra_flags" in Teensy's platform.txt.
I've added it and now it works !
Thanks a lot Martino.

Here's the corrected file : platform.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.