Build tries to build all .c files of a library which might not have all the dependent headers available

[masha256]

When using the find_arduino_library/link_arduino_library to include a library from my arduino sketchbook location, it looks like the build is trying to grab all .c files that are in the library. Sometimes libraries include tools or example code that might depend on headers that are not on the system, leading to a failed build, even though the code/headers for the library itself are just fine.

For example, I am using the Adafruit_GFX library that includes a font utility that depends on some freetype libs. The actual Adafruit_GFX lib does not depend on the font utility, its just there in case you want to use it, but since the utility cannot be built on my system (freetype headers unavailable) it causes my project build to fail:

[ 34%] Building C object CMakeFiles/Skywriter.dir/Users/machado/Dropbox/Arduino/libraries/Adafruit_GFX/fontconvert/fontconvert.c.obj
/Applications/arduino.app/Contents/Java/hardware/tools/avr/bin/avr-gcc  -I/Users/machado/Dropbox/Arduino/libraries/Adafruit_GFX -I/Users/machado/Dropbox/Arduino/libraries/Adafruit_GFX/Fonts -I/Users/machado/Dropbox/Arduino/libraries/Adafruit_NeoMatrix -I/Users/machado/Dropbox/Arduino/libraries/Adafruit_NeoPixel -I/Users/machado/Dropbox/Arduino/libraries/Adafruit_SSD1306 -I/Users/machado/Dropbox/Arduino/libraries/Arduino-SerialCommand -I/Users/machado/Dropbox/Arduino/libraries/ArduinoNunchuk -I/Users/machado/Dropbox/Arduino/libraries/EnableInterrupt -I/Users/machado/Dropbox/Arduino/libraries/EnableInterrupt/utility -I/Users/machado/Dropbox/Arduino/libraries/FastAPI_LED -I/Users/machado/Dropbox/Arduino/libraries/Keypad/src -I/Users/machado/Dropbox/Arduino/libraries/MFRC522/src -I/Users/machado/Dropbox/Arduino/libraries/MPU9250 -I/Users/machado/Dropbox/Arduino/libraries/MQTT/src -I/Users/machado/Dropbox/Arduino/libraries/MQTT/src/lib -I/Users/machado/Dropbox/Arduino/libraries/PinChangeInterrupt/src -I/Users/machado/Dropbox/Arduino/libraries/Skywriter -I/Users/machado/Dropbox/Arduino/libraries/TimerOne -I/Users/machado/Dropbox/Arduino/libraries/TimerOne/config -I/Users/machado/Dropbox/Arduino/libraries/Zumo32U4 -I/Users/machado/Dropbox/Arduino/libraries/grbl -I/Applications/arduino.app/Contents/Java/hardware/arduino/avr/cores/arduino -I/Applications/arduino.app/Contents/Java/hardware/arduino/avr/variants/standard -I/Applications/arduino.app/Contents/Java/hardware/arduino/avr/libraries/SPI/src -I/Applications/arduino.app/Contents/Java/hardware/arduino/avr/libraries/Wire/src -I/Applications/arduino.app/Contents/Java/hardware/arduino/avr/libraries/eeprom/src  -g   -o CMakeFiles/Skywriter.dir/Users/machado/Dropbox/Arduino/libraries/Adafruit_GFX/fontconvert/fontconvert.c.obj   -c /Users/machado/Dropbox/Arduino/libraries/Adafruit_GFX/fontconvert/fontconvert.c
/Users/machado/Dropbox/Arduino/libraries/Adafruit_GFX/fontconvert/fontconvert.c:23:22: fatal error: ft2build.h: No such file or directory
 #include <ft2build.h>
                      ^
compilation terminated.
make[2]: *** [CMakeFiles/Skywriter.dir/Users/machado/Dropbox/Arduino/libraries/Adafruit_GFX/fontconvert/fontconvert.c.obj] Error 1
make[1]: *** [CMakeFiles/Skywriter.dir/all] Error 2
make: *** [all] Error 2

[MrPointer]

@machadolab Well here's the thing:
As part of the support for 3rd Party libraries and the ability to treat them as "first-class" Arduino Libraries, I had to make the framework search for sources (and headers) recursively under the library's root directory, unlike the older version which searched only under the 'src' sub-directory (That's how built-in libs are structured).

Why am I telling you this? In order to truly find only sources that are relevant to the library itself, I had to exclude any sources not matching a certain pattern. In the case of the Adafruit_NeoPixel library which I've tested against, it's only the examples directory that should be excluded.
However, with this new type of library, it seems that fonts should be excluded too, and that's fine, but it leads me to think that there might be countless directory names which should be excluded, and the framework could never know that in advance.
Besides, excluding a directory to support one library could break the build for another, if the exclude pattern will name these directories explicitly.

It'll take some time to solve this - Any help or ideas are welcomed.

[masha256]

Interesting. Yeah, I thought it was going to be a tricky one. Do you have any idea what the official arduino IDE does? They seem to handle all situations...

[MrPointer]

Nope, absolutely no idea.
It'll be impossible for me to dig into their code right now to extract it, too much effort and time to spend...

[masha256]

Found this that might help: https://github.com/arduino/Arduino/wiki/Build-Process

What was the motivation for during the recursive code search in the first place?

[MrPointer]

Thanks, however, there's little I can learn from these specifications regarding our issue.
In fact, the Library Directory Structure/Layout Specifications specify that we're doing everything right in the framework.
Maybe, just maybe, the Arduino IDE reads all of your project's/sketches include files and only adds what it finds besides the hidden ones it always adds when building, such as Arduino.h... I hope that's not the case as it would make the framework extremely messy in that prospect.

Edit: As for why the search is done recursively - Some of the built-in libraries contain the utility sub-directory where additional sources reside. It seems reasonable that other names could exist as well, so in order to get all of these sources, recursive search has been used.

Edit 2: I think I've found a solution!
The Library Directory Structure/Layout Specifications specify that for newer libraries, sources under the src directory and all of its sub-directories are included, but for older libraries, only sources under the root directory are added.
It gives the framework an opportunity to include only the basic sources of a library, whereas additional sources would be supplied manually, maybe using an additional "utility" function to ease this burden on the user.

[masha256]

Excellent!