possible include detection bug

[deanm1278]

Hello! I am working on an Arduino package for blackfin+ DSP CPUs and am running into a strange issue with either the builder or the IDE.

Running with IDE 1.8.4 or 1.8.5 generates an error because it cannot correctly detect the used libraries:

Arduino: 1.8.4 (Windows 10), Board: "Feather Blackfin"

C:\Program Files (x86)\Arduino\arduino-builder -dump-prefs -logger=machine -hardware C:\Program Files (x86)\Arduino\hardware -hardware C:\Users\Dean\AppData\Local\Arduino15\packages -hardware C:\Users\Dean\Documents\Arduino\hardware -tools C:\Program Files (x86)\Arduino\tools-builder -tools C:\Program Files (x86)\Arduino\hardware\tools\avr -tools C:\Users\Dean\AppData\Local\Arduino15\packages -built-in-libraries C:\Program Files (x86)\Arduino\libraries -libraries C:\Users\Dean\Documents\Arduino\libraries -fqbn=Dean:bfin:feather_bfin -vid-pid=0X239A_0X001B -ide-version=10804 -build-path C:\Users\Dean\AppData\Local\Temp\arduino_build_854378 -warnings=all -build-cache C:\Users\Dean\AppData\Local\Temp\arduino_cache_165494 -prefs=build.warn_data_percentage=75 -verbose C:\Users\Dean\Documents\Arduino\bfin-test\bfin-test.ino
C:\Program Files (x86)\Arduino\arduino-builder -compile -logger=machine -hardware C:\Program Files (x86)\Arduino\hardware -hardware C:\Users\Dean\AppData\Local\Arduino15\packages -hardware C:\Users\Dean\Documents\Arduino\hardware -tools C:\Program Files (x86)\Arduino\tools-builder -tools C:\Program Files (x86)\Arduino\hardware\tools\avr -tools C:\Users\Dean\AppData\Local\Arduino15\packages -built-in-libraries C:\Program Files (x86)\Arduino\libraries -libraries C:\Users\Dean\Documents\Arduino\libraries -fqbn=Dean:bfin:feather_bfin -vid-pid=0X239A_0X001B -ide-version=10804 -build-path C:\Users\Dean\AppData\Local\Temp\arduino_build_854378 -warnings=all -build-cache C:\Users\Dean\AppData\Local\Temp\arduino_cache_165494 -prefs=build.warn_data_percentage=75 -verbose C:\Users\Dean\Documents\Arduino\bfin-test\bfin-test.ino
Using board 'feather_bfin' from platform in folder: C:\Users\Dean\Documents\Arduino\hardware\Dean\bfin
Using core 'arduino' from platform in folder: C:\Users\Dean\Documents\Arduino\hardware\Dean\bfin
Detecting libraries used...
"C:\cygwin64\bfin-gnu-new\buildscript\bfin-elf-win\bin\bfin-elf-g++" -mcpu=bf706 -c -w -std=gnu++11 -g3 -fno-rtti -ffunction-sections -fdata-sections -fno-threadsafe-statics -nostdlib -Os --param max-inline-insns-single=500  -w -x c++ -E -CC -DF_CPU=48000000L -DARDUINO=10804 -DARDUINO_FEATHER_BFIN -DARDUINO_ARCH_BFIN  -DARDUINO_BFIN -D__ADSPBF706__ "-IC:\cygwin64\bfin-gnu-new\include" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\cores\arduino" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\variants\feather_bfin" "C:\Users\Dean\AppData\Local\Temp\arduino_build_854378\sketch\bfin-test.ino.cpp" -o "nul"
"C:\cygwin64\bfin-gnu-new\buildscript\bfin-elf-win\bin\bfin-elf-g++" -mcpu=bf706 -c -w -std=gnu++11 -g3 -fno-rtti -ffunction-sections -fdata-sections -fno-threadsafe-statics -nostdlib -Os --param max-inline-insns-single=500  -w -x c++ -E -CC -DF_CPU=48000000L -DARDUINO=10804 -DARDUINO_FEATHER_BFIN -DARDUINO_ARCH_BFIN  -DARDUINO_BFIN -D__ADSPBF706__ "-IC:\cygwin64\bfin-gnu-new\include" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\cores\arduino" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\variants\feather_bfin" "C:\Users\Dean\AppData\Local\Temp\arduino_build_854378\sketch\bfin-test.ino.cpp" -o "C:\Users\Dean\AppData\Local\Temp\arduino_build_854378\preproc\ctags_target_for_gcc_minus_e.cpp"
C:\Users\Dean\Documents\Arduino\bfin-test\bfin-test.ino:1:10: fatal error: SPI.h: No such file or directory
 #include <SPI.h>
          ^~~~~~~
compilation terminated.
exit status 1
Error compiling for board Feather Blackfin.

But what's strange is that file does exist, the builder compiles the sketch successfully under Arduino IDE 1.6.5 AND when I run the builder from the command line it correctly detects the libraries and compiles. The debug output from the command line is pasted below. I am using gcc version bfin-elf-gcc (GCC) 8.0.0 20171013 (experimental)

Note 2 things with the below output:
1. I had to

C:\Users\Dean>mklink /J C:\ProgramFiles "C:\Program Files (x86)"
Junction created for C:\ProgramFiles <<===>> C:\Program Files (x86)

Because windows command line can't do spaces in paths. Other packages work fine in the IDE.

2. I cut off the remainder of the output after it detected the libraries and started successfully compiling files.

C:\Users\Dean>C:\ProgramFiles\Arduino\arduino-builder -compile -logger=machine -hardware C:\ProgramFiles\Arduino\hardware -hardware C:\Users\Dean\AppData\Local\Arduino15\packages -hardware C:\Users\Dean\Documents\Arduino\hardware -tools C:\ProgramFiles\Arduino\tools-builder -tools C:\ProgramFiles\Arduino\hardware\tools\avr -tools C:\Users\Dean\AppData\Local\Arduino15\packages -built-in-libraries C:\ProgramFiles\Arduino\libraries -libraries C:\Users\Dean\Documents\Arduino\libraries -fqbn=Dean:bfin:feather_bfin -vid-pid=0X239A_0X001B -ide-version=10804 -build-path C:\Users\Dean\AppData\Local\Temp\arduino_build_854378 -warnings=all -build-cache C:\Users\Dean\AppData\Local\Temp\arduino_cache_165494 -prefs=build.warn_data_percentage=75 -verbose -debug-level=10 C:\Users\Dean\Documents\Arduino\bfin-test\bfin-test.ino
===debug ||| Ts: {0} - Running: {1} ||| [1522205382 GenerateBuildPathIfMissing]
===info ||| Progress {0} ||| [0.00]
===debug ||| Ts: {0} - Running: {1} ||| [1522205382 EnsureBuildPathExists]
===info ||| Progress {0} ||| [2.94]
===debug ||| Ts: {0} - Running: {1} ||| [1522205382 ContainerSetupHardwareToolsLibsSketchAndProps]
===info ||| Progress {0} ||| [5.88]
===debug ||| Ts: {0} - Running: {1} ||| [1522205382 AddAdditionalEntriesToContext]
===debug ||| Ts: {0} - Running: {1} ||| [1522205382 FailIfBuildPathEqualsSketchPath]
===debug ||| Ts: {0} - Running: {1} ||| [1522205382 HardwareLoader]
===debug ||| Ts: {0} - Running: {1} ||| [1522205382 PlatformKeysRewriteLoader]
===debug ||| Ts: {0} - Running: {1} ||| [1522205382 RewriteHardwareKeys]
===debug ||| Ts: {0} - Running: {1} ||| [1522205382 ToolsLoader]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 TargetBoardResolver]
===info ||| Using board '{0}' from platform in folder: {1} ||| [feather_bfin C%3A%5CUsers%5CDean%5CDocuments%5CArduino%5Chardware%5CDean%5Cbfin]
===info ||| Using core '{0}' from platform in folder: {1} ||| [arduino C%3A%5CUsers%5CDean%5CDocuments%5CArduino%5Chardware%5CDean%5Cbfin]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 AddBuildBoardPropertyIfMissing]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 LibrariesLoader]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 SketchLoader]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 SetupBuildProperties]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 LoadVIDPIDSpecificProperties]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 SetCustomBuildProperties]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 AddMissingBuildPropertiesFromParentPlatformTxtFiles]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 ContainerBuildOptions]
===info ||| Progress {0} ||| [8.82]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 CreateBuildOptionsMap]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 LoadPreviousBuildOptionsMap]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 WipeoutBuildPathIfBuildOptionsChanged]
===info ||| Build options changed, rebuilding all ||| []
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 StoreBuildOptionsMap]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 WarnAboutPlatformRewrites]
===info ||| Progress {0} ||| [11.76]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 RecipeByPrefixSuffixRunner]
===info ||| Progress {0} ||| [14.71]
===debug ||| Looking for recipes like {0}*{1} ||| [recipe.hooks.prebuild .pattern]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 ContainerMergeCopySketchFiles]
===info ||| Progress {0} ||| [17.65]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 SketchSourceMerger]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 SketchSaver]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 AdditionalSketchFilesCopier]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 loggerAction]
===info ||| Progress {0} ||| [20.59]
===info ||| Detecting libraries used... ||| []
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 ContainerFindIncludes]
===info ||| Progress {0} ||| [23.53]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 GCCPreprocRunnerForDiscoveringIncludes]
"C:\cygwin64\bfin-gnu-new\buildscript\bfin-elf-win\bin\bfin-elf-g++" -mcpu=bf706 -c -w -std=gnu++11 -g3 -fno-rtti -ffunction-sections -fdata-sections -fno-threadsafe-statics -nostdlib -Os --param max-inline-insns-single=500  -w -x c++ -E -CC -DF_CPU=48000000L -DARDUINO=10804 -DARDUINO_FEATHER_BFIN -DARDUINO_ARCH_BFIN  -DARDUINO_BFIN -D__ADSPBF706__ "-IC:\cygwin64\bfin-gnu-new\include" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\cores\arduino" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\variants\feather_bfin" "C:\Users\Dean\AppData\Local\Temp\arduino_build_854378\sketch\bfin-test.ino.cpp" -o "nul"
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 IncludesFinderWithRegExp]
===debug ||| Ts: {0} - Running: {1} ||| [1522205383 GCCPreprocRunnerForDiscoveringIncludes]
"C:\cygwin64\bfin-gnu-new\buildscript\bfin-elf-win\bin\bfin-elf-g++" -mcpu=bf706 -c -w -std=gnu++11 -g3 -fno-rtti -ffunction-sections -fdata-sections -fno-threadsafe-statics -nostdlib -Os --param max-inline-insns-single=500  -w -x c++ -E -CC -DF_CPU=48000000L -DARDUINO=10804 -DARDUINO_FEATHER_BFIN -DARDUINO_ARCH_BFIN  -DARDUINO_BFIN -D__ADSPBF706__ "-IC:\cygwin64\bfin-gnu-new\include" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\cores\arduino" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\variants\feather_bfin" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\libraries\SPI" "C:\Users\Dean\AppData\Local\Temp\arduino_build_854378\sketch\bfin-test.ino.cpp" -o "nul"
===debug ||| Ts: {0} - Running: {1} ||| [1522205384 IncludesFinderWithRegExp]
===debug ||| Ts: {0} - Running: {1} ||| [1522205384 GCCPreprocRunnerForDiscoveringIncludes]
"C:\cygwin64\bfin-gnu-new\buildscript\bfin-elf-win\bin\bfin-elf-g++" -mcpu=bf706 -c -w -std=gnu++11 -g3 -fno-rtti -ffunction-sections -fdata-sections -fno-threadsafe-statics -nostdlib -Os --param max-inline-insns-single=500  -w -x c++ -E -CC -DF_CPU=48000000L -DARDUINO=10804 -DARDUINO_FEATHER_BFIN -DARDUINO_ARCH_BFIN  -DARDUINO_BFIN -D__ADSPBF706__ "-IC:\cygwin64\bfin-gnu-new\include" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\cores\arduino" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\variants\feather_bfin" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\libraries\SPI" "C:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\libraries\SPI\SPI.cpp" -o "nul"
===debug ||| Ts: {0} - Running: {1} ||| [1522205384 IncludesFinderWithRegExp]
===debug ||| Ts: {0} - Running: {1} ||| [1522205384 FailIfImportedLibraryIsWrong]
===debug ||| Ts: {0} - Running: {1} ||| [1522205384 WarnAboutArchIncompatibleLibraries]
===info ||| Progress {0} ||| [26.47]
===debug ||| Ts: {0} - Running: {1} ||| [1522205384 loggerAction]
===info ||| Progress {0} ||| [29.41]
===info ||| Generating function prototypes... ||| []
===debug ||| Ts: {0} - Running: {1} ||| [1522205384 ContainerAddPrototypes]
===info ||| Progress {0} ||| [32.35]
===debug ||| Ts: {0} - Running: {1} ||| [1522205384 GCCPreprocRunner]
"C:\cygwin64\bfin-gnu-new\buildscript\bfin-elf-win\bin\bfin-elf-g++" -mcpu=bf706 -c -w -std=gnu++11 -g3 -fno-rtti -ffunction-sections -fdata-sections -fno-threadsafe-statics -nostdlib -Os --param max-inline-insns-single=500  -w -x c++ -E -CC -DF_CPU=48000000L -DARDUINO=10804 -DARDUINO_FEATHER_BFIN -DARDUINO_ARCH_BFIN  -DARDUINO_BFIN -D__ADSPBF706__ "-IC:\cygwin64\bfin-gnu-new\include" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\cores\arduino" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\variants\feather_bfin" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\libraries\SPI" "C:\Users\Dean\AppData\Local\Temp\arduino_build_854378\sketch\bfin-test.ino.cpp" -o "C:\Users\Dean\AppData\Local\Temp\arduino_build_854378\preproc\ctags_target_for_gcc_minus_e.cpp"
===debug ||| Ts: {0} - Running: {1} ||| [1522205384 ReadFileAndStoreInContext]
===debug ||| Ts: {0} - Running: {1} ||| [1522205384 FilterSketchSource]
===debug ||| Ts: {0} - Running: {1} ||| [1522205384 CTagsTargetFileSaver]
===debug ||| Ts: {0} - Running: {1} ||| [1522205384 CTagsRunner]
"C:\ProgramFiles\Arduino\tools-builder\ctags\5.8-arduino11/ctags" -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzns --line-directives "C:\Users\Dean\AppData\Local\Temp\arduino_build_854378\preproc\ctags_target_for_gcc_minus_e.cpp"
===debug ||| Ts: {0} - Running: {1} ||| [1522205385 PrototypesAdder]
===debug ||| Ts: {0} - Running: {1} ||| [1522205385 SketchSaver]
===debug ||| Ts: {0} - Running: {1} ||| [1522205385 loggerAction]
===info ||| Progress {0} ||| [35.29]
===info ||| Compiling sketch... ||| []
===debug ||| Ts: {0} - Running: {1} ||| [1522205385 RecipeByPrefixSuffixRunner]
===info ||| Progress {0} ||| [38.24]
===debug ||| Looking for recipes like {0}*{1} ||| [recipe.hooks.sketch.prebuild .pattern]
===debug ||| Ts: {0} - Running: {1} ||| [1522205385 SketchBuilder]
===info ||| Progress {0} ||| [41.18]

[matthijskooijman]

Weird stuff. Thanks for the detailed report so far.

Because windows command line can't do spaces in paths. Other packages work fine in the IDE.

I think you can fix this by adding quotes around the arguments with spaces, IIRC that's also what the IDE does internally when calling arduino-builder (though it does not actually display these quotes, it seems).

As for the command that should be detecting SPI.h, that seems to be
exactly the same in both the broken and working case:

"C:\cygwin64\bfin-gnu-new\buildscript\bfin-elf-win\bin\bfin-elf-g++" -mcpu=bf706 -c -w -std=gnu++11 -g3 -fno-rtti -ffunction-sections -fdata-sections -fno-threadsafe-statics -nostdlib -Os --param max-inline-insns-single=500  -w -x c++ -E -CC -DF_CPU=48000000L -DARDUINO=10804 -DARDUINO_FEATHER_BFIN -DARDUINO_ARCH_BFIN  -DARDUINO_BFIN -D__ADSPBF706__ "-IC:\cygwin64\bfin-gnu-new\include" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\cores\arduino" "-IC:\Users\Dean\Documents\Arduino\hardware\Dean\bfin\variants\feather_bfin" "C:\Users\Dean\AppData\Local\Temp\arduino_build_854378\sketch\bfin-test.ino.cpp" -o "nul"

There are two possible causes that spring to mind:

1. arduino-builder does not parse the SPI.h error message properly in
   the broken case. Perhaps that error is subtly different, not output
   on stderr, or otherwise something fails here.
2. arduino-builder cannot resolve SPI.h to an available library in
   the broken case. Perhaps there are permission problems, or it fails
   to properly list the platform-specific libraries or something?

In the output, I can't see anything particular that points to either of
these, so there might be something different entirely.

It's super-weird that it works when calling arduino-builder manually,
and this also complicates debugging.

Perhaps the spaces thing changes something, can you try again with
quotes instead of the symlink?

[kgoveas]

@deanm1278,

Can you try instead #include "SPI.h"?

[deanm1278]

hey @matthijskooijman @kgoveas thanks very much for the quick response!
Turns out it's a permissions problem maybe due to the fact that I'm using a compiler and some includes in a C:\cygwin\ directory until I get an actual arduino package set up.

Anyway if I run the IDE as administrator everything works as expected. Thanks for your time and the great work on the builder.