delete auto downloaded SD library in users <sb>/libraries directory, does not then use board specific one

[KurtE]

Describe the problem

I installed RC7 on new machine (RPI4 running Ubuntu 22.04). But pretty sure other setups would work the same.

I then installed the Teensy boards, which installs it's own SD library (~/.arduino15/packages/teensy/hardware...)

I then opened a sketch and tried to compile it, I received several strange errors and then remembered it was using the old SD library. So opened up Arduino/libraries folder and deleted SD.

I then tried to build again.

The compiler then failed with error saying that it used: SD.h from the Arduino/libraries/SD folder and not the one from the teensy install. And then gave error that the file did not exist.

To reproduce

Described above.

Expected behavior

Would expect it to see that the folder did not exist and then do the find of it again.

Note: I also tried closing the window and reopen it and it still gave same error.

Arduino IDE version

RC7

Operating system

Linux

Operating system version

Ubuntu 22.04 (RPI 64 bit)

Additional context

Sort of related to my arduino/arduino-ide#798, where I would hope if I had a custom version already installed it would not overwrite.

Also tried to search for duplicate but hard to find search terms.

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the latest nightly build
• My report contains all necessary details

[per1234]

Thanks so much for your report @KurtE!

I was able to determine that this is actually a bug in Arduino CLI, so I have transferred the issue to the appropriate repository.

This is not in any way specific to the Teensy Arduino boards platform, but that platform does provide a dramatic demonstration of the bug due to the Library Manager installation of "SD" being given priority over the platform bundled installation. In most cases (e.g., "USBHost" lib and arduino:samd platform), the bundled library gets priority, so the bug only manifests in the oddity of the library removed from <directories.user>/libraries still appearing under a "Multiple libraries were found for" section of the verbose compilation output.

I bisected the bug to f0245bc

In case it might assist the developers in the investigation, I'll provide a demo of the issue using Arduino CLI directly:

$ arduino-cli version
arduino-cli.exe  Version: 0.23.0 Commit: 899dc91b Date: 2022-06-11T07:40:48Z

$ mkdir "/tmp/FooSketch"

$ printf "#include <SD.h>\nvoid setup(){}\nvoid loop(){}" > "/tmp/FooSketch/FooSketch.ino"

$ export ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS="https://www.pjrc.com/teensy/td_156/package_teensy_index.json"

$ arduino-cli core update-index

[...]

$ arduino-cli core install teensy:avr  # platform chosen because it has bundled lib w/ header filename also present in an LM lib

[...]

$ arduino-cli lib install SD  # this lib also bundled w/ teensy:avr

[...]

$ arduino-cli daemon

Run these grpcurl commands in another terminal from the root of the Arduino CLI repository:

$ grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Create

{
  "instance": {
    "id": 1
  }
}

$ grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  -d '{"instance": {"id": 1}}' \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Init  # use the instance ID provided by the Create response in this and all subsequent commands

$ grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  -proto cc/arduino/cli/commands/v1/compile.proto \
  -d '{"instance": {"id": 1}, "fqbn": "teensy:avr:teensy41", "sketch_path": "/tmp/FooSketch/FooSketch.ino"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Compile  # compile the sketch

[...]

$ echo $?  # compilation was successful as expected (the LM library will compile for teensy:avr:teensy41 with this simple sketch)
0

$ arduino-cli lib uninstall SD  # remove the LM installation of the lib
Uninstalling SD@1.2.4...

$ grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  -proto cc/arduino/cli/commands/v1/compile.proto \
  -d '{"instance": {"id": 1}, "fqbn": "teensy:avr:teensy41", "sketch_path": "/tmp/FooSketch/FooSketch.ino"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Compile  # compile the sketch again

[...]

$ echo $?  # compilation failed unexpectedly, it should have passed, using the platform bundled SD library this time
77

$ grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  -d '{"instance": {"id": 1}}' \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Init  # reinitialize the instance

$ grpcurl \
  -plaintext \
  -import-path ./rpc \
  -proto cc/arduino/cli/commands/v1/commands.proto \
  -proto cc/arduino/cli/commands/v1/compile.proto \
  -d '{"instance": {"id": 1}, "fqbn": "teensy:avr:teensy41", "sketch_path": "/tmp/FooSketch/FooSketch.ino"}' \
  127.0.0.1:50051 \
  cc.arduino.cli.commands.v1.ArduinoCoreService.Compile  # compile the sketch

[...]

$ echo $?  # compilation now passes
0

🐛 The second compilation failed. It was necessary to reinitialize Arduino CLI in order to get it to recognize the removal of the library.

[KurtE]

@per1234 @cmaglie

Sorry maybe it is just me, but I think there is something missing in the transition from Arduino 1.x to Arduino 2.x as it relates to library search order.

That is with for example Arduino 1.8.19, The Arduino installer, did install a version of the SD library.
In: C:\arduino-1.8.19\libraries

When I install Teensyduino: there is a version of it installed at: C:\arduino-1.8.19\hardware\teensy\avr\libraries
Likeise ESP32 has it's version installed at: C:\Users\kurte\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.3\libraries

Now if I am working on the SD library, I may put a version up at: C:\Users\kurte\Documents\Arduino\libraries

And if I am not mistaken, the library search order is:
a) Highest priority: C:\Users\kurte\Documents\Arduino\libraries
b) Second Priority: C:\arduino-1.8.19\hardware\teensy\avr\libraries
c) lowest priority: C:\arduino-1.8.19\libraries

Which makes sense.

But now with the Current RC for Arduino V2 you got rid of lowest priority and now install the default IDE libraries in the sketch folder (highest priority), which breaks all of the setups that use board specific libraries.

Also as per my other issue (arduino/arduino-ide#798)
The install at least at that time, wiped out my custom library I had installed in the /libraries/SD
directory.

[per1234]

the library search order is:

This is a subject I absolutely love to discuss, but that discussion would be completely off topic for this issue. I will point you to the complete documentation here in case you would like to learn about how it really works:

https://arduino.github.io/arduino-cli/dev/sketch-build-process/#dependency-resolution

If you have any questions or would like to discuss that more, please post over on the Arduino forum. I'll be happy to give you the full "-v" over there.

with the Current RC for Arduino V2 you got rid of lowest priority and now install the default IDE libraries in the sketch folder (highest priority), which breaks all of the setups that use board specific libraries.

That is off topic for discussion in this issue and for discussion in this repository. If there are bug report or feature request you would like to make that is distinct from the existing one at arduino/arduino-ide#798, please do submit them. But let's keep the discussion in this issue focused exclusively on Arduino CLI not recognizing library removals during a session.

[PaulStoffregen]

@per1234 - At least from this 1 quick test, Arduino CLI is properly recognizing library removals. I'm pretty sure this is an IDE 2.0 issue, not a problem with Arduino CLI. I'd recommend moving this back to the IDE repository. Perhaps the title should be edited too, to more clearly emphasize the real problem that the wrong library ends up getting used.

I'm not quite sure what "during a session" means regarding Arduino CLI, but I here's the test I tried just now to check whether Arduino CLI is not recognizing library removals. But here's the test I ran just now.

First, I'm using the latest nightly, Linux x86-64 build.

paul@preston:/tmp > /tmp/arduino-cli version
arduino-cli  Version: nightly-20220612 Commit: 899dc91 Date: 2022-06-12T01:40:11Z

My machine's current packages are a result of having recently run IDE 2.0.0-rc7.

When I run it to compile a sketch which uses the SD library, I get this:

paul@preston:/tmp > /tmp/arduino-cli  compile --clean -b teensy:avr:teensy36 ~/teensy/sketch/T40_Tester
Sketch uses 119748 bytes (11%) of program storage space. Maximum is 1048576 bytes.
Global variables use 14952 bytes (5%) of dynamic memory, leaving 247192 bytes for local variables. Maximum is 262144 bytes.

Used library Version Path                                                                           
SPI          1.0     /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/SPI        
SD           1.2.4   /home/paul/Arduino/libraries/SD                                                
EEPROM       2.0     /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/EEPROM     
Entropy              /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/Entropy    
FreqMeasure  1.2     /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/FreqMeasure
Wire         1.0     /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/Wire       

Used platform Version Path                                                     
teensy:avr    1.56.1  /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1

The real problem is use of /home/paul/Arduino/libraries/SD. I don't understand why this happens well enough to create a narrowly focused issue.

But I do know I can test whether Arduino CLI recognizes its removal.

paul@preston:~ > rm -rf /home/paul/Arduino/libraries/SD

Then when I run the same Arduino CLI command again, it does indeed use the SD library from Teensy's package.

paul@preston:/tmp > /tmp/arduino-cli  compile --clean -b teensy:avr:teensy36 ~/teensy/sketch/T40_Tester
Sketch uses 149088 bytes (14%) of program storage space. Maximum is 1048576 bytes.
Global variables use 15712 bytes (5%) of dynamic memory, leaving 246432 bytes for local variables. Maximum is 262144 bytes.

Used library Version Path                                                                           
SPI          1.0     /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/SPI        
SD           2.0.0   /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/SD         
SdFat        2.1.0   /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/SdFat      
EEPROM       2.0     /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/EEPROM     
Entropy              /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/Entropy    
FreqMeasure  1.2     /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/FreqMeasure
Wire         1.0     /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/Wire       

Used platform Version Path                                                     
teensy:avr    1.56.1  /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1

Again, I don't fully understand this problem. But from this test, I'm pretty confident it's not within Arduino CLI. Or if some bug within Arduino CLI was the cause of a conflicting copy of SD appearing at /home/paul/Arduino/libraries/SD, it definitely did respond properly when I deleted that errant copy, and in using only CLI that errant SD library has not magically reappeared. So I can't 100% rule out CLI, but all signs point to CLI is working as expected and the cause of this conflict is most likely something going wrong with library management in the IDE.

Recommend moving this issue back to IDE.

[per1234]

At least from this 1 quick test, Arduino CLI is properly recognizing library removals.

That is because you are using the command line interface. The library removal is recognized on initialization, which happens every time you run an arduino-cli command. This is why you must use the gRPC interface like the Arduino IDE 2.x does in to reproduce the bug, as shown in the demo I provided above. That is the only way to run multiple compile operations in a single session.

[PaulStoffregen]

Ah, ok. I'm not familiar with the gRPC interface. Will put it on my bucket list...

Where would be the appropriate place to discuss the problem that something is causing installation of the generic SD library at /home/paul/Arduino/libraries/SD, which overrides the platform specific SD library? Is an issue already open for that problem?

[per1234]

something is causing installation of the generic SD library at /home/paul/Arduino/libraries/SD

This is the intended behavior for the first run of the Arduino IDE: arduino/arduino-ide#663

which overrides the platform specific SD library

This can be fixed by simply adjusting the metadata of the Teensy platform's bundled SD library. @KurtE has already explained it over on the forum thread:

https://forum.pjrc.com/threads/53548-Arduino-CLI-Alpha-Release-Teensy-Support/page9

If you have any questions about that, I will be happy to answer them over there.

Is an issue already open for that problem?

There isn't anything other than the only peripherally related arduino/arduino-ide#798 about the built-in libraries installation. If you have a proposal about refining its behavior separate from the overwrite issue tracked by arduino/arduino-ide#798 (which clearly needs to be fixed), feel free to submit an issue over on the arduino/arduino-ide repository.

As for the override issue, this was discussed extensively back when Library Manager was first implemented. I'm sure there are multiple issues about it, but I found this one:

arduino/Arduino#4064