IDE picks old version of installed 3rd party library after library upgrade

[anno73]

Hi!

I am currently working on an ESP8266 based project in Arduino IDE v1.8.13.
I installed Streaming.h via IDE library manager and subsequently upgraded Streaming.h library to the latest version. As it seems I now have two versions of Streaming,h because during compilation I get

Alternatives for Streaming.h: [arduino_467138@6.0.8 arduino_412692@5.0.0]
ResolveLibrary(Streaming.h)
  -> candidates: [arduino_467138@6.0.8 arduino_412692@5.0.0]

Unfortunately the IDE picks version 5.0.0 instead of 6.0.8. (see last -I parameter):

"C:\\Users\\alex\\AppData\\Local\\Arduino15\\packages\\esp8266\\tools\\xtensa-lx106-elf-gcc\\2.5.0-4-b40a506/bin/xtensa-lx106-elf-g++" 
-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ 
"-IC:\\Users\\alex\\AppData\\Local\\Arduino15\\packages\\esp8266\\hardware\\esp8266\\2.7.4/tools/sdk/include" 
"-IC:\\Users\\alex\\AppData\\Local\\Arduino15\\packages\\esp8266\\hardware\\esp8266\\2.7.4/tools/sdk/lwip2/include" 
"-IC:\\Users\\alex\\AppData\\Local\\Arduino15\\packages\\esp8266\\hardware\\esp8266\\2.7.4/tools/sdk/libc/xtensa-lx106-elf/include" 
"-IC:\\Users\\alex\\AppData\\Local\\Temp\\arduino_build_979772/core" -c -w -Os -g -mlongcalls -mtext-section-literals 
-fno-rtti -falign-functions=4 -std=gnu++11 -ffunction-sections -fdata-sections -fno-exceptions -w -x c++ -E -CC 
-DNONOSDK22x_190703=1 -DF_CPU=80000000L -DLWIP_OPEN_SRC -DTCP_MSS=536 -DLWIP_FEATURES=1 -DLWIP_IPV6=0 -DARDUINO=10813 
-DARDUINO_ESP8266_WEMOS_D1MINIPRO -DARDUINO_ARCH_ESP8266 "-DARDUINO_BOARD=\"ESP8266_WEMOS_D1MINIPRO\"" 
-DFLASHMODE_DIO -DESP8266 
"-IC:\\Users\\alex\\AppData\\Local\\Arduino15\\packages\\esp8266\\hardware\\esp8266\\2.7.4\\cores\\esp8266" 
"-IC:\\Users\\alex\\AppData\\Local\\Arduino15\\packages\\esp8266\\hardware\\esp8266\\2.7.4\\variants\\d1_mini" 
"-ID:\\CloudStorage\\Dropbox\\Work\\Arduino\\libraries\\arduino_485728\\src" 
"-ID:\\CloudStorage\\Dropbox\\Work\\Arduino\\libraries\\arduino_412692\\src" 
"C:\\Users\\alex\\AppData\\Local\\Temp\\arduino_build_979772\\sketch\\MAX7219_MQTT_IoTWebConf.ino.cpp" 
-o nul -DARDUINO_LIB_DISCOVERY_PHASE
:
:
Multiple libraries were found for "Streaming.h"
 Used: D:\CloudStorage\Dropbox\Work\Arduino\libraries\arduino_412692
 Not used: D:\CloudStorage\Dropbox\Work\Arduino\libraries\arduino_467138
:

This is also true for other libraries I have installed via library manager, e.g.
The same behavior also on version 1.8.14 hourly build 2021/12/15 11:33.

My expectation was that during compile time the latest version of installed libraries matching board/core/... is used or that I can instruct in the IDE (or even better in the sketch) which library version should be used.

Strangely enough the order of the detected library version list is not the same after restarts. And during compile time most of the time the library version as the first element gets used. So I get errors when using new features not present in old version.

Alex.

[matthijskooijman]

I believe the problem is caused by some interaction between Dropbox and the IDE that prevents the IDE from properly renaming the library directory during an install or upgrade, so this leaves multiple library versions lingering around. That issue is tracked here: arduino/Arduino#11203

However, there is another issue here: How should identical libraries be disambiguated. It might make sense to always take the newest library, though in practice the IDE might not really realize that it's looking at two version of the same library (as opposed to two different libraries that both happen to supply the same header file name, in which case comparing version numbers is meaningless). Also, in practice having multiple versions of the same library installed doesn't really make sense, so I'm not sure if this is a case to really support. So I'm inclined to close this issue, and leave just the other one?

[ubidefeo]

@matthijskooijman
in theory we do not support multipla library versions in the sketchbook libraries folder.
The Library Manager always overwrites the content of the library.
There is a scoring system that assigns priorities based on the library location.

When it comes to Dropbox I also keep my sketchbook on it and what many users face is an issue with the specific folder not getting synced.
The Sketchbook should be marked as "always sync" on any system that has the specific Dropbox account setup.
New installations default to selective sync which can often be a pain in the teeth.

[anno73]

@matthijskooijman

However, there is another issue here: How should identical libraries be disambiguated. It might make sense to always take the newest library, though in practice the IDE might not really realize that it's looking at two version of the same library (as opposed to two different libraries that both happen to supply the same header file name, in which case comparing version numbers is meaningless).

If the libraries provide libraries.properties this could be resolved via the URL and VERSION keys. I assume that different libraries have different URLs.

Also, in practice having multiple versions of the same library installed doesn't really make sense, so I'm not sure if this is a case to really support. So I'm inclined to close this issue, and leave just the other one?

The reality is, when I want to adapt some code of an older project the first thing is to fix the code due to breaking changes in upgraded libs. Like IotWebConf or ArduinoJSON. Or revert library versions to the ones used at the time. This requires of course documentation of the used versions at the time in the code.
There have been a lot of times where I had wished that e.g. #include <SomeLib.h> // Version=x.y.z would have used the lib in the version the program was developped with automatically.

[anno73]

@ubidefeo

When it comes to Dropbox I also keep my sketchbook on it and what many users face is an issue with the specific folder not getting synced.
The Sketchbook should be marked as "always sync" on any system that has the specific Dropbox account setup.
New installations default to selective sync which can often be a pain in the teeth.

I had a look at the dropbox sync client and the dropbox web site. In the client I found Preferences->Sync->Selective Sync which opens up a window where one can select specific folders which should not be synced from dropbox to local PC. I have ticked Select All here.
I could not find an option for always sync. Could you please be more specific on this one so I can check? On the other hand, I never had an issue with folders not getting synced though. Just these race conditions on folder rename.
As you use dropbox too, do you also have these problems?

[ubidefeo]

@matthijskooijman

There have been a lot of times where I had wished that e.g. #include <SomeLib.h> // Version=x.y.z would have used the lib in the version the program was developped with automatically.

Using CLI you can put different versions of a library (same headers and files) into a folder and use --libraries as a flag to pass the path to the folder containing the library/libraries you want to get higher priority.
In the meantime we're looking into adding a --library flag to pass in the path to a specific library.
This will be leveraged in the future to allow versioned libraries such as Arduino_ConnectionHandler@x.y.z.
I'm still on the proposal phase but it seems to be a welcome addition.
I mean... my team hasn't told me to "rot op" :D

[ubidefeo]

@anno73

I have ticked Select All here.

that's what I meant, I just didn't remember the words.
I use Dropbox to sync my Sketchbook across 3 machines but they are only used one at a time.
I have never experienced these renaming conflicts

[anno73]

@ubidefeo

I use Dropbox to sync my Sketchbook across 3 machines but they are only used one at a time.

Same here.

I have never experienced these renaming conflicts

Interesting. I am using Windows 10 and NTFS. I am curious, do you too?

[ubidefeo]

Interesting. I am using Windows 10 and NTFS. I am curious, do you too?

nope. Mac OS here

[umbynos]

Hi @anno73, we modified how the libraries are installed. Before the library was unpacked in the target dir and then renamed. Now if the rename goes wrong, it is copied as a fallback. Could you please try if your problem still exists with the latest version?