Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On multi environment project all libraries are included/Compiled #1696

Closed
Pedroalbuquerque opened this issue Jun 20, 2018 · 16 comments

Comments

7 participants
@Pedroalbuquerque
Copy link

commented Jun 20, 2018

Configuration

PIO 3.5 with ATOM on MAC 10.12.6

Description of problem

When defining multiple environments on a project as [env:nodemcuv2] and [env:esp32]

I need to include different libraries for each of the environment, so I use lib_deps var to define for each environment the needed libraries.

The problem I face is that if I compile a single environment using default_env var, there is not problem, the includes work fine for each one of the environment,

If I do not specify de env to use, both are built and I need to explicitly exclude some library on ESP8266 env that are included in esp32.

What I believe is happening is that the library dependency folder is common for both environements so ALL included libraries for both environment are copied to that .piolibdeps folder so I need in each environment exclude the libs included in the other environment that do not apply.

Some libraries are not compatible with both environments raising error during compilation, even if they are no #include in the code.

A second effect (I believe may come from same root cause is that in case I use compile directive that exclude some lib #include, the lib is still compiled and not being compatible with the platform, will raise error.

Steps to Reproduce

Actual Results

with code used bellow the second effect is visible and I do not have currently code example prepared fro first symptom.
So result for second is that error code is generated when
env_default=moteinomega
because TFT_eSPI lib is not compatible with moteinoMega

but copiles ok with
env_default=nodemcuv2

Indexing .pioenvs/moteinomega/libe97/libdisplay.a
Archiving .pioenvs/moteinomega/lib4e9/libAdafruit_GFX_Library.a
/Users/pedroalbuquerque/documents/googledrive/projects/libraries/TFT_eSPI/TFT_eSPI.cpp:4147:2
5: fatal error: soc/spi_reg.h: No such file or directory
#include "soc/spi_reg.h"
^
compilation terminated.
Archiving .pioenvs/moteinomega/lib135/libAdafruit_PCD8544.a
*** [.pioenvs/moteinomega/lib7bb/TFT_eSPI/TFT_eSPI.cpp.o] Error 1
Indexing .pioenvs/moteinomega/lib4e9/libAdafruit_GFX_Library.a

Indexing .pioenvs/moteinomega/lib135/libAdafruit_PCD8544.a
========================== [ERROR] Took 2.90 seconds ==========================

Expected Results

I would expect that due to the fact that macro ESP32 is not defined,
everything after #if defined(ESP32)
would not be compiled avoiding the error.
Better said, the library referenced on an #include after that #if would not be compiled.
Actually I believe the include does not take place, and the object instance using that lib is not compiled, but the lib itself is compiled anyway.

When using arduino IDE to compile both with platform MoteinoMega as well with nodemcuv2, it compiles ok with no error.

If problems with PlatformIO Build System:

The content of platformio.ini:

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; http://docs.platformio.org/page/projectconf.html

[platformio]
env_default=moteinomega
src_dir=./
lib_dir=~/documents/googledrive/projects/libraries
lib_extra_dirs =/Users/pedroalbuquerque/Documents/GoogleDrive/Projects/Lib_dev

[common]
build_flags = -g
    -DECHO_PORT=Serial
    -DDEBUG_PORT=Serial
    -DMQTT_MAX_PACKET_SIZE=400
    -Wcomment
lib_ignore = ESP32WebServer
    RFM69_OTA

[env:nodemcuv2]
platform = espressif8266
board = nodemcuv2
framework = arduino
build_flags = ${common.build_flags}

[env:moteinomega]
platform = atmelavr
board = moteinomega
framework = arduino
;upload_port=/dev/cu.usbserial-DN00Z51U
;upload_port=/dev/cu.usbserial-DA01I8AV
;upload_speed=115200
build_flags = ${common.build_flags}
;  -D__AVR_ATmega1284P__
lib_ignore = ${common.lib_ignore}, TFT_eSPI

Source file to reproduce issue:
"Main file":

#include <libT2.h>
//#include <adafruit
#include <displayGeneric.h>


TTf f;


void setup(){
  Serial.begin(115200);

}

void loop(){

  uint8_t myint = 20;
  uint16_t myint16 = 11111;
  float myfloat = 12.3;
  struct {
      int i = -10;
      char str[10]="hello!";
  } mystruct;

  f.bugger();
  f.myf(myint);
  f.myf(myint16);
  f.myf(myfloat);
  //f.myf(myfloat);


  delay(1000);

}

Lib file that exclude another lib with compile directives

#ifndef DISPLAY_H
#define DISPLAY_H

#define DISPLAY_LCD
//#define DISPLAY_ST7735  //uncomment to select this display type/model


#include <Adafruit_GFX.h>    // Core graphics library
#if defined(DISPLAY_LCD)

	#include <Adafruit_PCD8544.h> //Required for the Nokia 5110 LCD display

	// Initiate the display instance - Software SPI (slower updates, more flexible pin options):
	// pin 24/A0 - Serial clock out (SCLK)
	// pin 25/A1 - Serial data out (DIN)
	// pin 26/A2 - Data/Command select (D/C)
	// pin 27/A3 - LCD chip select (CS)
	// pin 28/A4 - LCD reset (RST)

  #if defined(ESP32)
	    // pins defined @ User_Setup.h on TFT_eSPI folder
	    /*
	    #include <Adafruit_ST7735.h> //Required for OLED LCD
	    #define RST  26 //13 //1 //A5 //18
	    #define RS   27 //12 //0 //A4 //19
	    #define SDA  14 // A2 //20
	    #define SCL  12 //27 // A0 //21
	    #define CS   13 //26 //3 //A3 //22

	    Adafruit_ST7735 display = Adafruit_ST7735(CS, RS, SDA, SCL, RST); //, RST);
	    */
	    #include <TFT_eSPI.h> //Required for OLED LCD
	    TFT_eSPI display = TFT_eSPI();

  #else //defined(ESP8266)
    //D2 (GPIO4)	0 RST	Output from ESP to reset display
    //D1 (GPIO5)	1 CE	Output from ESP to chip select/enable display
    //D6 (GPIO12)	2 DC	Output from display data/command to ESP
    //D7 (GPIO13)	3 Din	Output from ESP SPI MOSI to display data input
    //D5 (GPIO14)	4 Clk	Output from ESP SPI clock
    //3V3	5 Vcc	3.3V from ESP to display
    //D0 (GPIO16)	6 BL	3.3V to turn backlight on, or PWM
    //G (GND)	7 Gnd	Ground

  	#define RST  4 //A5 //18   // RST pin on LCD
  	#define RS   10 //A4 //19   // DC pin on LCD
  	#define SDA  MISO //20       // DIN pin on LCD
  	#define SCL  6 //21       // CLK pin on LCD
  	#define CS   9             // CE pin o LCD

    Adafruit_PCD8544 display = Adafruit_PCD8544(SCL, SDA, CS, RS, RST);
    #define PIN_LCD_LIGHT 5 //Backlight pin for NOKIA LCD (LIGHT pin on LCD)

	#endif


	// character size from Adafruit_GXF lib
	#define CHARWIDTH 6
	#define CHARHEIGHT 8
	#define SCRROTATION 0 // 0 rotation
	#define CHARSCALE 1
	#define SCRPIXELX 240
	#define SCRPIXELY 320
	#define SCRLINES 6    // 6 lines
	#define SCRCHARS 14   // 14 characters
///  #define BOXSIZE 120

#endif



Additional info

@ivankravets ivankravets added this to the 3.6.0 milestone Jun 21, 2018

@ivankravets

This comment has been minimized.

Copy link
Member

commented Jun 21, 2018

I think PlatformIO should install libraries to %projectdepsdir%/%env%/. In this case, we will not have conflict.

Possible issues:

  • PIO Home depends on the one libdeps_dir
  • Lib-changes in platformio.ini should lead to removing of libdeps_dir

@ivankravets ivankravets modified the milestones: 3.6.0, 3.5.4 Jul 2, 2018

@sarfata

This comment has been minimized.

Copy link

commented Jul 6, 2018

@ivankravets I think this issue was recently introduced by 3.5.3 or 3.5.4. In my opinion, breaking projects that used to compile should qualify as a bug, not an enhancement.

With platformio 3.5.2 my project with multi environments compiles fine with 3.5.2 and now with 3.5.4 it does not build anymore.

sarfata added a commit to sarfata/kbox-firmware that referenced this issue Jul 6, 2018

Update config to work with latest platformio 3.5.4
This addresses two separate "issues" in the latest platformio:
 - CCFLAGS are now defined separately for frameworks and project files
platformio/platformio-core#1719
 - platformio builds all libraries for all platforms if they are not
excluded
platformio/platformio-core#1696
@p3p

This comment has been minimized.

Copy link

commented Oct 16, 2018

@ivankravets Is there a solution for this in the works? I seem to be able to work around the issue with

lib_ldf_mode      = chain+
lib_compat_mode   = strict

but it more than doubles our Travis test times, it is more of a dev issue than end user as they tend to just build one env but it still causes problems.

@ivankravets

This comment has been minimized.

Copy link
Member

commented Oct 16, 2018

@p3p you can manually control LDF with lib_deps and lib_ignore options instead of lib_ldf_mode = chain+

@p3p

This comment has been minimized.

Copy link

commented Oct 16, 2018

Indeed I can specify everything myself or just disable the ldf as I have needed to in the past, but the issue of building incompatible libraries for envs that don't need them if they have already been installed by another env seems to be something that should be fixed?

@ivankravets

This comment has been minimized.

Copy link
Member

commented Oct 16, 2018

but the issue of building incompatible libraries for platforms that don't need them if they have already been installed by another env seems to be something that should be fixed?

Yes, this should be fixed with this feature request.

Indeed I can specify everything myself or just disable the ldf

  1. Disable lib_ldf_mode = chain+
  2. Start building
  3. If some library was included and you don't need it, exclude with lib_ignore
  4. If some library was not included in build process, add it manually to lib_deps.

All these options put into specific env.

@p3p

This comment has been minimized.

Copy link

commented Oct 16, 2018

We already do a lot of cherry picking libraries over at Marlin we support a lot of environments, I was just going to report the bug and found this issue with a changing milestone so thought I'd make sure it was still in the works ;)

@ivankravets

This comment has been minimized.

Copy link
Member

commented Oct 16, 2018

It's not an old issue :) We plan to start a work on PlatformIO Core 4.0 soon.

@p3p

This comment has been minimized.

Copy link

commented Oct 16, 2018

Didn't mean to imply you were sitting on your hands, sometimes issues can just fly under the radar, as you say it's not a major issue as I can work around it.

@aster94

This comment has been minimized.

Copy link

commented Mar 23, 2019

even if this is not completely related to this issue: note that one of the problem of the LDF is that it should include the right library based on the architecture/platform
example: https://platformio.org/lib/show/870/WiFi as you can see from the platform on the right it is wrong since that library is only for Arduino WiFi Shield but it has the same name of the ESP32 version. The same happens, i guess, for the other platform

Edit: I will try to propose a way to do it: platformio should compute the priority of two libraries using the library.properties file especially the architectures field, a function like: computePriorities(library x, library y, used_platform) could be made that gives higher priority to the library which has the same architecture to the used_platform. Also built-in libraries (like in this case) should be preferred instead of external one

@ristomatti

This comment has been minimized.

Copy link

commented Apr 27, 2019

I ran across this while attempting to make Embedded Template Library Arduino compatible with PlatformIO. I've got to the point that espressif8266 and espressif32 platforms seem to work but supporting atmelavr (without specifically documenting it) would require setting the library to depend on ArduinoSTL. This causes build to fail on other platforms even when ArduinoSTL.h is not included.

If I define lib_compat_mode = strict in platformio.ini everything works great. So I thought maybe if in library.json I define:

  "build": {
    "libCompatMode": "strict"

... it will do the same but it doesn't. When I run pio run -e d1_mini -v I see a message indicating the unsupported library would be skipped but it is still included in the compiler parameters:

CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/d1_mini.html
PLATFORM: Espressif 8266 > WeMos D1 R2 and mini
HARDWARE: ESP8266 80MHz 80KB RAM (4MB Flash)
Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF MODES: FINDER(chain) COMPATIBILITY(soft)
Collected 79 compatible libraries
Scanning dependencies...
Skip platform incompatible dependency {u'frameworks': [u'arduino'], u'version': u'>=1.1.0', u'name': u'ArduinoSTL', u'platforms': [u'atmelavr']}

In PlatformIO library index ArduinoSTL seems to also be defined only for atmelavr and atmelsam.

I also noticed the library.json documentation instructs libCompatMode should be defined as Integer but the referred documentation uses String. I tried giving it numeric values 2 and 3 but it ddn't seem to make a difference.

@ivankravets ivankravets moved this from To do to In progress in PlatformIO 4.0 May 20, 2019

@ivankravets ivankravets moved this from In progress to Done in PlatformIO 4.0 May 22, 2019

@ivankravets

This comment has been minimized.

Copy link
Member

commented May 22, 2019

Please re-test with pio upgrade --dev. Updated docs => http://docs.platformio.org/en/latest/userguide/lib/index.html

pio lib --help
@mcspr

This comment has been minimized.

Copy link

commented May 24, 2019

Based on OP issue, isn't it better to make libdeps dir name based on platform+framework+(+board?) instead of env name? If multiple environments have the same lib_deps variable and the same platform,framework, libs are installed each time.

@ivankravets

This comment has been minimized.

Copy link
Member

commented May 25, 2019

@mcspr and we will back to the same problem which we had before. For example, some envs can have different lib_deps and platform+framework+(+board?) ID will lead to a conflict issue.

The only solution is to provide isolated build environment for project configuration. This is actually what PlatformIO Core 4.0 will do.

Please note that we don't do multiple downloads of project dependencies. We cache them. Normally, the size of a library is 50-300Kb.

@mcspr

This comment has been minimized.

Copy link

commented May 25, 2019

Yes, I did miss possibility of different library versions.

However, I think it still force downloads github repos? For example

lib_deps =
    ArduinoJson@5.13.4
    https://github.com/me-no-dev/ESPAsyncTCP#7e9ed22
    https://github.com/xoseperez/NtpClient.git#0942ebc

Fixed version works fine with platformio registry, but next github repo is one picking specific commit mid versions, second one is forked library. Both will be git-cloned.
Or some other method should be used, like using branch / commit archive zips?(e.g replace the first link with https://github.com/me-no-dev/ESPAsyncTCP/archive/7e9ed22ed0078097f895dfee6ebcec90046b81b9.zip)

I think I found a bug (warrants another issue to track?)
Caching is evidently broken: #2559

@ivankravets

This comment has been minimized.

Copy link
Member

commented May 26, 2019

@mcspr thank you so much for the issue #2559

I'll investigate!

@ivankravets ivankravets added this to Done in PlatformIO 4.0 May 29, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.