Unable to clone full platform (LWIP2 git offline) #6832

TD-er · 2019-11-24T21:35:41Z

The repository https://git.savannah.nongnu.org/git/lwip.git is offline.
This means the platform cannot be fetched.

Is it possible to define some archive somewhere as backup?
Now it is this external repository, but it may happen to some external repository some other time, so maybe it is good to be able to define some fallback location somewhere?

Since I'm not really sure where this could be done, I have also added a similar issue here: platformio/platform-espressif8266#184

devyte · 2019-11-25T20:27:52Z

Savannah seems to be online now.

possible to define some archive somewhere as backup

The only thing I can think of would be to clone it into github, but that would require maintenance to keep it up to date, which we're not willing to do atm. If anyone can think of some other way (mirror? automated update?), please post it here.

Closing for now.

d-a-v · 2019-11-25T21:41:26Z

@TD-er
This repository and its submodules should be cloned not recursively,
with git submodule update --init
not with git submodule update --init --recursive
Upstream lwip being a submodule of lwip2 which is a submodule of this repo, it doesn't matter if savannah is unreachable at the time of cloning. The core uses precompiled version of lwip2 anyway.

TD-er · 2019-11-25T21:47:24Z

@d-a-v So that's a configuration error where?
In this repo, or in platformio?

d-a-v · 2019-11-25T21:59:07Z

It's the way this repo is cloned.
What you reported was also an issue in CI, solved like this (unfortunately Travis CI's submodule option is implicitely recursive so it is disabled and made with fingers instead)

Can you point me to the PIO git-setup scripts ?

TD-er · 2019-11-25T22:02:02Z

Can you point me to the PIO git-setup scripts ?

Well, that question alone I cannot truthfully answer, since I hardly understand myself how the PIO stuff is being handled.
I think the magic is happening in the builder folder of this repo: https://github.com/platformio/platform-espressif8266

mcspr · 2019-11-25T22:06:06Z

See https://github.com/platformio/platformio-core/blob/develop/platformio/vcsclient.py
afaik those are invoked when package points to a git url

d-a-v · 2019-11-25T22:12:08Z

@ivankravets
Do you think --recursive can be optional in these lines or per-project disabled ?

ivankravets · 2019-11-26T16:18:15Z

@d-a-v Oh, I'm not sure. I think it's better to use PRE script and clone framework before to "packages" folder. See http://docs.platformio.org/en/latest/projectconf/advanced_scripting.html

TD-er · 2019-11-30T17:24:52Z

The site for the LWIP git may be online, but more than 50% of the time it is queried, it will fail to download.
So a lot of my Travis builds fail on this LWIP install.

d-a-v · 2019-12-01T16:24:58Z

@TD-er the reason why we use this external repository is because it's the official one and we prefer to not maintain an up-to-date copy.
I understand your issue and we had it too, like mentioned above (I explained how we solved it)

We can

try and fix PIO scripts to not recursively download submodule
modify lwip2 to download lwIP upstream sources by script instead of by submodule

TD-er · 2019-12-01T20:31:23Z

@d-a-v
Is either of these 2 options something I should do/change myself? Or is it something you plan to change in the scripts for this repo?

d-a-v · 2019-12-04T22:33:13Z

@TD-er I'd prefer some help from PIO nerds :]

TD-er · 2019-12-04T23:23:43Z

Hmm it may be hard to find nerds here.... ;)

TD-er · 2019-12-06T15:54:05Z

Can this be re-opened, as it is really a problem.
It takes a lot of time and lots and lots of trial and error to get the repo to be initialized.

Submodule path 'tools/sdk/lwip2/builder': checked out '354887a25f83064dc0c795e11704190845812713'
Submodule 'lwip2-src' (https://git.savannah.nongnu.org/git/lwip.git) registered for path 'tools/sdk/lwip2/builder/lwip2-src'
Cloning into 'C:/Users/gijs/.platformio/packages/_tmp_installing-4ht_rpqq-package/tools/sdk/lwip2/builder/lwip2-src'...
fatal: unable to access 'https://git.savannah.nongnu.org/git/lwip.git/': The requested URL returned error: 502
fatal: clone of 'https://git.savannah.nongnu.org/git/lwip.git' into submodule path 'C:/Users/gijs/.platformio/packages/_tmp_installing-4ht_rpqq-package/tools/sdk/lwip2/builder/lwip2-src' failed
Failed to clone 'lwip2-src'. Retry scheduled
Cloning into 'C:/Users/gijs/.platformio/packages/_tmp_installing-4ht_rpqq-package/tools/sdk/lwip2/builder/lwip2-src'...
fatal: unable to access 'https://git.savannah.nongnu.org/git/lwip.git/': The requested URL returned error: 502
fatal: clone of 'https://git.savannah.nongnu.org/git/lwip.git' into submodule path 'C:/Users/gijs/.platformio/packages/_tmp_installing-4ht_rpqq-package/tools/sdk/lwip2/builder/lwip2-src' failed
Failed to clone 'lwip2-src' a second time, aborting
Submodule path 'tools/sdk/ssl/bearssl': checked out '89454af34e3e61ddfc9837f3da5a0bc8ed44c3aa'
Failed to recurse into submodule path 'tools/sdk/lwip2/builder'
Error: VCS: Could not process command ['git', 'clone', '--recursive', '--depth', '1', '--branch', '2.6.2', 'https://github.com/esp8266/Arduino.git', 'C:\\Users\\gijs\\.platformio\\packages\\_tmp_installing-4ht_rpqq-package']
The terminal process terminated with exit code: 1

Not sure why it keeps on trying to fetch it, as I have not removed my framework directories..
Maybe these get thrown away as soon as you upgrade platformIO?

Also just updating of platform io frameworks keeps failing:

(penv) PS C:\GitHub\TD-er\ESPEasy> pio update
**********************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Obsolete PIO Core v4.1.0 is used (previous was 4.1.1b5)
Please remove multiple PIO Cores from a system:
https://docs.platformio.org/page/faq.html#multiple-pio-cores-in-a-system
**********************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Updating contrib-piohome                 @ 3.0.1          [Up-to-date]
Updating contrib-pysite                  @ 2.38.191019    [Up-to-date]
Updating tool-unity                      @ 1.20403.0      [Up-to-date]
Updating tool-scons                      @ 3.30101.0      [Up-to-date]
Updating tool-clangtidy                  @ 1.90000.0      [Up-to-date]

Platform Manager
================
Platform Espressif 32
--------
Updating espressif32                     @ 1.11.1         [Up-to-date]
Updating toolchain-xtensa32              @ 2.50200.80     [Up-to-date]
Updating framework-arduinoespressif32    @ 2.10004.191002 [Up-to-date]
Updating tool-esptoolpy                  @ 1.20600.0      [Up-to-date]
Updating tool-mkspiffs                   @ 2.230.0        [Up-to-date]

Platform Espressif 32
--------
Updating espressif32                     @ 1.10.0         [Detached]
Updating toolchain-xtensa32              @ 2.50200.80     [Up-to-date]
Updating framework-arduinoespressif32    @ 2.10003.190916 [Up-to-date]
Updating tool-esptoolpy                  @ 1.20600.0      [Up-to-date]
Updating tool-mkspiffs                   @ 2.230.0        [Up-to-date]

Platform Espressif 32
--------
Updating espressif32                     @ 1.11.0         [Detached]
Updating toolchain-xtensa32              @ 2.50200.80     [Up-to-date]
Updating framework-arduinoespressif32    @ 2.10004.191002 [Up-to-date]
Updating tool-esptoolpy                  @ 1.20600.0      [Up-to-date]
Updating tool-mkspiffs                   @ 2.230.0        [Up-to-date]

Platform Espressif 8266
--------
Updating espressif8266                   @ 2.3.1          [Up-to-date]
Updating toolchain-xtensa                @ 2.40802.191122 [Up-to-date]
Updating framework-arduinoespressif8266  @ 2.20602.191128 [Up-to-date]
Updating tool-esptool                    @ 1.413.0        [Up-to-date]
Updating tool-esptoolpy                  @ 1.20800.0      [Up-to-date]
Updating tool-mkspiffs                   @ 1.200.0        [Up-to-date]

Platform Espressif 8266
--------
Updating espressif8266                   @ 2.2.3          [Detached]
Updating toolchain-xtensa                @ 2.50400.170719 [Up-to-date]
Updating tool-esptool                    @ 1.413.0        [Up-to-date]
Updating tool-esptoolpy                  @ 1.20600.0      [Up-to-date]
Updating tool-mkspiffs                   @ 1.200.0        [Up-to-date]

Platform Espressif 8266
--------
Updating espressif8266                   @ 7507605        [Up-to-date]
Updating toolchain-xtensa                @ 1.40802.0      [Up-to-date]
Updating tool-esptool                    @ 1.413.0        [Up-to-date]
Updating tool-mkspiffs                   @ 1.200.0        [Up-to-date]

Platform Espressif 8266
--------
Updating espressif8266                   @ 043f5fd        [Detached]
Updating toolchain-xtensa                @ 2.50400.170719 [Up-to-date]
Updating framework-arduinoespressif8266  @ 2.20601.191124 [Up-to-date]
Updating tool-esptool                    @ 1.413.0        [Up-to-date]
Updating tool-esptoolpy                  @ 1.20800.0      [Up-to-date]
Updating tool-mkspiffs                   @ 1.200.0        [Up-to-date]

Platform Espressif 8266
--------
Updating espressif8266                   @ e687cfb        [Detached]
Updating toolchain-xtensa                @ 2.50400.170719 [Up-to-date]
Updating tool-esptool                    @ 1.413.0        [Up-to-date]
Updating tool-esptoolpy                  @ 1.20600.0      [Up-to-date]
Updating tool-mkspiffs                   @ 1.200.0        [Up-to-date]

Platform Espressif 8266 (Stage)
--------
Updating espressif8266                   @ dc6771b        [Up-to-date]
Updating toolchain-xtensa                @ 2.40802.191122 [Up-to-date]
Updating framework-arduinoespressif8266  @ 5aefed2        [8a6a7f7]
git version 2.21.0.windows.1
Fetching submodule libraries/ESP8266SdFat
Fetching submodule libraries/LittleFS/lib/littlefs
Fetching submodule libraries/SoftwareSerial
Fetching submodule tools/esptool
Fetching submodule tools/pyserial
Fetching submodule tools/sdk/lwip2/builder
Fetching submodule tools/sdk/lwip2/builder/lwip2-src
fatal: unable to access 'https://git.savannah.nongnu.org/git/lwip.git/': The requested URL returned error: 502
Fetching submodule tools/sdk/ssl/bearssl
Error: VCS: Could not process command ['git', 'pull', '--recurse-submodules']
(penv) PS C:\GitHub\TD-er\ESPEasy>

OK, now it really is messed up here.
I cannot build ANYTHING anymore due to this damned repo :(
Please tell me where to look, as I may loose the carefully freed weekend for programming.

mcspr · 2019-12-06T16:03:46Z

Perhaps "modify lwip2 to download lwIP upstream sources by script instead of by submodule" is the correct solution? Maybe even other things related to the tree only tangentially and are useless for a basic user of the repo where the goal is sketch not the Core development.

e.g. things like [submodule "libraries/SoftwareSerial"] are, but both bearssl and lwip are already shipped as pre-built libraries

TD-er · 2019-12-06T16:12:12Z

@ivankravets Can you give me any pointers here, where to look?
I am willing to clone everything I needed to some server I have, just to get this working again.

devyte closed this as completed Nov 25, 2019

earlephilhower mentioned this issue Dec 1, 2019

Need to rebuild eboot to pull in PR #6832 changes #6863

Closed

TD-er mentioned this issue Dec 6, 2019

Cannot build minimal_core_261_ESP8266_1M_OTA letscontrolit/ESPEasy#2801

Closed

d-a-v mentioned this issue Dec 7, 2019

lwip2 updates: no more git sub-sub-module deps, faster checksum, backlog limitation and other fixes #6887

Merged

d-a-v reopened this Jan 24, 2020

d-a-v added the fixed-by-PR label Feb 5, 2020

d-a-v closed this as completed in #6887 Feb 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to clone full platform (LWIP2 git offline) #6832

Unable to clone full platform (LWIP2 git offline) #6832

TD-er commented Nov 24, 2019 •

edited

devyte commented Nov 25, 2019

d-a-v commented Nov 25, 2019 •

edited

TD-er commented Nov 25, 2019

d-a-v commented Nov 25, 2019

TD-er commented Nov 25, 2019

mcspr commented Nov 25, 2019

d-a-v commented Nov 25, 2019

ivankravets commented Nov 26, 2019

TD-er commented Nov 30, 2019

d-a-v commented Dec 1, 2019

TD-er commented Dec 1, 2019

d-a-v commented Dec 4, 2019

TD-er commented Dec 4, 2019

TD-er commented Dec 6, 2019 •

edited

mcspr commented Dec 6, 2019

TD-er commented Dec 6, 2019

Unable to clone full platform (LWIP2 git offline) #6832

Unable to clone full platform (LWIP2 git offline) #6832

Comments

TD-er commented Nov 24, 2019 • edited

devyte commented Nov 25, 2019

d-a-v commented Nov 25, 2019 • edited

TD-er commented Nov 25, 2019

d-a-v commented Nov 25, 2019

TD-er commented Nov 25, 2019

mcspr commented Nov 25, 2019

d-a-v commented Nov 25, 2019

ivankravets commented Nov 26, 2019

TD-er commented Nov 30, 2019

d-a-v commented Dec 1, 2019

TD-er commented Dec 1, 2019

d-a-v commented Dec 4, 2019

TD-er commented Dec 4, 2019

TD-er commented Dec 6, 2019 • edited

mcspr commented Dec 6, 2019

TD-er commented Dec 6, 2019

TD-er commented Nov 24, 2019 •

edited

d-a-v commented Nov 25, 2019 •

edited

TD-er commented Dec 6, 2019 •

edited