Vagrant build environment

[TD-er]

A warning to start with:

DO NOT try to run HyperV as a test.

See: #2594 (comment) to help fully disabling HyperV again.
For now, HyperV is not supported as build platform since we have no easy way to share the build files.

---

As shortly introduced here: #2590 (comment)

This is the first initial test to see if using Vagrant is a viable option for quickly building your own binaries.

To get started:

• checkout the latest from the mega branch and go to tools/vagrant.
• Read the (very short) readme about what to install
• Go to that folder with a terminal/shell.

In Windows it can be CMD or Windows PowerShell and in Linux it can be any terminal.
N.B. in Windows you can also quickly open a command shell by holding the shift and right click on the folder and select the Windows Power Shell.

In the opened shell:

• Type vagant up

Please let me know if anything fails.
If it is finished building something in the newly created build folder, then you can type vagrant halt in the same window you typed vagrant up.

Ideas for improving on this concept:

• Parse a TXT (or YAML) file describing a build, or set of builds
• Install Apache in the VM and run a web page where you can select your own set of builds
• Accept JSON in the same Apache to let a node request its own new build (this requires the VM to have its own IP on the network)
• Select a Git tag or PR to build. (the build script in tools already has a -p flag for building a PR)

For debugging:
When the Vagrant box is running, you can connect to it using Putty (or any other ssh client).

• User: vagrant
• Pass: vagrant
• Hostname: localhost
• Port: 2222 (may change if you run multiple instances, see output on terminal)

The Github repository is cloned into:

• Dir: ~/GitHub/letscontrolit/ESPEasy
• User: vagrant

At boot of the Vagrant VM, the file bootstrap.sh is copied to the home directory of the vagrant user and stripped of typical Windows line endings (if present) (CRLF => Unix line endings)
This is then executed, so for now, you can test with it to see if it needs to build some other environment.

Please note there is a shared folder between the Vagrant VM and your own OS.
This is the same folder as where the Vagrantfile (no extension) and bootstrap.sh are located.
From within the Vagrant VM these are available as a mount on /vagrant

The build is zipped like any nightly build and then this zip file is moved to the shared folder in a separate directory called build. (will be created by the script)
So from within Vagrant this is /vargant/build/

One last note on the Vagrantfile. Do not use tabs in there, only spaces.
On every call of the vagrant command, this file will be checked for errors.

To start over with a clean instance, call:

• vagrant destroy

So in short, the commands:

• vagrant up
• vagrant up --provision (to run the apt-get install commands again)
• vagrant halt
• vagrant destroy

The build script will not (yet) update the Git Repository if the directory is already present.
So either start over, or remove the Git directory or run from within ~/GitHub/letscontrolit/ESPEasy :

• git fetch
• git pull

[LeeNX]

@TD-er I have an ascii screen recording, about 275KB ... What is easiest to get this too you?

From what I can see, I think the problem is an missing apt-get update before doing apt-get install, as I remember seeing errors with missing debs.

[uzi18]

On linux tools/build_espeasy.sh should still work easily without vargant

virtualbox has got recording feature

[TD-er]

There is a call to apt-get update in the Vagrant file:

  config.vm.provision "shell", inline: <<-SHELL
    apt-get update
    apt-get install -y python-minimal virtualenv build-essential zip binutils
  SHELL

Maybe you could run vagrant up --provision on that one?
It should be done the first time you call a vagrant up.

[LeeNX]

The error I saw might have been my VB plugin to upgrade the VB Tools.

But later, I have the following red/errors

    default: Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 virtualenv all 15.1.0+ds-1.1 [4,476 B]
    default: Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 zip amd64 3.0-11build1 [167 kB]

The following in red default: dpkg-preconfigure: unable to re-open stdin: No such file or directory

    default: Fetched 2,253 kB in 8s (297 kB/s)
    default: Selecting previously unselected package python-pip-whl.

Later, but I think it's just a warning ...

    default: Cloning into '/home/vagrant/GitHub/letscontrolit/ESPEasy'...
    default: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support

then alot of red ...

    default: Platform Manager
    default: ================
    default:
    default: /home/vagrant/.platformio/packages/framework-arduinoespressif8266*//cores/esp8266/Esp.cpp
    default: : No such file or directory
    default: Patching /home/vagrant/.platformio/packages/framework-arduinoespressif8266*/
    default: patch: **** Can't change to directory '/home/vagrant/.platformio/packages/framework-arduinoespressif8266*/'
    default:  : No such file or directory
    default: Error: Unknown environment names 'custom_ESP8266_4M'. Valid names are 'memanalyze_ESP8266, normal_core_241_ESP8266_4M1M, dev_ESP8266_4M1M, normal_core_260_sdk222_alpha_ESP8266_4M1M, hard_SONOFF_POW_4M1M, normal_WROOM02_2M, normal_ESP8266_1M, test_core_260_sdk222_alpha_ESP8266_16M, esp32test_1M8_partition, test_core_260_sdk222_alpha_ESP8266_4M1M, normal_core_260_sdk222_alpha_ESP8266_16M, hard_other_POW_ESP8285_1M, minimal_IRext_ESP8266_4M2M, normal_core_241_ESP8266_1M, test_core_252_ESP8266_16M, minimal_core_252_ESP8285_1M_OTA, normal_core_242_ESP8266_1M, test_core_252_ESP8266_4M1M, esp-wrover-kit_test_1M8_partition, test_ESP8266_4M_VCC, hard_core_252_Shelly_1_2M256, normal_ESP8266_1M_VCC, custom_ESP8266_4M1M, test_core_242_ESP8266_4M1M, normal_core_252_ESP8266_16M, normal_ESP8285_1M, test_core_260_sdk3_alpha_ESP8266_4M1M, hard_Ventus_W266, normal_core_252_ESP8266_4M1M, minimal_IRext_ESP8266_4M1M, hard_core_252_SONOFF_POW_4M1M, normal_ESP8266_4M1M, normal_core_252_ESP8266_1M, minimal_core_242_ESP8285_1M_OTA, minimal_core_242_ESP8266_1M_OTA, normal_core_252_WROOM02_2M256, minimal_core_252_ESP8266_1M_OTA, hard_core_252_other_POW_ESP8285_1M, test_core_260_sdk3_alpha_ESP8266_16M, minimal_IRext_ESP8266_1M'
    default: /home/vagrant/bootstrap.sh: line 56: DESCRIPTION: command not found
    default: stat:
    default: cannot stat '.pio/build/custom_ESP8266_4M1M/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment custom_ESP8266_4M1M for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/custom_ESP8266_4M1M/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/custom_ESP8266_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_custom_ESP8266_4M1M.bin'
    default: mv:
    default: cannot stat '.pio/build/custom_ESP8266_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_custom_ESP8266_4M1M.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/dev_ESP8266_4M1M/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment dev_ESP8266_4M1M for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/dev_ESP8266_4M1M/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/dev_ESP8266_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_dev_ESP8266_4M1M.bin'
    default: mv:
    default: cannot stat '.pio/build/dev_ESP8266_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_dev_ESP8266_4M1M.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/esp-wrover-kit_test_1M8_partition/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment esp-wrover-kit_test_1M8_partition for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/esp-wrover-kit_test_1M8_partition/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/esp-wrover-kit_test_1M8_partition/ESP_Easy_mega-20190830-16-ga68dc43_esp-wrover-kit_test_1M8_partition.bin'
    default: mv:
    default: cannot stat '.pio/build/esp-wrover-kit_test_1M8_partition/ESP_Easy_mega-20190830-16-ga68dc43_esp-wrover-kit_test_1M8_partition.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/esp32test_1M8_partition/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment esp32test_1M8_partition for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/esp32test_1M8_partition/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/esp32test_1M8_partition/ESP_Easy_mega-20190830-16-ga68dc43_esp32test_1M8_partition.bin'
    default: mv:
    default: cannot stat '.pio/build/esp32test_1M8_partition/ESP_Easy_mega-20190830-16-ga68dc43_esp32test_1M8_partition.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/hard_SONOFF_POW_4M1M/firmware.bin'
    default:
    default: ### Deploying environment hard_SONOFF_POW_4M1M for version mega-20190830-16-ga68dc43
    default: : No such file or directory
    default: cp:
    default: cannot stat '.pio/build/hard_SONOFF_POW_4M1M/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:     with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError: [Errno 2] No such file or directory: '.pio/build/hard_SONOFF_POW_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_hard_SONOFF_POW_4M1M.bin'
    default: mv:
    default: cannot stat '.pio/build/hard_SONOFF_POW_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_hard_SONOFF_POW_4M1M.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/hard_Ventus_W266/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment hard_Ventus_W266 for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/hard_Ventus_W266/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/hard_Ventus_W266/ESP_Easy_mega-20190830-16-ga68dc43_hard_Ventus_W266.bin'
    default: mv:
    default: cannot stat '.pio/build/hard_Ventus_W266/ESP_Easy_mega-20190830-16-ga68dc43_hard_Ventus_W266.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/hard_core_252_SONOFF_POW_4M1M/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment hard_core_252_SONOFF_POW_4M1M for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/hard_core_252_SONOFF_POW_4M1M/firmware.bin': No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/hard_core_252_SONOFF_POW_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_hard_core_252_SONOFF_POW_4M1M.bin'
    default: mv:
    default: cannot stat '.pio/build/hard_core_252_SONOFF_POW_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_hard_core_252_SONOFF_POW_4M1M.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/hard_core_252_Shelly_1_2M256/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment hard_core_252_Shelly_1_2M256 for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/hard_core_252_Shelly_1_2M256/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/hard_core_252_Shelly_1_2M256/ESP_Easy_mega-20190830-16-ga68dc43_hard_core_252_Shelly_1_2M256.bin'
    default: mv:
    default: cannot stat '.pio/build/hard_core_252_Shelly_1_2M256/ESP_Easy_mega-20190830-16-ga68dc43_hard_core_252_Shelly_1_2M256.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/hard_core_252_other_POW_ESP8285_1M/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment hard_core_252_other_POW_ESP8285_1M for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/hard_core_252_other_POW_ESP8285_1M/firmware.bin': No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/hard_core_252_other_POW_ESP8285_1M/ESP_Easy_mega-20190830-16-ga68dc43_hard_core_252_other_POW_ESP8285_1M.bin'
    default: mv:
    default: cannot stat '.pio/build/hard_core_252_other_POW_ESP8285_1M/ESP_Easy_mega-20190830-16-ga68dc43_hard_core_252_other_POW_ESP8285_1M.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/hard_other_POW_ESP8285_1M/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment hard_other_POW_ESP8285_1M for version mega-20190830-16-ga68dc43
    default: cp: cannot stat '.pio/build/hard_other_POW_ESP8285_1M/firmware.bin': No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file

Might be just easier with the screen recording?

[TD-er]

Just make sure you are running the current state of the Github.
So I guess perform a vagrant destroy on the VM, make sure you are up to date with the current HEAD of the Git branch and do a new vagrant up
I think you may have tried the first time just between 2 commits.

[LeeNX]

Clean up the test, and did a git pull and looks much better. Testing again, just to make sure ... But I think this is working as expected!!! Thanks @TD-er

[LeeNX]

Clean re-run works as expected

    default: hard_Ventus_W266                           IGNORED
    default: ========================= 1 succeeded in 00:01:30.255 =========================
    default:
    default: ### Deploying environment custom_ESP8266_4M1M for version 20190902_vagrant
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default:
    default:
    default: BINARY PART
    default: Segments: 0x1
    default: SEGMENT 0: memory position: 0x4010f000 to 0x4010f568 length: 0x568
    default:
    default:
    default: BINARY PART
    default: Segments: 0x5
    default: SEGMENT 1: memory position: 0x40201010 to 0x402c2984 length: 0xc1974
    default: SEGMENT 2: memory position: 0x40100000 to 0x401000b0 length: 0xb0
    default: SEGMENT 3: memory position: 0x401000b0 to 0x40107684 length: 0x75d4
    default: SEGMENT 4: memory position: 0x3ffe8000 to 0x3ffe86d0 length: 0x6d0 <-- CRC is here. <-- filename is here.
    default: SEGMENT 5: memory position: 0x3ffe86e0 to 0x3ffe9ac0 length: 0x13e0
    default: hash includes segments: 1 2
    default: MD5 hash: a39c54476156b92412b2f174855b7eb9
    default:
    default: writing output file:
    default: .pio/build/custom_ESP8266_4M1M/ESP_Easy_20190902_vagrant_custom_ESP8266_4M1M.bin
    default:
    default: ### Creating zip archive

[LeeNX]

@TD-er ping me if you need anything else tested, else I think the base of the vagrant build system seems to work.

[TD-er]

Maybe @micropet (see #2590 (comment) ) can test by changing the version of the Vagrant image to see if it also works on HyperV. Described here

[workgroupengineering]

Hi @TD-er ,
Why didn't you use Docker?

[Grovkillen]

@workgroupengineering Why should we? Please give pros and cons.

[TD-er]

Just as a reply to @micropet on his post here: #2590 (comment)

You do need any of the supported visualization platforms.
If you plan to use HyperV, then you don't need to install Virtualbox (I would say you should even not install it)
I don't have any experience with HyperV, but what I've read from it, you need administrator rights to create a new instance, so I guess you should start the "vagrant up" from a shell with administrator rights.
Right now, we don't have the click-and-build site yet, so this is just a test to see if this is the right way to go.
It will just build a predefined build. But later on it is planned to make this a very simple build env. to make your own build.
Let's call it a pre-alpha stage :)

@workgroupengineering
The reason I chose Vagrant, over Docker is very simple.
I do know Vagrant and I have 0 experience with Docker.
And also some of the already existing scripts we use for the nightly builds and for making a quick test build is based on running in Ubuntu. So for a quick-and-dirty test using Vagrant is by far the easiest for me to get some environment setup.

Also this is very easy to explain to the inexperienced and took me about 3 - 4 hours yesterday to setup.
I guess it would take me more to read into Docker, create and test images and find a way to make running Docker images easy on all platforms people use.
I know Tasmota does have a Docker image to build and I know you can run Docker images even on your NAS (the ones running x86 CPUs), so maybe for a future build env. it could be a good choice.
For now just let's focus on one and see if people actually can run it.

[LeeNX]

So I can't demand a DOS build env? Hahaha ... Just adding my +1, because this will/does make it easier for me to build custom firmware for the ESPEasy eco system, thanks @TD-er

[micropet]

Thanks for the information. I can not get any further.
I also have the same error message with Admin right:

d:\Soft\ESPEasy-mega\tools\vagrant>vagrant up
Vagrant failed to initialize at a very early stage:

There was an error while executing VBoxManage, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["--version"]

Stderr: Der Befehl ""c:\Program Files\Oracle\VirtualBox\VBoxManage.exe"" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.

[TD-er]

OK, I think you'll also need to patch the Vagrantfile then to set a provider.
I can make a test Vagrantfile tomorrow and if that does work, we can have 2 definitions in the Vagrantfile so you just have to add an extra parameter to the vagrant up command
For example vagrant up hyperv (or whatever name we will give the other definition)

[micropet]

Great.

[TD-er]

Have a look at Vagrant providers
Especially the "Default Provider" section.

It seems like we can add a set of providers here.
But as a first test, so we don't have to change anything, you could try:
vagrant up --provider=hyperv

Edit:
You still need to change the Vagrantfile like described here: #2590 (comment)
Or maybe this will also work:

vagrant up --provider=hyperv --box-version 201812.27.0

[micropet]

With the original Vagrant File I get the following message:

vagrant up --provider=hyperv
Bringing machine 'default' up with 'hyperv' provider...
==> default: Verifying Hyper-V is enabled...
==> default: Verifying Hyper-V is accessible...
==> default: Importing a Hyper-V instance
default: Creating and registering the VM...
default: Successfully imported VM
default: Please choose a switch to attach to your Hyper-V instance.
default: If none of these are appropriate, please open the Hyper-V manager
default: to create a new virtual switch.
default:
default: 1) Default Switch
default: 2) V-Switch
default:
default: What switch would you like to use? 2
default: Configuring the VM...
==> default: Starting the machine...
==> default: Waiting for the machine to report its IP address...
default: Timeout: 120 seconds
default: IP: 192.168.0.236
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 192.168.0.236:22
default: SSH username: vagrant
default: SSH auth method: private key
default:
default: Vagrant insecure key detected. Vagrant will automatically replace
default: this with a newly generated keypair for better security.
default:
default: Inserting generated public key within guest...
default: Removing insecure key from the guest if it's present...
default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!

Vagrant requires administrator access for pruning SMB shares and
may request access to complete removal of stale shares.
==> default: Preparing SMB shared folders...
default: You will be asked for the username and password to use for the SMB
default: folders shortly. Please use the proper username/password of your
default: account.
default:
default: Username: xxxxxx
default: Password (will be hidden):

Vagrant requires administrator access to create SMB shares and
may request access to complete setup of configured shares.
==> default: Mounting SMB shared folders...
We couldn't detect an IP address that was routable to this
machine from the guest machine! Please verify networking is properly
setup in the guest machine and that it is able to access this
host.

As another option, you can manually specify an IP for the machine
to mount from using the smb_host option to the synced folder.

[workgroupengineering]

Hi @Grovkillen ,

Why docker?

• runs on Raspberry Pi !!!!
• has become a world standard
• there is so much documentation on the net
• works with both virtualbox and hyper-v without any configuration

Why not docker?

• high learning curve

[TD-er]

• runs on Raspberry Pi !!!!

Compiling on a R'pi is not really fun.
But it is one of the most available "servers" among users.
I think the build script we use, may also run on a Pi, so I guess we can check where things need to be adapted to work on a Pi too.

Like I said, we can later look into it, but for now the 'con' outweighs the 'pros'.
First let's see what is a setup with a high UX and is easy to use and then we can focus on other container techniques.

[TD-er]

Vagrant requires administrator access to create SMB shares and
may request access to complete setup of configured shares.
==> default: Mounting SMB shared folders...
We couldn't detect an IP address that was routable to this
machine from the guest machine! Please verify networking is properly
setup in the guest machine and that it is able to access this
host.

As another option, you can manually specify an IP for the machine
to mount from using the smb_host option to the synced folder.

Right now I did not add the option for routable IPs, since it is for now a test setup to see where people run into.
Also making a VM available on the entire network does require some extra precautions.
What if we 'depend' on it and there is no DHCP running?
How about security? (default no problem, but if we add web services later it might)
If it is routable, then people will make it run 24/7 on their network.

So for the first test I explicitly did not add the routed IP option.

Did it build a version for you?

[uzi18]

on linux (rpi) you not need this, just use buildESPEasy.sh script it use virtualenv so will no mess with your system/profile dir

[TD-er]

Only thing may be that the reported packages you should install using apt-get may have different names on the Pi, since it is Debian based.

[micropet]

Good question. Not in my ESPEasy folder.

Where should the version be found?

[TD-er]

Good question. Not in my ESPEasy folder.

Where should the version be found?

In a newly created build folder at the same location where the Vagrantfile is located.

N.B. you should also have the bootstrap.sh file in the same directory as the Vagrantfile or else it will do nothing.

Maybe you have run it multiple times already, so it is not considering it a provision step.

vagrant halt
vagrant destroy
vagrant up --provider=hyperv --box-version 201812.27.0

You can also trigger a provision with adding --provision

[TD-er]

Where do you try to connect to?
It should be running an SSH on port 2222 of the VM, so you should be able to SSH into localhost with port 2222

If that doesn't work, please show all output of the provisioning of the VM.
Otherwise we keep on guessing what's wrong here.

[micropet]

I connect on the IP that is displayed to me and on port 22.

==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 192.168.0.239:22
default: SSH username: vagrant
default: SSH auth method: private key

here the complete Output:

vagrant>vagrant up
Bringing machine 'default' up with 'hyperv' provider...
==> default: Verifying Hyper-V is enabled...
==> default: Verifying Hyper-V is accessible...
==> default: Importing a Hyper-V instance
default: Creating and registering the VM...
default: Successfully imported VM
default: Please choose a switch to attach to your Hyper-V instance.
default: If none of these are appropriate, please open the Hyper-V manager
default: to create a new virtual switch.
default:
default: 1) V-Switch
default: 2) Default Switch
default:
default: What switch would you like to use? 1
default: Configuring the VM...
==> default: Starting the machine...
==> default: Waiting for the machine to report its IP address...
default: Timeout: 120 seconds
default: IP: 192.168.0.239
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 192.168.0.239:22
default: SSH username: vagrant
default: SSH auth method: private key
default:
default: Vagrant insecure key detected. Vagrant will automatically replace
default: this with a newly generated keypair for better security.
default:
default: Inserting generated public key within guest...
default: Removing insecure key from the guest if it's present...
default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Preparing SMB shared folders...
default: You will be asked for the username and password to use for the SMB
default: folders shortly. Please use the proper username/password of your
default: account.
default:
default: Username: vagrant
default: Password (will be hidden):

Vagrant requires administrator access to create SMB shares and
may request access to complete setup of configured shares.
==> default: Mounting SMB shared folders...
We couldn't detect an IP address that was routable to this
machine from the guest machine! Please verify networking is properly
setup in the guest machine and that it is able to access this
host.

As another option, you can manually specify an IP for the machine
to mount from using the smb_host option to the synced folder.

d:\Soft\ESPEasy-mega\tools\vagrant>

I try with password and username vagrant

[TD-er]

I still have the impression something is really different with that image.
It looks like it is quite a different image used for HyperV.

• It has a routable IP and running ssh on it instead of mapping it to port 2222
• Does not seem to have a mount point on /vagrant for whatever reason.

[micropet]

Yes, that's how it looks.
I finish the experiments with Vagrant.
I'll do some googling and then I'll wait until someone gets it to work.

[uzi18]

works for me :)

==> default: Successfully added box 'bento/ubuntu-18.04' (v201906.18.0) for 'virtualbox'!
==> default: Importing base box 'bento/ubuntu-18.04'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'bento/ubuntu-18.04' version '201906.18.0' is up to date...
==> default: Setting the name of the VM: vagrant_default_1567722187644_49884
Vagrant is currently configured to create VirtualBox synced folders with
the `SharedFoldersEnableSymlinksCreate` option enabled. If the Vagrant
guest is not trusted, you may want to disable this option. For more
information on this option, please refer to the VirtualBox manual:

  https://www.virtualbox.org/manual/ch04.html#sharedfolders

This option can be disabled globally with an environment variable:

  VAGRANT_DISABLE_VBOXSYMLINKCREATE=1

or on a per folder basis within the Vagrantfile:

  config.vm.synced_folder '/host/path', '/guest/path', SharedFoldersEnableSymlinksCreate: false
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection reset. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...

[TD-er]

I think this part is the key here:

'bento/ubuntu-18.04' (v201906.18.0) for 'virtualbox'!

I will later this week or beginning next week enable HyperV here on my computer.
As I understood it, it is default enabled on Enterprise versions of Windows and some sources claim it is also default enabled on Windows 10 pro. But my laptop came with Windows 10 pro, but it is not enabled here by default.
Maybe some change HP made with their pre-install image?

[uzi18]

you need to call this in command line first: Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All but beware it will break virtualbox ;)

[TD-er]

OK, it is now downloading (slowly)....

PS C:\GitHub\TD-er\ESPEasy\tools\vagrant> vagrant up --provider=hyperv
Bringing machine 'default' up with 'hyperv' provider...
==> default: Verifying Hyper-V is enabled...
==> default: Verifying Hyper-V is accessible...
==> default: Box 'bento/ubuntu-18.04' could not be found. Attempting to find and install...
    default: Box Provider: hyperv
    default: Box Version: >= 0
==> default: Loading metadata for box 'bento/ubuntu-18.04'
    default: URL: https://vagrantcloud.com/bento/ubuntu-18.04
==> default: Adding box 'bento/ubuntu-18.04' (v201812.27.0) for provider: hyperv
    default: Downloading: https://vagrantcloud.com/bento/boxes/ubuntu-18.04/versions/201812.27.0/providers/hyperv.box
    default: Download redirected to host: vagrantcloud-files-production.s3.amazonaws.com
    default: Progress: 39% (Rate: 1453k/s, Estimated time remaining: 0:02:15)

[TD-er]

OK, some things are more clear now....

Vagrant HyperV limitations
So it does ask for a SMB share when provisioned for the first time:

==> default: Successfully added box 'bento/ubuntu-18.04' (v201812.27.0) for 'hyperv'!
==> default: Importing a Hyper-V instance
    default: Creating and registering the VM...
    default: Successfully imported VM
    default: Configuring the VM...
==> default: Starting the machine...
==> default: Waiting for the machine to report its IP address...
    default: Timeout: 120 seconds
    default: IP: 172.17.99.235
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 172.17.99.235:22
    default: SSH username: vagrant
    default: SSH auth method: private key
    default:
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default:
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Preparing SMB shared folders...
    default: You will be asked for the username and password to use for the SMB
    default: folders shortly. Please use the proper username/password of your
    default: account.
    default:
    default: Username:

However, I don't have a SMB share defined on my host and I'm not sure if the defaults of Windows 10 will allow me to create a SMB/CIFS share compatible with Samba on Linux.
Also the generated IP address used in the VM does not have access to the network (192.168.1.x) to connect to a NAS.
This makes it quite useless with HyperV enabled, unless we do not need a shared folder to store the built files.
Problem remains that the ESP nodes also cannot access the VM to ask for a built file. (e.g. when running Apache in the VM)

The reason why I wanted to use a shared folder is to allow for easy definition of build parameters in a text file.
If I cannot access the shared folder, then the file must be handed via some URL or via a browser, which makes it rather complicated again.

So for now, HyperV is not (yet) a supported platform for this use case.
It has been made way too complicated by MS.

[TD-er]

OK, one big fat warning here.

DO NOT try to run HyperV as a test.

Disabling it, does also disable other stuff, so even after several reboots, it will not run any Virtualbox environment anymore.
Virtualbox removed, rebooted, reinstalled etc. but it has mangled in some way my setup so I can now no longer run Virtualbox on my machine.

Also tried these steps
but it still claims:

Hyper-V Requirements: A hypervisor has been detected. Features required for Hyper-V will not be displayed.

I have no clue anymore whatsoever and right now I really don't know if I should continue with this Vagrant option, which was meant to make stuff easy for people.
But right now it already is taking way too many hours of Windows crap.

Edit:
This seems to disable HyperV:

• Remove all virtualization tools like Virtualbox
• Reboot
• Start PowerShell as administrator
• run bcdedit /set hypervisorlaunchtype off
• power down (not reboot)
• Power on
• Check with PowerShell as administrator systeminfo.exe

Hyper-V Requirements:      VM Monitor Mode Extensions: Yes
                           Virtualization Enabled In Firmware: Yes
                           Second Level Address Translation: Yes
                           Data Execution Prevention Available: Yes

This should allow again to run VirtualBox etc. at full speed (not software emulated virtualization).

[micropet]

Oh man, I've done you a lot of work, sorry.
But now we have become smarter. :)

[TD-er]

Oh man, I've done you a lot of work, sorry.
But now we have become smarter. :)

Yep, that's the only way to learn things, by failing.
If all went well at the first attempt, we would not have learned anything.
This makes running ESPeasy such an educational project... oh wait....

[uzi18]

this is exactly what have found some posts up about work on hyperv and virtualbox on one machine, you need to add menu to choose before boot what you want to use

[TD-er]

Well, that may already be outdated information, since Virtualbox also does seem to support HyperV as a hypervisor.
But still, if it does not allow to add a shared directory or allows to define the network topology from within the Vagrantfile, then we still have to jump through too many hoops to get it to work.
Then starting "Ubuntu on Windows" (from the Windows store) is maybe easier and I tested it, that one still works regardless the presence or absence of HyperV on the system.
It is also not running on top of a hypervisor, but only as an executable in Windows.

But at least it has made it clear to me that Vagrant does not work the same on any existing Windows install out there.

[ghtester]

I have tested the Vagrant environment on Ubuntu 18.04 LTS with Virtualbox, it looks to be working fine but I am missing an info how to configure the plugins custom selection or change the preselected ESP Easy firmware compilation list. Could you please direct me to the most accurate information source? Thanks in advance!

[TD-er]

@ghtester You're not only missing the instructions, but also the functionality to make it easy to select.
This was just a test to see if it would work without much hassle on most computers.
It currently just fetches the ESPeasy Github repo, setup Python virtual environment and build the "custom" build, zip it and copy to the build folder.
Nothing more right now.

Like I said, it is until now just a test setup to see what problems do arise.
What is built is defined in the bootstrap.sh file.
Later I will add the option to parse a separate file with a list of plugins and controllers to see if that's usable enough and also try to make a web page served from the VM to trigger a build.

But I thought to have it tested first instead of making a completely fool proof system which takes 100's of hours to develop only to find out almost anybody had issues with starting a Vagrant environment.

[LeeNX]

You build it and they will come! ... And now it has a life of it's own!! ;-)

Thanks @TD-er ... I hope the M$ Hyper-V problem does not slow or stop this option to build custom firmware. So far this has been the easiest for this n00b to build a test custom firmware.

Almost a single click/command, thou some deps are needed, which in my case are already in place. I don't believe this option will be for everybody, but I really hope there might be enough energy and time to take this to the next level.

All said by a n00b that can mostly test and help a little with debugging and poking fun at the guys that paid for their OS ... Last part a joke ... }-)

[uzi18]

just add test if custom-build.h exist so copy it to source dir and use it

[TD-er]

@uzi18 That's a pretty neat one :)

[TD-er]

Last night I added the option to use your own Custom.h file. (mind the capital C)
You can copy the Custom-sample.h file and copy/rename it to Custom.h in the tools/vagrant folder.

I did include all plugin/controller/notifier defines with comment // in front of them.
So just un-comment them and you should be presented with a build of your own.
You can even set your own WiFi credentials there or other defaults.

[LeeNX]

@TD-er - Is the Custom-sample.h expected to work without any changes?

    default: ========================= [FAILED] Took 74.95 seconds =========================
    default: Environment                                Status    Duration
    default: -----------------------------------------  --------  ------------
    default: custom_ESP8266_4M1M                        FAILED    00:01:14.947
    default: memanalyze_ESP8266                         IGNORED

Also quite a few warnings, is it possible to change the output text for warnings to yellow? Would be easier to spot real errors in red.

Really looking forward to easy builds ... And now with pre-setup configs, this is just awesome!

[LeeNX]

Answering my own question, by experimenting, still could be wrong, the shipped Custom-sample.h is not a working setup. Adding a plugin and controller completed a build.

An idea might be to have a default working build to start with - ie Custom-sample-default.h. I always find it easier to know that my first test is going to build without any changes. Then tweak incrementally. Gives me a sense of progress.

[TD-er]

I can add a version which will build a very minimal version.
The sample version does not include any plugins or controllers, which may indeed lead to warnings for unused variables.

Also there must be some extra feedback if a build failed, why it failed.
For example if a file has become too large, it will not be included in the ZIP file, but without feedback you wouldn't know.
So I think I will output the build log to a file for starters.

About the colors. I don't know if the Windows command prompt or PowerShell has ANSI terminal enabled by default.
If not, then you may see extra ANSI-term control characters which makes stuff harder to read.

[LeeNX]

Minimal build version would be nice. Also can be a ref, ie if you can't build the minimal, you have a big problem ... ;-)

Build log is a good idea. Detailed vs warnings. I am just thinking if something fail, an error log would be quickest. Just looking throu a very big log file for an error in the middle a pain in the @$$.

Colour output would have been a nice to have, but log files I think are better.

Thanks @TD-er

[micropet]

I went back into the past, uninstalled Hyper-V and installed Virtualbox. In Custem.h is a controller and a few plugins enabled.

default: Checking size .pio/build/custom_ESP8266_4M1M/firmware.elf
default: Memory Usage -> http://bit.ly/pio-memory-usage
default: DATA: [===== ] 50.9% (used 41700 bytes from 81920 bytes)
default: PROGRAM: [====== ] 64.3% (used 671156 bytes from 1044464 bytes)
default: ========================= [SUCCESS] Took 19.94 seconds =========================
default:
default: Environment Status Duration
default: ----------------------------------------- -------- ------------
default: custom_ESP8266_4M1M SUCCESS 00:00:19.941
default: memanalyze_ESP8266 IGNORED
default: esp-wrover-kit_test_1M8_partition IGNORED
default: esp32test_1M8_partition IGNORED
default: normal_ESP8266_1M IGNORED
default: normal_ESP8266_1M_VCC IGNORED
default: minimal_core_242_ESP8266_1M_OTA IGNORED
default: minimal_core_242_ESP8285_1M_OTA IGNORED
default: minimal_core_252_ESP8266_1M_OTA IGNORED
default: minimal_core_252_ESP8285_1M_OTA IGNORED
default: normal_core_241_ESP8266_1M IGNORED
default: normal_core_242_ESP8266_1M IGNORED
default: normal_core_252_ESP8266_1M IGNORED
default: normal_ESP8285_1M IGNORED
default: normal_WROOM02_2M IGNORED
default: normal_core_252_WROOM02_2M256 IGNORED
default: normal_ESP8266_4M1M IGNORED
default: normal_core_241_ESP8266_4M1M IGNORED
default: normal_core_252_ESP8266_4M1M IGNORED
default: normal_core_252_ESP8266_16M IGNORED
default: minimal_IRext_ESP8266_1M IGNORED
default: minimal_IRext_ESP8266_4M1M IGNORED
default: minimal_IRext_ESP8266_4M2M IGNORED
default: normal_IRext_no_rx_ESP8266_4M2M IGNORED
default: test_core_242_ESP8266_4M1M IGNORED
default: test_core_252_ESP8266_4M1M IGNORED
default: normal_core_260_sdk222_alpha_ESP8266_4M1M IGNORED
default: test_core_260_sdk222_alpha_ESP8266_4M1M IGNORED
default: test_core_260_sdk3_alpha_ESP8266_4M1M IGNORED
default: test_core_252_ESP8266_16M IGNORED
default: normal_core_260_sdk222_alpha_ESP8266_16M IGNORED
default: test_core_260_sdk222_alpha_ESP8266_16M IGNORED
default: test_core_260_sdk3_alpha_ESP8266_16M IGNORED
default: test_ESP8266_4M_VCC IGNORED
default: dev_ESP8266_4M1M IGNORED
default: hard_SONOFF_POW_4M1M IGNORED
default: hard_core_252_SONOFF_POW_4M1M IGNORED
default: hard_other_POW_ESP8285_1M IGNORED
default: hard_core_252_other_POW_ESP8285_1M IGNORED
default: hard_core_252_Shelly_1_2M256 IGNORED
default: hard_Ventus_W266 IGNORED
default: ========================= 1 succeeded in 00:00:19.941 =========================

I have another question: If I change something, how does the Compieler process restarted?
Is there something like a clean?
What I did not understand yet: Which source files are used - those from Github or from the ESPEasy directory?

Edit:
I have once again started "vagrant provision".
Now a ESP_"Easy_20190911_vagrant_custom_ESP8266_4M1M.bin" is generated which work on my Wemos D1.

Thank you..

[TD-er]

The script does fetch the latest ESPeasy branch from Github and uses that.
I also want to add an option to either select a PR or a tag so you can build based on a specific version.

Currently I think it is the easiest to just call vagrant halt and then vagrant up
Not sure what will happen if you call vagrant up on an already running instance.

[micropet]

Thank you.
I am successful with "vagrant provision".
I can add plugins, which will then be compiled.

[TD-er]

Today I added the option to define a list of PIO environments to build in Vagrant.
All you have to do is add a txt file (next to the Custom.h and Vagrantfile file) called pio_envlist.txt.
In here there must be a list of environments to build, one per line.
No extra lines should be present in that file.

For example:

custom_ESP32_4M316k
custom_ESP8266_4M1M
custom_ESP8266_4M2M
custom_beta_ESP8266_4M1M
custom_sdk3_ESP8266_4M1M