improve http uploader for platformio upload comand

[pgollor]

Description:

• Add additional script to pio-tools to upload gzip firmware files via platformio upload command
• Removed dublicate of espupload.py
• modify http-uploader.py to work with other espupload.py script

To work it is needed to modify the [env] section like:

[env]
upload_port              = -i domus1 -p 80 -u /tasmota/uploader.php
extra_scripts            = ${esp_defaults.extra_scripts}
                           pio-tools/http-gz-uploader.py

[env:tasmota32_base]
upload_port             = -i domus1 -p 80 -u /tasmota/uploader.php
extra_scripts           = ${esp32_defaults.extra_scripts}
                          pio-tools/http-uploader.py

PR for documentation will comming soon.

Checklist:

• The pull request is done against the latest development branch
• Only relevant files were touched
• Only one feature/fix was added per PR and the code change compiles without warnings
• The code change is tested and works with Tasmota core ESP8266 V.2.7.4.9
• The code change is tested and works with Tasmota core ESP32 V.2.0.5
• I accept the CLA.

[Jason2866]

I thought only Theo is using ;-)

[arendst]

Oh my.

I'm just in the process of rewriting this stuff. I'm doing the gz stuff on the server side using a php script like:

<?php
// mkdir and chmod arduino folder to 777
//
//var_dump($_FILES);

/**
 * GZIPs a file on disk (appending .gz to the name)
 *
 * From http://stackoverflow.com/questions/6073397/how-do-you-create-a-gz-file-using-php
 * Based on function by Kioob at:
 * http://www.php.net/manual/en/function.gzwrite.php#34955
 *
 * @param string $source Path to file that should be compressed
 * @param integer $level GZIP compression level (default: 9)
 * @return string New filename (with .gz appended) if success, or false if operation fails
 */
function gzCompressFile($source, $level = 9){
    $dest = $source . '.gz';
    $mode = 'wb' . $level;
    $error = false;
    if ($fp_out = gzopen($dest, $mode)) {
        if ($fp_in = fopen($source,'rb')) {
            while (!feof($fp_in))
                gzwrite($fp_out, fread($fp_in, 1024 * 512));
            fclose($fp_in);
        } else {
            $error = true;
        }
        gzclose($fp_out);
    } else {
        $error = true;
    }
    if ($error)
        return false;
    else
        return $dest;
}

$image = basename($_FILES["file"]["name"]);
//$image = $_FILES["file"]["name"];  // Solves an issue where basename returns Array
$target_file = "tasmota/".$image;
$hostname = $_SERVER['SERVER_NAME'];

if (move_uploaded_file($_FILES["file"]["tmp_name"], $target_file)) {
  if (strpos($target_file, "tasmota32")) {
    echo "The file $image has been uploaded to OTA server $hostname. \n";
  } else {
    gzCompressFile($target_file);
    echo "The files $image and $image.gz have been uploaded to OTA server $hostname. \n";
  }
} else {
  echo "Sorry, there was an error uploading your file $image to OTA server $hostname. \n";
}
?>

I also wonder how on earth the espupload.py ever got in the tools folder. It should have been in the pio-tools folder

[Jason2866]

@pgollor
adding the changes like you suggested in the env can generate weird issues

[env]
upload_port              = -i domus1 -p 80 -u /tasmota/uploader.php
extra_scripts            = ${esp_defaults.extra_scripts}
                           pio-tools/http-gz-uploader.py

[env:tasmota32_base]
upload_port             = -i domus1 -p 80 -u /tasmota/uploader.php
extra_scripts           = ${esp32_defaults.extra_scripts}
                          pio-tools/http-uploader.py

it needs to be added in the main scripts section and the upload_port define ONLY in the [user_def_env]

[arendst]

Agree. It has to be in the platform_override.ini file. This is a snapshot of mine:

; *** Serial port used for erasing/flashing the ESP82xx
;upload_port             = ${common.upload_port}
;upload_port             = COM21
;upload_resetmethod      = ${common.upload_resetmethod}
;extra_scripts           = ${esp_defaults.extra_scripts}
;                          pio-tools/obj-dump.py

; *** Upload file to OTA server in folder api/arduino using HTTP
;upload_port             = domus1:80/api/upload-arduino.php
upload_port             = otaserver/ota/upload-tasmota.php
extra_scripts           = ${esp_defaults.extra_scripts}
;                          pio-tools/obj-dump.py
                          pio-tools/http-uploader.py

[pgollor]

Okay sorry. I used it only in the platformio_override.ini file but there in the [env] section to define the main destination and in some user sections to upload some custom targets into a subfolder.

[Jason2866]

@arendst

I also wonder how on earth the espupload.py ever got in the tools folder. It should have been in the pio-tools folder

Never used and tbh not knowing how it is working, i didnt touch, when doing all the platformio rework. Dont touch stuff from "Master" :-)

[arendst]

That 's the best approach LoL.

Anyway, @pgoller let me do some tweaking first before continuing with this PR.

[pgollor]

A snapshot of mine platformio_override.ini:

[env]
upload_port             = -i <privacy placeholder> -u /tasmota/upload.php
extra_scripts           = ${esp_defaults.extra_scripts}
                          pio-tools/http-gz-uploader.py

[env:tasmota-2M]
board                   = esp8266_2M256
upload_port             = -i <privacy placeholder> -u /tasmota/2M/upload.php

[pgollor]

That 's the best approach LoL.

Anyway, @pgoller let me do some tweaking first before continuing with this PR.

okay thanks. Maybe the PR tasmota/docs#1128 in the documentation should also be adjusted

[arendst]

Have a look at my current, much cut down version of, espupload.py. It needs a complete URL instead of seperate parameters.

#!/usr/bin/env python

"""
  espupload.py - for Tasmota

  Copyright (C) 2021  Theo Arends

  This program is free software: you can redistribute it and/or modify
  it under the terms of the GNU General Public License as published by
  the Free Software Foundation, either version 3 of the License, or
  (at your option) any later version.

  This program is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.

  You should have received a copy of the GNU General Public License
  along with this program.  If not, see <http://www.gnu.org/licenses/>.

Provides:
  Uploads binary file to OTA server.
  Usually initated from http-uploader.py

Requirements:
  - Python
  - pip install requests

Usage:
  ./espupload -u <Host_IP_address>:<Host_port>/<Host_path> -f <sketch.bin>
"""

import sys
import os
import shutil
import argparse
import requests

#HOST_URL = "domus1:80/api/upload-arduino.php"
HOST_URL = "otaserver/ota/upload-tasmota.php"

def main(args):
#  print(sys.argv[0:])

  # get arguments
  parser = argparse.ArgumentParser(
    usage = "%prog [arguments]",
    description = "Upload image to over the air Host server for the esp8266 or esp32 module with OTA support."
  )
  parser.add_argument("-u", "--host_url", dest = "host_url", action = "store", help = "Host url", default = HOST_URL)
  parser.add_argument("-f", "--file", dest = "image", help = "Image file.", metavar = "FILE", default = None)
  args = parser.parse_args()

  if (not args.host_url or not args.image):
    print("Not enough arguments.")
    return 1
  # end if

  if not os.path.exists(args.image):
    print('Sorry: the file %s does not exist', args.image)
    return 2
  # end if

  # copy firmware.bin to tasmota.bin or tasmota32.bin
  tname = os.path.normpath(os.path.dirname(args.image))
  new_filename = tname + os.sep + os.path.basename(tname) + '.bin'
  shutil.copy2(args.image, new_filename)

  url = 'http://%s' % (args.host_url)
  files = {'file': open(new_filename, 'rb')}
  req = requests.post(url, files=files)
  print(req.text)
# end main

if __name__ == '__main__':
  sys.exit(main(sys.argv))
# end if

It also doesn't use curl anymore in favour of requests.

[pgollor]

I also wonder how on earth the espupload.py ever got in the tools folder. It should have been in the pio-tools folder

thre was a 50% 50% chance to do the right one fpr me. Maybe we can move the script from tools to pio-tools. I used the script at my jenkins server and not only from platformio. But I don't really care in which subfolder it is.

[Jason2866]

This is very intrusive since used for all env.

[env]
upload_port             = -i <privacy placeholder> -u /tasmota/upload.php
extra_scripts           = ${esp_defaults.extra_scripts}
                          pio-tools/http-gz-uploader.py

so no local upload (flash) via esptool.py possible anymore for every! env

If you do overrides on more places, this will generate "fun". Been there, and wasted some time with such and similar stuff. Platformio will behave at some point eratic when to many overrides, on different places, are done. Result are strange compile errors which does not give a clue what the real reason is.
I spent many hours to get a nice, easy, flexible, stable predictable Tasmota Platformio setup done.
Platformio has some "nice" bugs...

[pgollor]

Oh okay sorry.
Okay im a little bit confused about all the different versions of espupload.py ;-)

[pgollor]

Whether it's a complete URL or individual parameters, I think that's a personal preference. You decide what you think makes more sense.

[pgollor]

And i think repleacing the curl stuff with the requests library is a good idea.
If you have time, you could add your new version in combination with the new http uploader files?

[arendst]

What I'll do is renaming the current espupload.py into espupload_legacy.py and keep it in pio-tools removing the one from tools.

I'll then add my new espupload.py with the requests library. This way you can still select from one of the two.

I'll have a look at your gz implementation too.

[pgollor]

This is very intrusive since used for all env.

[env]
upload_port             = -i <privacy placeholder> -u /tasmota/upload.php
extra_scripts           = ${esp_defaults.extra_scripts}
                          pio-tools/http-gz-uploader.py

so no local upload (flash) via esptool.py possible anymore for every! env

If you do overrides on more places, this will generate "fun". Been there, and wasted some time with such and similar stuff. Platformio will behave at some point eratic when to many overrides, on different places, are done. Result are strange compile errors which does not give a clue what the real reason is. I spent many hours to get a nice, easy, flexible, stable predictable Tasmota Platformio setup done. Platformio has some "nice" bugs...

Okay sorry for that and thanks for your hint.

[pgollor]

What I'll do is renaming the current espupload.py into espupload_legacy.py and keep it in pio-tools removing the one from tools.

I'll then add my new espupload.py with the requests library. This way you can still select from one of the two.

I'll have a look at your gz implementation too.

Thanks for your great work.

[arendst]

My latest PR builds on your groundwork and updates the following files:

• espupload.py and http-uploader.py in pio-tools
• upload-tasmota.php in your OTA server

The default behaviour now is that if a .bin.gz file is present it is uploaded to the OTA server. If not found it tries to upload the .bin file and generates a .bin.gz on the OTA server using the php script.

Give it a try.

I hope we do not need to add some kind of override to enforce .bin over .bin.gz ;-)