Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Advanced OTA from Server with PHP script [Update Failed] #3043

Closed
nanofortran opened this issue Mar 13, 2017 · 15 comments
Closed

Advanced OTA from Server with PHP script [Update Failed] #3043

nanofortran opened this issue Mar 13, 2017 · 15 comments

Comments

@nanofortran
Copy link

Basic Infos

I am unclear about the proper use of ESPhttpUpdate in the 'advanced' scenario (ie, where the ESP calls the server-side script (PHP) and then replies with the .bin file if all the appropriate conditions are met.) If I point the ESP directly to the .bin file, there is no issue, and the file is delivered as expected, the board reboots, and new .bin executes. (Nicely done!) I believe the issue is related to my use of the .update() method and PHP script provided with the documentation.

I can confirm the server is running fine and the PHP is executable. When I hit it with a browser it says "only for ESP8266 updater!" as it should, so it looks good there. I am not too familiar with PHP, so a few questions:

  1. Am I calling the method correctly, I am unsure about the fourth argument:
If version string argument is given, it will be sent to the server. Server side script can use this to check if update should be performed.

What is the 'version' argument? The version being uploaded, the version that is already on the server, is there a versioning format that should be used?

  1. Does the PHP require the .bin file to be in a sub-directory called /bin/, such that the directory structure is such:
/var/www/html/esp/update/
    -->arduino.php
     /var/www/html/esp/update/bin/
       -->IOT4.ino.bin
  1. Did I place the MAC address of the ESP and the name of the file to be downloaded (without the .bin suffix) in the db$ array correctly?

I surmise that the ESP is connecting correctly (it does download the .bin when pointed at it directly -- the so-called 'Basic' Server method), the issue is probably with my PHP file and .update call.

Thank you, great project.

Hardware

Hardware: Adafruit Huzzah
Core Version: 2.3.0-rc2

Description

Problem description

Settings in IDE

Module: Adafruit Huzzah
Flash Size: 4MB
CPU Frequency: 80Mhz
Upload Using: OTA / SERIAL

Sketch

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>
#include <ESP8266HTTPClient.h>
#include <ESP8266httpUpdate.h>
#define USE_SERIAL Serial

ESP8266WiFiMulti WiFiMulti;

void setup() {

  USE_SERIAL.begin(115200);
  USE_SERIAL.setDebugOutput(true);

  USE_SERIAL.println();
  USE_SERIAL.println();
  USE_SERIAL.println();

  for (uint8_t t = 4; t > 0; t--) {
    USE_SERIAL.printf("[SETUP] WAIT %d...\n", t);
    USE_SERIAL.flush();
    delay(1000);
  }

  WiFiMulti.addAP("Cen---------02", "12-----------G"); //Redacted for privacy
}

void loop() {
  // wait for WiFi connection
  if ((WiFiMulti.run() == WL_CONNECTED)) {

    t_httpUpdate_return ret = ESPhttpUpdate.update("http://www.mediatedspaces.net", 80, "/esp/update/arduino.php");
    switch (ret) {
      case HTTP_UPDATE_FAILED:
        Serial.println("[update] Update failed.");
        break;
      case HTTP_UPDATE_NO_UPDATES:
        Serial.println("[update] Update no Update.");
        break;
      case HTTP_UPDATE_OK:
        Serial.println("[update] Update ok."); // may not called we reboot the ESP
        break;
    }
  }
}

And here is the PHP

<?PHP

header('Content-type: text/plain; charset=utf8', true);

function check_header($name, $value = false) {
    if(!isset($_SERVER[$name])) {
        return false;
    }
    if($value && $_SERVER[$name] != $value) {
        return false;
    }
    return true;
}

function sendFile($path) {
    header($_SERVER["SERVER_PROTOCOL"].' 200 OK', true, 200);
    header('Content-Type: application/octet-stream', true);
    header('Content-Disposition: attachment; filename='.basename($path));
    header('Content-Length: '.filesize($path), true);
    header('x-MD5: '.md5_file($path), true);
    readfile($path);
}

if(!check_header('HTTP_USER_AGENT', 'ESP8266-http-Update')) {
    header($_SERVER["SERVER_PROTOCOL"].' 403 Forbidden', true, 403);
    echo "only for ESP8266 updater!\n";
    exit();
}

if(
    !check_header('HTTP_X_ESP8266_STA_MAC') ||
    !check_header('HTTP_X_ESP8266_AP_MAC') ||
    !check_header('HTTP_X_ESP8266_FREE_SPACE') ||
    !check_header('HTTP_X_ESP8266_SKETCH_SIZE') ||
    !check_header('HTTP_X_ESP8266_SKETCH_MD5') ||
    !check_header('HTTP_X_ESP8266_CHIP_SIZE') ||
    !check_header('HTTP_X_ESP8266_SDK_VERSION')
) {
    header($_SERVER["SERVER_PROTOCOL"].' 403 Forbidden', true, 403);
    echo "only for ESP8266 updater! (header)\n";
    exit();
}

$db = array(
    "18:FE:AA:AA:AA:AA" => "DOOR-7-g14f53a19",
    "18:FE:AA:AA:AA:BB" => "TEMP-1.0.0",
    "5C:CF:7F:88:B6:B9" => "IOT4.ino"  # <-- I have added the ESP MAC address and file name
);

if(!isset($db[$_SERVER['HTTP_X_ESP8266_STA_MAC']])) {
    header($_SERVER["SERVER_PROTOCOL"].' 500 ESP MAC not configured for updates', true, 500);
}

$localBinary = "./bin/".$db[$_SERVER['HTTP_X_ESP8266_STA_MAC']].".bin";

// Check if version has been set and does not match, if not, check if
// MD5 hash between local binary and ESP8266 binary do not match if not.
// then no update has been found.
if((!check_header('HTTP_X_ESP8266_SDK_VERSION') && $db[$_SERVER['HTTP_X_ESP8266_STA_MAC']] != $_SERVER['HTTP_X_ESP826$
    || $_SERVER["HTTP_X_ESP8266_SKETCH_MD5"] != md5_file($localBinary)) {
    sendFile($localBinary);
} else {
    header($_SERVER["SERVER_PROTOCOL"].' 304 Not Modified', true, 304);
}

header($_SERVER["SERVER_PROTOCOL"].' 500 no version for ESP MAC', true, 500);

?>

Debug Messages

Here is the serial output . . .

[SETUP] WAIT 4...
scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 6
cnt 
[SETUP] WAIT 3...

connected with CenturyLink0502, channel 11
dhcp client start...
ip:192.168.0.28,mask:255.255.255.0,gw:192.168.0.1
[SETUP] WAIT 2...
[SETUP] WAIT 1...
pm open,type:2 0
[update] Update failed.
[update] Update failed.
@davisonja
Copy link

davisonja commented Mar 14, 2017 via email

@Humancell
Copy link

Just a few comments ... I use this extensively, but have not yet added any of the MD5 checking ...

In my ESP code I'm using:

t_httpUpdate_return ret = ESPhttpUpdate.update("update.myserver.net", 80, "/v1/ESPUpdate/", firmwareVersion);

In this case my script is "index.php" in the path specified, and then I'm passing a "firmwareVersion" string that I have assigned in my sketch code. The string is like: "mySensor-1.55"

This is the string that is specified in your array using the MAC as a key.

Then in my PHP code, I'm doing:

if(isset($db[$_SERVER['HTTP_X_ESP8266_STA_MAC']])) {
    if($db[$_SERVER['HTTP_X_ESP8266_STA_MAC']] != $_SERVER['HTTP_X_ESP8266_VERSION']) {
	error_log("~~ Update required ...\n", 3, "/var/log/ESP8266-ESPUpdate.log");
	error_log("Sending file: /var/www/update.myserver.net/html/v1/ESPUpdate/bin/".$db[$_SERVER['HTTP_X_ESP8266_STA_MAC']].".bin\n", 3, "/var/log/wovyn/ESP8266-ESPUpdate.log");
	sendFile("/var/www/update.myserver.net/html/v1/ESPUpdate/bin/".$db[$_SERVER['HTTP_X_ESP8266_STA_MAC']].".bin");
	error_log("~~ Update file sent.\n\n", 3, "/var/log/ESP8266-ESPUpdate.log");
    } else {
	error_log("~~ No Update Required.\n\n", 3, "/var/log/ESP8266-ESPUpdate.log");
	header($_SERVER["SERVER_PROTOCOL"].' 304 Not Modified', true, 304);
    }
    exit();
}

error_log("~~ Not Found - Unknown Device.\n\n", 3, "/var/log/ESP8266-ESPUpdate.log");
header($_SERVER["SERVER_PROTOCOL"].' 500 no version for ESP MAC', true, 500);

This works ... but again is not verifying the MD5 on the PHP side ... I wasn't aware the extra header was now being sent. How does the ESP get the proper MD5 of a new fimrware version that it hasn't seen yet?

@davisonja
Copy link

davisonja commented Mar 14, 2017 via email

@davisonja
Copy link

I'm using the release and git version in different places but want to use the MD5 in each, so I have:

void checkUpdate(void) {
  String versionStr = "0.1.2 MD5:" + ESP.getSketchMD5();

  Serial.println("[checkUpdate] Enacting update request");

  t_httpUpdate_return ret = ESPhttpUpdate.update("http://sl.davison.org.nz/fw/aota/update", versionStr);

My PHP webapp will use the md5-header-field if it's there, and if it's not see if there's an 'MD5:' in the version string, using whatever follows it if there is.
It probably doesn't need a huge amount of cleanup if anyone else was interested.

@nanofortran
Copy link
Author

Thanks for all the replies, unfortunately I am still having problems. After trying differing variations of the code offered, I went back to the beginning and started all over (hopefully with better knowledge). Via some error-log()-ing, I had the PHP script log a message whenever the script ran. If I hit the end point (index.php) with a browser, I get a 'script ran!' message in my logs. If I hit it with the ESP, no message. I assume the script is not running. So Firefox fires the script (with the 'ESP only' message) where the ESP returns:

HTTP_UPDATE_FAILD Error (-1): HTTP error: connection refused

via serial.

On the server side the permissions are wide open, and I can always download the update file by hitting the .bin directly via both browser and ESP -- that works perfectly.

I am going to check out: https://www.youtube.com/watch?v=UiAc3yYBsNU this afternoon to see if there is anything in there.

If anyone knows of a tutorial that goes through the entire process of OTA with PHP script, that would be great. If I ever figure this out, I will do it!

Thanks!

@nanofortran
Copy link
Author

nanofortran commented Mar 20, 2017 •
edited

Yeah, no luck. The error.log file for apache2 says:

[Sun Mar 19 19:48:57.486000 2017] [core:error] [pid 24281] [client 71.217.72.45:10325] AH00126: Invalid URI in request GET esp.php HTTP/1.0

and the access.log file says:

71.217.72.45 - - [19/Mar/2017:19:48:57 +0000] "GET esp.php HTTP/1.0" 400 486 "-" "ESP8266-http-Update"

Which all points to the client. In my code, the update request is formed thus, and fails:

t_httpUpdate_return ret = ESPhttpUpdate.update("93.95.228.77", 80, "/esp.php", "httpUpdate");

Where this works:

t_httpUpdate_return ret = ESPhttpUpdate.update("http://93.95.228.77/bin/Blink.bin");

and this fails as well when trying to hook up to iotappstore.org

t_httpUpdate_return ret = ESPhttpUpdate.update("iotappstore.org", 80, "/iotappstorev10.php", "httpupdate");

So, what is the correct way to form my URI? FYI, my files are in /var/www/html/ on the server side, so maybe that is the problem?

Any ideas?

Thanks to the community!

@davisonja
Copy link

Can you just say

t_httpUpdate_return ret = ESPhttpUpdate.update("http://93.95.228.77/esp.php", version);

?

@nanofortran
Copy link
Author

Yeah, that fails, too. It would be interesting to see what ESPhttpUpdate is actually sending . . . a String, I guess? Any suggested debugging strategies/schema would be appreciated.

J

@nanofortran
Copy link
Author

[Solved]
Here's how I debugged it. Briefly, there are a few things in the doc files that are not well documented.

I first tried to debug the server/php script by finding the values for the headers that are passed during the update() call:

    t_httpUpdate_return ret = ESPhttpUpdate.update("93.95.228.77", 80, "/esp.php", "httpUpdate");

by Serial.println()-ing out the values:

  USE_SERIAL.println(WiFi.macAddress());
  USE_SERIAL.println(WiFi.softAPmacAddress());
  USE_SERIAL.println(ESP.getFreeSketchSpace());
  USE_SERIAL.println(ESP.getSketchSize());
  USE_SERIAL.println(ESP.getSketchMD5());
  USE_SERIAL.println(ESP.getFlashChipRealSize());
  USE_SERIAL.println(ESP.getSdkVersion());

Which threw an error basically saying ESP has no method called .getSketchMD5() . . . which a bit more search took me to this issue:

#2228

With the main thrust being that, historically it would seem, MD5 was left out of both Esp.cpp/Esp.h because MD5 was not implemented early on in the HTTPupdate roll out. And yet, the script in the docs is looking for the MD5 key/string and it just ain't there. So, the update fails.

I was using the Board Manager to embed Arduino ESP in the IDE using this uri:

http://arduino.esp8266.com/staging/package_esp8266com_index.json

as suggested in the README. Unfortunately, this does not have the patches necessary to call ESPhttpUpdate correctly. (It all might have been different if I compiled from source, who knows?) So, you need to go in an manually do the deed: to paraphrase #2228:

  1. Be sure you are using 2.3.0-rc2.

  2. Replace Esp.h with the one found here:

https://github.com/esp8266/Arduino/blob/master/cores/esp8266/Esp.h
  1. Replace Esp.cpp with the one found here:
https://github.com/esp8266/Arduino/blob/master/cores/esp8266/Esp.cpp

So, I wish that the documentation would have been a bit more clear on that, or perhaps I missed it? But, to be clear, 2.3.0-rc2 through Board Manager does not have the MD5 functionality implemented.

Thanks!

@davisonja
Copy link

davisonja commented Mar 20, 2017 via email

@nanofortran
Copy link
Author

Well, that is probably the confusion then, I though the staging version was necessary in order to invoke OTA w/ Server functionality (I read that somewhere). So, for my education, what is the significance of rc2 in the naming convention? To my mind, a 'staging' version would be cutting edge, you know, 'staging' for the next release, FWIW.

Thanks again to the community, I am about to unleash 25 students on the library and wanted to work out all the kinks.

jg

@vicnevicne
Copy link
Contributor

vicnevicne commented Mar 20, 2017 •
edited

RC stands for Release Candidate, so in the logical order, you have alpha (buggy version with new features), beta (version with no known outstanding issues), then one (or more) Release Candidate, which is the version that will ship unless a last minute issue is found. In this case, there have been 2 RC before the actual release, which probably means that 2.3rc2 is very close or identical to the final 2.3.
Now that's the theory. Practically speaking, there are a few differences between rc2 and final - see release notes - but you'll see that there was only 8 days between rc2 and final, so the differences are tiny.
Oh, and "cutting edge" is the git version, but you're really on the "nightly" version then so the behaviour can change between one day and the next. So I wouldn't advise it for use with students because there's no guarantee your lesson will still work by the time you get to school.
We're all eagerly waiting for a new "release" (even alpha or beta, just something we can refer people too) because there have been so many improvements and bug fixes since 2.3.0...

@devyte
Copy link
Collaborator

devyte commented Mar 20, 2017 via email

@nanofortran
Copy link
Author

Yeah, I just grabbed the release candidate instead of the 2.3.0 proper. Well, learned a good deal about servers and updates and what is under the hood along the way.

Thanks community . . . I believe I will close this.

Solution: grab 2.3.0 vanilla.

@davisonja
Copy link

davisonja commented Mar 20, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@davisonja @vicnevicne @Humancell @nanofortran @devyte