HTTP client file download.

[fredericplante]

Is there an example that use the HTTP client to connect to a HTTP server, request a file, download it and then save it to SPIFFS?

You know, what a HTTP client should be doing. :D

[vicnevicne]

:-)
Please note that for usability questions, you'll have much more attention on the esp8266.com forums. Issues are meant to report bugs or problems in the project itself.
That being said, a few month ago I worked on an http proxy on the ESP that uses SPIFFS for temporary storage, so it does what you request, plus more.
The code is at http://www.esp8266.com/viewtopic.php?f=32&t=10103&start=16#p48617

[devyte]

@fredericplante there is an httpClient class implementation. But this is not the right place to ask for usage help.
Closing due to off-topic, see #3655 .

[Erol444]

I got it working like this :) you put file url and program saves the file to SPIFFS.

 url = String(UPDATE_URL) + response.substring(4);
    //url = http://123.123.123.123/SPIFFS/test2.htm
    String file_name=response.substring(response.lastIndexOf('/'));
    //file_name = test2.htm
    Serial.println(url);
    File f = SPIFFS.open(file_name, "w");
    if (f) {
      http.begin(url);
      int httpCode = http.GET();
      if (httpCode > 0) {
        if (httpCode == HTTP_CODE_OK) {
          http.writeToStream(&f);
        }
      } else {
        Serial.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
      }
      f.close();
    }
    http.end();

[KushlaVR]

http - what is it? what type? Where did you copy this pice of code?

[Erol444]

http - what is it? what type? Where did you copy this pice of code?

If you want to download A file(any type I assume) and save it to SPIFFS you can use the code above. This code should be copied to your ESP8266 program.

[KushlaVR]

http - what is it? what type? Where did you copy this pice of code?

If you want to download A file(any type I assume) and save it to SPIFFS you can use the code above. This code should be copied to your ESP8266 program.

To make this code work you'll nead:

1. #include <ESP8266HTTPClient.h>
2. declare http variable
   HTTPClient http;

This code work only to http://... url. If I want to download file from https://... url - it not work.
For example if I downloaf raw files from github - i got an Error

For https url I write next method (I actually get it from example and make some adaptation for my own newads)
This void download data from url + file link and store it into SPIFFS to file with path file

It work fine for small files (2..3 Kbytes). But when file is larger (30Kbytes) - the download stops before I reach Content-lenth
I dont know why

//Small file download: https://raw.githubusercontent.com/KushlaVR/WiFi-relay/master/MQTT-Relay/data/html/index.html
updateFile("https://raw.githubusercontent.com/KushlaVR/WiFi-relay/master/MQTT-Relay/data", "/html/index.html");

//Big file download: https://raw.githubusercontent.com/KushlaVR/WiFi-relay/master/MQTT-Relay/data/html/js/jquery.min.js
updateFile("https://raw.githubusercontent.com/KushlaVR/WiFi-relay/master/MQTT-Relay/data", "/html/js/jquery.min.js");

Code:

uint16_t httpsPort = 443;
const char * host = "raw.githubusercontent.com";//TOD: read from settings file
const char * fingerprint = "CC AA 48 48 66 46 0E 91 53 2C 9C 7C 23 2A B1 74 4D 29 9D 33";//TOD: read from settings file (can see in browser certificate info, MUST BE UPPERCASE!)

void updateFile(String url, String file) {
	WiFiClientSecure client;
	Serial.print("connecting to ");
	Serial.println(host);
	if (!client.connect(host, httpsPort)) {
		Serial.println("connection failed");
		return;
	}
	if (client.verify(fingerprint, host)) {
		Serial.println("certificate matches");
	}
	else {
		Serial.println("certificate doesn't match");
	}

	String action = url.substring(url.indexOf(host) + strlen(host)) + file;

	client.print(String("GET ") + action + " HTTP/1.1\r\n" +
		"Host: " + host + "\r\n" +
		"User-Agent: ESP8266\r\n" +
		"Connection: close\r\n\r\n");

	long len = -1;
	while (client.connected()) {
		String line = client.readStringUntil('\n');
		if (line.startsWith("Content-Length:")) {
			len = line.substring(15).toInt();
		}
		if (line == "\r") {
			Serial.println("headers received");
			break;
		}
	}

	Serial.printf("[HTTPS] GET: %s\n",action.c_str());
	File f = SPIFFS.open(file, "w");
	if (f) {
		uint8_t buff[128];
		while (client.connected() && (len > 0 || len == -1)) {
			size_t size = client.available();
			if (size>0) {
				int c = client.readBytes(buff, ((size > 128) ? 128 : size));
				f.write(buff, c);
				if (len > 0) 	len -= c;
			}
			else { break; }
			Serial.printf("bytes left %i", len);
			ESP.wdtDisable();
			ESP.wdtEnable(1000);
		}
		Serial.println("done.");
		f.close();
	}
}

[rtek1000]

Hello,

Since it is possible to download,

Would it be possible to measure the internet speed (up to the bandwidth that the ESP8266 supports)?

Would it be possible to download the data without saving to be able to download a larger file, maybe a 5MB or 10MB to perform an average speed?

Any suggestion?

[KushlaVR]

If you commentout f variable - then you can read data without saving.
Or if uou use Erol444 example (do download http:// files) - you'll nead to write empty class based on Stram
Then you can cound number of bytes recived and timespan.

[Erol444]

@rtek1000 Well actually HTTPClient::writeToStream function(the one I used in example code above) returns bytes written (negative values are error codes ).

As for not saving the file, it's been quite some time since I last programmed esp32 (I don't have one to test anymore) but you should also check in HttpClient class, there is a function getSize, maybe you can get it working:

//size of message body / payload
//@return -1 if no info or > 0 when Content-Length is set by server

int HTTPClient::getSize(void)
{
return _size;
}

[fredericplante]

It is nice to see that this turn out to be usefull after all.

Thank you all for the suggestion, I see it could also be easily modified to save to a SDcard, that is nice.

:)