Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

esp_http_client redirect to new host not supported (IDFGH-434) #2631

Closed
anders-above opened this issue Oct 25, 2018 · 2 comments
Closed

esp_http_client redirect to new host not supported (IDFGH-434) #2631

anders-above opened this issue Oct 25, 2018 · 2 comments

Comments

@anders-above
Copy link

Environment

  • Development Kit: [none]
  • Core (if using chip or module): [ESP32-WROOM-32]
  • IDF version (git rev-parse --short HEAD to get the commit id.): 22da5f6
  • Development Env: [Make]
  • Operating System: [Ubuntu]
  • Power Supply: [USB]

Problem Description

esp_http_client_perform() fails if the server returns a HTTP/301 Redirect with Location set to a different hostname
It will retry with the new URL, but without disconnecting from the current server.

This code in esp_http_client_set_url() doesn't work, since http_utils_assign_string overwrites the data that should have been stored in old_host:

    old_host = client->connection_info.host;
    old_port = client->connection_info.port;

    if (purl.field_data[UF_HOST].len) {
        http_utils_assign_string(&client->connection_info.host, url + purl.field_data[UF_HOST].off, purl.field_data[UF_HOST].len);
        HTTP_MEM_CHECK(TAG, client->connection_info.host, return ESP_ERR_NO_MEM);
    }
    // Close the connection if host was changed
    if (old_host && client->connection_info.host
            && strcasecmp(old_host, (const void *)client->connection_info.host) != 0) {
        ESP_LOGD(TAG, "New host assign = %s", client->connection_info.host);
        if (esp_http_client_set_header(client, "Host", client->connection_info.host) != ESP_OK) {
            return ESP_ERR_NO_MEM;
        }
        esp_http_client_close(client);
    }

Expected Behavior

It should reconnect to the new host

Actual Behavior

Continues to connect to the same host, which will trigger HTTP_CLIENT: Error, reach max_redirection_count count=10

Steps to repropduce

  1. esp_http_client_set_url() to a URL that returns HTTP 301 Redirect to another host
  2. esp_http_client_perform()

Debug Logs

I (161349) server_sync_start: hostname: nginx-redirect-2.test.com
I (161349) server_sync_send: url: /test
D (161359) HTTP_CLIENT: Begin connect to: https://nginx-redirect-2.test.com:443
D (161369) TRANS_SSL: Connect to nginx-redirect-2.test.com:443
D (161429) TRANS_SSL: Performing the SSL/TLS handshake...
D (162459) eventbits_get: 0x21
D (162579) TRANS_SSL: Verifying peer X.509 certificate...
D (162579) TRANS_SSL: Certificate verified.
D (162579) TRANS_SSL: Cipher suite is TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384
I (162589) http_event_handler: HTTP_EVENT_ON_CONNECTED
D (162599) HTTP_CLIENT: Write header[4]: GET /test HTTP/1.1
User-Agent: ESP32 HTTP Client/1.0
Host: nginx-redirect-2.test.com
Accept: application/x-msgpack
Content-Length: 0


D (162669) HTTP_CLIENT: on_message_begin
D (162669) HTTP_CLIENT: HEADER=Server:nginx/1.13.9
I (162669) http_event_handler: HTTP_EVENT_ON_HEADER, key=Server, value=nginx/1.13.9
D (162679) HTTP_CLIENT: HEADER=Date:Thu, 25 Oct 2018 11:36:56 GMT
I (162679) http_event_handler: HTTP_EVENT_ON_HEADER, key=Date, value=Thu, 25 Oct 2018 11:36:56 GMT
D (162689) HTTP_CLIENT: HEADER=Content-Type:text/html
I (162699) http_event_handler: HTTP_EVENT_ON_HEADER, key=Content-Type, value=text/html
D (162699) HTTP_CLIENT: HEADER=Content-Length:185
I (162709) http_event_handler: HTTP_EVENT_ON_HEADER, key=Content-Length, value=185
D (162719) HTTP_CLIENT: HEADER=Connection:keep-alive
I (162719) http_event_handler: HTTP_EVENT_ON_HEADER, key=Connection, value=keep-alive
D (162729) HTTP_CLIENT: HEADER=Location:https://resource.test.com/test
I (162739) http_event_handler: HTTP_EVENT_ON_HEADER, key=Location, value=https://resource.test.com/test
D (162749) HTTP_CLIENT: HEADER=Strict-Transport-Security:max-age=15724800; includeSubDomains;
I (162759) http_event_handler: HTTP_EVENT_ON_HEADER, key=Strict-Transport-Security, value=max-age=15724800; includeSubDomains;
D (162769) HTTP_CLIENT: http_on_headers_complete, status=301, offset=297, nread=297
D (162779) HTTP_CLIENT: http_on_body 185
I (162779) http_event_handler: HTTP_EVENT_ON_DATA, len=185
D (162789) HTTP_CLIENT: http_on_message_complete, parser=3ffdffbc
D (162799) HTTP_CLIENT: content_length = 185
I (162799) HTTP_CLIENT: Redirect to https://resource.test.com/test
I (162809) http_event_handler: HTTP_EVENT_ON_FINISH
D (162819) HTTP_CLIENT: Write header[4]: GET /test HTTP/1.1
User-Agent: ESP32 HTTP Client/1.0
Host: nginx-redirect-2.test.com
Accept: application/x-msgpack
Content-Length: 0


D (162889) HTTP_CLIENT: on_message_begin
D (162889) HTTP_CLIENT: HEADER=Server:nginx/1.13.9
I (162889) http_event_handler: HTTP_EVENT_ON_HEADER, key=Server, value=nginx/1.13.9
D (162899) HTTP_CLIENT: HEADER=Date:Thu, 25 Oct 2018 11:36:56 GMT
I (162909) http_event_handler: HTTP_EVENT_ON_HEADER, key=Date, value=Thu, 25 Oct 2018 11:36:56 GMT
D (162909) HTTP_CLIENT: HEADER=Content-Type:text/html
I (162919) http_event_handler: HTTP_EVENT_ON_HEADER, key=Content-Type, value=text/html
D (162929) HTTP_CLIENT: HEADER=Content-Length:185
I (162929) http_event_handler: HTTP_EVENT_ON_HEADER, key=Content-Length, value=185
D (162939) HTTP_CLIENT: HEADER=Connection:keep-alive
I (162949) http_event_handler: HTTP_EVENT_ON_HEADER, key=Connection, value=keep-alive
D (162959) HTTP_CLIENT: HEADER=Location:https://resource.test.com/test
I (162959) http_event_handler: HTTP_EVENT_ON_HEADER, key=Location, value=https://resource.test.com/test
D (162979) HTTP_CLIENT: HEADER=Strict-Transport-Security:max-age=15724800; includeSubDomains;
I (162979) http_event_handler: HTTP_EVENT_ON_HEADER, key=Strict-Transport-Security, value=max-age=15724800; includeSubDomains;
D (162999) HTTP_CLIENT: http_on_headers_complete, status=301, offset=297, nread=297
D (162999) HTTP_CLIENT: http_on_body 185
I (163009) http_event_handler: HTTP_EVENT_ON_DATA, len=185
D (163009) HTTP_CLIENT: http_on_message_complete, parser=3ffdffbc
D (163019) HTTP_CLIENT: content_length = 185
I (163029) HTTP_CLIENT: Redirect to https://resource.test.com/test
I (163029) http_event_handler: HTTP_EVENT_ON_FINISH
D (163039) HTTP_CLIENT: Write header[4]: GET /test HTTP/1.1
User-Agent: ESP32 HTTP Client/1.0
Host: nginx-redirect-2.test.com
Accept: application/x-msgpack
Content-Length: 0

.....
.....
@Alvin1Zhang Alvin1Zhang changed the title esp_http_client redirect to new host not supported [TW#27006] esp_http_client redirect to new host not supported Oct 26, 2018
@jitin17
Copy link
Contributor

jitin17 commented Oct 27, 2018

@anders-above Thanks for bringing this to our notice. We'll address it soon.

@jitin17
Copy link
Contributor

jitin17 commented Nov 6, 2018

I tried reproducing the issue by running a local server which redirects to another host, and it seems to work. So, I am sharing the log and python script with you. Let me know if I am missing something.

D (6840) event: SYSTEM_EVENT_STA_GOT_IP, ip:10.0.0.212, mask:255.255.255.0, gw:10.0.0.1
I (6850) event: sta ip: 10.0.0.212, mask: 255.255.255.0, gw: 10.0.0.1
I (6850) HTTP_CLIENT: Connected to AP, begin http example
D (6860) HTTP_CLIENT: Begin connect to: http://10.0.0.98:80
D (6870) TRANS_TCP: [sock=54],connecting to server IP:10.0.0.98,Port:80...
D (6890) HTTP_CLIENT: HTTP_EVENT_ON_CONNECTED
D (6890) HTTP_CLIENT: Write header[3]: GET / HTTP/1.1
User-Agent: ESP32 HTTP Client/1.0
Host: 10.0.0.98
Content-Length: 0


D (6910) HTTP_CLIENT: on_message_begin
D (6910) HTTP_CLIENT: HEADER=Server:SimpleHTTP/0.6 Python/2.7.10
D (6910) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Server, value=SimpleHTTP/0.6 Python/2.7.10
D (6920) HTTP_CLIENT: HEADER=Date:Tue, 06 Nov 2018 11:32:51 GMT
D (6920) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Date, value=Tue, 06 Nov 2018 11:32:51 GMT
D (6930) HTTP_CLIENT: HEADER=Location:https://www.google.com
D (6940) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Location, value=https://www.google.com
D (6940) HTTP_CLIENT: http_on_headers_complete, status=301, offset=143, nread=143
D (6950) HTTP_CLIENT: content_length = -1
I (6960) HTTP_CLIENT: Redirect to https://www.google.com
D (6960) HTTP_CLIENT: New host assign = www.google.com
D (6970) HTTP_CLIENT: HTTP_EVENT_DISCONNECTED
D (6970) HTTP_CLIENT: HTTP_EVENT_DISCONNECTED
D (6980) HTTP_CLIENT: Read finish or server requests close
D (6980) HTTP_CLIENT: HTTP_EVENT_ON_FINISH
D (6990) HTTP_CLIENT: Close connection
D (6990) HTTP_CLIENT: HTTP_EVENT_DISCONNECTED
D (6990) HTTP_CLIENT: Begin connect to: https://www.google.com:443
D (7000) esp-tls: host:www.google.com: strlen 14
D (7210) esp-tls: handshake in progress...
D (9860) HTTP_CLIENT: HTTP_EVENT_ON_CONNECTED
D (9860) HTTP_CLIENT: Write header[3]: GET / HTTP/1.1
User-Agent: ESP32 HTTP Client/1.0
Host: www.google.com
Content-Length: 0


D (10060) HTTP_CLIENT: on_message_begin
D (10070) HTTP_CLIENT: HEADER=Date:Tue, 06 Nov 2018 11:32:54 GMT
D (10070) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Date, value=Tue, 06 Nov 2018 11:32:54 GMT
D (10070) HTTP_CLIENT: HEADER=Expires:-1
D (10080) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Expires, value=-1
D (10080) HTTP_CLIENT: HEADER=Cache-Control:private, max-age=0
D (10090) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Cache-Control, value=private, max-age=0
D (10100) HTTP_CLIENT: HEADER=Content-Type:text/html; charset=ISO-8859-1
D (10100) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Content-Type, value=text/html; charset=ISO-8859-1
D (10110) HTTP_CLIENT: HEADER=P3P:CP="This is not a P3P policy! See g.co/p3phelp for more info."
D (10120) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=P3P, value=CP="This is not a P3P policy! See g.co/p3phelp for more info."
D (10130) HTTP_CLIENT: HEADER=Server:gws
D (10140) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Server, value=gws
D (10140) HTTP_CLIENT: HEADER=X-XSS-Protection:1; mode=block
D (10150) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=X-XSS-Protection, value=1; mode=block
D (10160) HTTP_CLIENT: HEADER=X-Frame-Options:SAMEORIGIN
D (10160) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=X-Frame-Options, value=SAMEORIGIN
D (10170) HTTP_CLIENT: HEADER=Set-Cookie:1P_JAR=2018-11-06-11; expires=Thu, 06-Dec-2018 11:32:54 GMT; path=/; domain=.google.com
D (10180) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Set-Cookie, value=1P_JAR=2018-11-06-11; expires=Thu, 06-Dec-2018 11:32:54 GMT; path=/; domain=.google.com
D (10190) HTTP_CLIENT: HEADER=Set-Cookie:NID=144=nWS8r-Z7lIz8Vq84E0NHImoDLohDaUUrUoXnCopHUbJvW3IgOryl_rA2_DM1qmpFFAIFey0DZ94IsFQoHiXWo6xQMkwkGAzHahm4
D (10210) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Set-Cookie, value=NID=144=nWS8r-Z7lIz8Vq84E0NHImoDLohDaUUrUoXnCopHUbJvW3IgOryl_rA2_DM1qmpFFAIFey0DZ94IsFQoHiXWo6xQMkwkGAzHahm4
D (10220) HTTP_CLIENT: HEADER=Alt-Svc:quic=":443"; ma=2592000; v="44,43,39,35"
D (10230) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Alt-Svc, value=quic=":443"; ma=2592000; v="44,43,39,35"
D (10240) HTTP_CLIENT: HEADER=Accept-Ranges:none
D (10250) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Accept-Ranges, value=none
D (10250) HTTP_CLIENT: HEADER=Vary:Accept-Encoding
D (10260) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Vary, value=Accept-Encoding
D (10260) HTTP_CLIENT: HEADER=Transfer-Encoding:chunked
D (10270) HTTP_CLIENT: HTTP_EVENT_ON_HEADER, key=Transfer-Encoding, value=chunked
D (10280) HTTP_CLIENT: http_on_headers_complete, status=200, offset=788, nread=788
D (10280) HTTP_CLIENT: http_on_body 231
D (10290) HTTP_CLIENT: HTTP_EVENT_ON_DATA, len=231
D (10290) HTTP_CLIENT: content_length = -1
D (10300) HTTP_CLIENT: data_process=231, content_length=-1
D (10300) HTTP_CLIENT: http_on_body 377
D (10310) HTTP_CLIENT: HTTP_EVENT_ON_DATA, len=377
D (10310) HTTP_CLIENT: data_process=608, content_length=-1
D (10320) HTTP_CLIENT: http_on_body 512
D (10320) HTTP_CLIENT: HTTP_EVENT_ON_DATA, len=512
D (10330) HTTP_CLIENT: data_process=1120, content_length=-1
D (10330) HTTP_CLIENT: http_on_body 512

Redirect python server script( copied from https://gist.github.com/shreddd/b7991ab491384e3c3331)

import SimpleHTTPServer
import SocketServer
import sys
import argparse



def redirect_handler_factory(url):
    """
    Returns a request handler class that redirects to supplied `url`
    """
    class RedirectHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
       def do_GET(self):
           self.send_response(301)
           self.send_header('Location', url)
           self.end_headers()

    return RedirectHandler


def main():

    parser = argparse.ArgumentParser(description='HTTP redirect server')

    parser.add_argument('--port', '-p', action="store", type=int, default=80, help='port to listen on')
    parser.add_argument('--ip', '-i', action="store", default="", help='host interface to listen on')

    myargs = parser.parse_args()

    port = myargs.port
    host = myargs.ip

    redirectHandler = redirect_handler_factory('https://www.google.com')

    handler = SocketServer.TCPServer((host, port), redirectHandler)
    print("serving at port %s" % port)
    handler.serve_forever()

if __name__ == "__main__":
    main()

@projectgus projectgus changed the title [TW#27006] esp_http_client redirect to new host not supported esp_http_client redirect to new host not supported (IDFGH-434) Mar 12, 2019
igrr pushed a commit to espressif/esp-adf that referenced this issue Mar 29, 2019
@vikramdattu
Fix URL redirection issue
bf02ad4 
Operation:
In esp_http_client_set_url, we check for if old_host is same as new_host.
Delete and open new connection if host is different.
Issue:
We just pointed client->connection_info.host to old_host and reassigned it.
This made old_host and new_host always point to same location and hence, using old_host with new request.
Fix:
Made a separate copy for old_host using strdup.

Related issue on github: espressif/esp-idf#2631
Related merge request on idf: https://gitlab.espressif.cn:6688/idf/esp-idf/merge_requests/4623/

Signed-off-by: Vikram Dattu <vikram.dattu@espressif.com>
@igrr igrr closed this as completed in a875505 Apr 11, 2019
igrr pushed a commit that referenced this issue Apr 28, 2019
@vikramdattu
Fix url redirection issue.
344cb99 
Operation:
In `esp_http_client_set_url`, we check for if old_host is same as new_host.
Delete and open new connection if host is different.

Issue:
We just pointed `client->connection_info.host` to `old_host` and reassigned it.
This made old_host and new_host always point to same location and hence, using old_host with new request.

Fix:
Made a separate copy for old_host using strdup.

Closes #2631

Signed-off-by: Vikram Dattu <vikram.dattu@espressif.com>
igrr pushed a commit that referenced this issue May 10, 2019
@vikramdattu
Fix url redirection issue.
d154723 
Operation:
In `esp_http_client_set_url`, we check for if old_host is same as new_host.
Delete and open new connection if host is different.

Issue:
We just pointed `client->connection_info.host` to `old_host` and reassigned it.
This made old_host and new_host always point to same location and hence, using old_host with new request.

Fix:
Made a separate copy for old_host using strdup.

Closes #2631

Signed-off-by: Vikram Dattu <vikram.dattu@espressif.com>
catalinio pushed a commit to catalinio/pycom-esp-idf that referenced this issue Jun 28, 2019
@vikramdattu
Fix url redirection issue.
7fff20c 
Operation:
In `esp_http_client_set_url`, we check for if old_host is same as new_host.
Delete and open new connection if host is different.

Issue:
We just pointed `client->connection_info.host` to `old_host` and reassigned it.
This made old_host and new_host always point to same location and hence, using old_host with new request.

Fix:
Made a separate copy for old_host using strdup.

Closes espressif/esp-idf#2631

Signed-off-by: Vikram Dattu <vikram.dattu@espressif.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jitin17 @anders-above