Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use http1.1 from curl not to get compressed output #3

Merged
merged 1 commit into from Dec 27, 2016

Conversation

satoru-takeuchi
Copy link
Contributor

Compressed data is output with recent curl. This problem happens because of
the following logic.

  1. Recent curl uses http2 by default when sending requests to twitter API
  2. twitter API always returns gzip-compressed data even if any encoding is
    prohibited by Accept-Encoding header(*1).
  3. curl doesn't handle Content-Encoding when --compressed option is not set

*1) When using http1.1, twitter API doesn't compress the output by default

For more information about this twitter API's problem, please see
the following URL.

https://twittercommunity.com/t/twitter-api-doesnt-look-header-accept-encoding-identity/56571

  • Step to reproduce

===============================================================================
$ ./tweet.sh showme | file -

Probably this problem happens with any other subcommands.

===============================================================================
$ ./tweet.sh showme | file -
/dev/stdin: gzip compressed data, from FAT filesystem (MS-DOS, OS/2, NT)
$

  • Expected result

===============================================================================
$ ./tweet.sh showme | file -
/dev/stdin: ASCII text, with very long lines, with no line terminators
$

  • Software environment

curl 7.50.1 (x86_64-pc-linux-gnu) libcurl/7.50.1 GnuTLS/3.5.7 zlib/1.2.8 libidn/1.33 libssh2/1.7.0 nghttp2/1.17.0 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets

I confirmed this problem didn't happen with curl 7.35.0. This version uses
http1.1 by default.

Compressed data is output with recent curl. This problem happens because of
the following logic.

 1. Recent curl uses http2 by default when sending requests to twitter API
 2. twitter API always returns gzip-compressed data even if any encoding is
    prohibited by Accept-Encoding header(*1).
 3. curl doesn't handle Content-Encoding when --compressed option is not set

*1) When using http1.1, twitter API doesn't compress the output by default

For more information about this twitter API's problem, please see
the following URL.

https://twittercommunity.com/t/twitter-api-doesnt-look-header-accept-encoding-identity/56571

* Step to reproduce

===============================================================================
$ ./tweet.sh showme | file -
===============================================================================

Probably this problem happens with any other subcommands.

* Actual result (with 62ad663)

===============================================================================
$ ./tweet.sh showme | file -
/dev/stdin: gzip compressed data, from FAT filesystem (MS-DOS, OS/2, NT)
$
===============================================================================

* Expected result

===============================================================================
$ ./tweet.sh showme | file -
/dev/stdin: ASCII text, with very long lines, with no line terminators
$
===============================================================================

* Software environment

curl 7.50.1 (x86_64-pc-linux-gnu) libcurl/7.50.1 GnuTLS/3.5.7 zlib/1.2.8 libidn/1.33 libssh2/1.7.0 nghttp2/1.17.0 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets

I confirmed this problem didn't happen with curl 7.35.0. This version uses
http1.1 by default.
@piroor piroor merged commit 341c5a8 into piroor:master Dec 27, 2016
@piroor
Copy link
Owner

piroor commented Dec 27, 2016

Thanks a lot!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants