#270: update curl from 7.59.0 to 7.61.0

[patrikjuvonen]

GitHub issue:
#270

Summary:

• Contains various bug fixes, security patches and some other minor changes
• Update callRemote to send the header Content-Type: application/json as supposed to
• Changelog: https://curl.haxx.se/changes.html
• Releases: 7.60.0, 7.61.0

Tests:

• test_fetchRemote.zip (preferable not to run client and server simultaneously)
• fetchRemote URL for unauthenticated requests: http://ptsv2.com/t/mtasa-test-fetchRemote
• fetchRemote URL for authenticated requests: http://ptsv2.com/t/mtasa-test-fetchRemote-auth
• test_callRemote.zip
• callRemote URL for unauthenticated requests: http://ptsv2.com/t/mtasa-test-callRemote
• callRemote URL for authenticated requests: http://ptsv2.com/t/mtasa-test-callRemote-auth
• Please flush all your requests from ptsv2 after testing!

Copy of changelog:

Fixed in 7.61.0 - July 11 2018

Changes:

getinfo: add microsecond precise timers for seven intervals
curl: show headers in bold, switch off with --no-styled-output
httpauth: add support for Bearer tokens
Add CURLOPT_TLS13_CIPHERS and CURLOPT_PROXY_TLS13_CIPHERS
curl: --tls13-ciphers and --proxy-tls13-ciphers
Add CURLOPT_DISALLOW_USERNAME_IN_URL
curl: --disallow-username-in-url

Bugfixes:

CVE-2018-0500: smtp: fix SMTP send buffer overflow
schannel: disable client cert option if APIs not available
schannel: disable manual verify if APIs not available
tests/libtest/Makefile: Do not unconditionally add gcc-specific flags
openssl: acknowledge --tls-max for default version too
stub_gssapi: fix 'unused parameter' warnings
examples/progressfunc: make it build on both new and old libcurls
docs: mention it is HA Proxy protocol "version 1"
curl_fnmatch: only allow two asterisks for matching
docs: clarify CURLOPT_HTTPGET
configure: replace a AC_TRY_RUN with CURL_RUN_IFELSE
configure: do compile-time SIZEOF checks instead of run-time
checksrc: make sure sizeof() is used *with* parentheses
CURLOPT_ACCEPT_ENCODING.3: add brotli and clarify a bit
schannel: make CAinfo parsing resilient to CR/LF
tftp: make sure error is zero terminated before printfing it
http resume: skip body if http code 416 (range error) is ignored
configure: add basic test of --with-ssl prefix
cmake: set -d postfix for debug builds
multi: provide a socket to wait for in Curl_protocol_getsock
content_encoding: handle zlib versions too old for Z_BLOCK
winbuild: only delete OUTFILE if it exists
winbuild: In MakefileBuild.vc fix typo DISTDIR->DIRDIST
schannel: add failf calls for client certificate failures
cmake: Fix the test for fsetxattr and strerror_r
curl.1: Fix cmdline-opts reference errors
cmdline-opts/gen.pl: warn if mutexes: or see-also: list non-existing options
cmake: check for getpwuid_r
configure: fix ssh2 linking when built with a static mbedtls
psl: use latest psl and refresh it periodically
fnmatch: insist on escaped bracket to match
KNOWN_BUGS: restore text regarding #2101
INSTALL: LDFLAGS=-Wl,-R/usr/local/ssl/lib
configure: override AR_FLAGS to silence warning
os400: implement mime api EBCDIC wrappers
curl.rc: embed manifest for correct Windows version detection
strictness: correct {infof, failf} format specifiers
tests: update .gitignore for libtests
configure: check for declaration of getpwuid_r
fnmatch: use the system one if available
CURLOPT_RESOLVE: always purge old entry first
multi: remove a potentially bad DEBUGF()
curl_addrinfo: use same #ifdef conditions in source as header
build: remove the Borland specific makefiles
axTLS: not considered fit for use
cmdline-opts/cert-type.d: mention "p12" as a recognized type
system.h: add support for IBM xlc C compiler
tests/libtest: Add lib1521 to nodist_SOURCES
mk-ca-bundle.pl: leave certificate name untouched
boringssl + schannel: undef X509_NAME in lib/schannel.h
openssl: assume engine support in 1.0.1 or later
cppcheck: fix warnings
test 46: make test pass after year 2025
schannel: support selecting ciphers
Curl_debug: remove dead printhost code
test 1455: unflakified
Curl_init_do: handle NULL connection pointer passed in
progress: remove a set of unused defines
mk-ca-bundle.pl: make -u delete certdata.txt if found not changed
GOVERNANCE.md: explains how this project is run
configure: use pkg-config for c-ares detection
configure: enhance ability to build with static openssl
maketgz: fix sed issues on OSX
multi: fix memory leak when stopped during name resolve
CURLOPT_INTERFACE.3: interface names not supported on Windows
url: fix dangling conn->data pointer
cmake: allow multiple SSL backends
system.h: fix for gcc on 32 bit OpenServer
ConnectionExists: make sure conn->data is set when "taking" a connection
multi: fix crash due to dangling entry in connect-pending list
CURLOPT_SSL_VERIFYPEER.3: Add performance note
netrc: use a larger buffer to support longer passwords
url: check Curl_conncache_add_conn return code
configure: Add dependent libraries after crypto
easy_perform: faster local name resolves by using *multi_timeout()
getnameinfo: not used, removed all configure checks
travis: add a build using the synchronous name resolver
CURLINFO_TLS_SSL_PTR.3: improve the example
openssl: allow TLS 1.3 by default
openssl: make the requested TLS version the *minimum* wanted
openssl: Remove some dead code
telnet: fix clang warnings
DEPRECATE: new doc describing planned item removals
example/crawler.c: simple crawler based on libxml2
libssh: goto DISCONNECT state on error, not SESSION_FREE
CMake: Remove unused functions
darwinssl: allow High Sierra users to build the code using GCC
scripts: include _curl as part of CLEANFILES

---
Fixed in 7.60.0 - May 16 2018

Changes:

Add CURLOPT_HAPROXYPROTOCOL, support for the HAProxy PROXY protocol
Add --haproxy-protocol for the command line tool
Add CURLOPT_DNS_SHUFFLE_ADDRESSES, shuffle returned IP addresses

Bugfixes:

FTP: shutdown response buffer overflow CVE-2018-1000300
RTSP: bad headers buffer over-read CVE-2018-1000301
FTP: fix typo in recursive callback detection for seeking
test1208: marked flaky
HTTP: make header-less responses still count correct body size
user-agent.d:: mention --proxy-header as well
http2: fixes typo
cleanup: misc typos in strings and comments
rate-limit: use three second window to better handle high speeds
examples/hiperfifo.c: improved
pause: when changing pause state, update socket state
multi: improved pending transfers handling => improved performance
curl_version_info.3: fix ssl_version description
add_handle/easy_perform: clear errorbuffer on start if set
darwinssl: fix iOS build
cmake: add support for brotli
parsedate: support UT timezone
vauth/ntlm.h: fix the #ifdef header guard
lib/curl_path.h: added #ifdef header guard
vauth/cleartext: fix integer overflow check
CURLINFO_COOKIELIST.3: made the example not leak memory
cookie.d: mention that "-" as filename means stdin
CURLINFO_SSL_VERIFYRESULT.3: fixed the example
http2: read pending frames (including GOAWAY) in connection-check
timeval: remove compilation warning by casting
cmake: avoid warn-as-error during config checks
travis-ci: enable -Werror for CMake builds
openldap: fix for NULL return from ldap_get_attribute_ber()
threaded resolver: track resolver time and set suitable timeout values
cmake: Add advapi32 as explicit link library for win32
docs: fix CURLINFO_*_T examples use of CURL_FORMAT_CURL_OFF_T
test1148: set a fixed locale for the test
cookies: when reading from a file, only remove_expired once
cookie: store cookies per top-level-domain-specific hash table
openssl: fix build with LibreSSL 2.7
tls: fix mbedTLS 2.7.0 build + handle sha256 failures
openssl: RESTORED verify locations when verifypeer==0
file: restore old behavior for file:////foo/bar URLs
FTP: allow PASV on IPv6 connections when a proxy is being used
build-openssl.bat: allow custom paths for VS and perl
winbuild: make the clean target work without build-type
build-openssl.bat: Refer to VS2017 as VC14.1 instead of VC15
curl: retry on FTP 4xx, ignore other protocols
configure: detect (and use) sa_family_t
examples/sftpuploadresume: Fix Windows large file seek
build: cleanup to fix clang warnings/errors
winbuild: updated the documentation
lib: silence null-dereference warnings
travis: bump to clang 6 and gcc 7
travis: build libpsl and make builds use it
proxy: show getenv proxy use in verbose output
duphandle: make sure CURLOPT_RESOLVE is duplicated
all: Refactor malloc+memset to use calloc
checksrc: Fix typo
system.h: Add sparcv8plus to oracle/sunpro 32-bit detection
vauth: Fix typo
ssh: show libSSH2 error code when closing fails
test1148: tolerate progress updates better
urldata: make service names unconditional
configure: keep LD_LIBRARY_PATH changes local
ntlm_sspi: fix authentication using Credential Manager
schannel: add client certificate authentication
winbuild: Support custom devel paths for each dependency
schannel: add support for CURLOPT_CAINFO
http2: handle on_begin_headers() called more than once
openssl: support OpenSSL 1.1.1 verbose-mode trace messages
openssl: fix subjectAltName check on non-ASCII platforms
http2: avoid strstr() on data not zero terminated
http2: clear the "drain counter" when a stream is closed
http2: handle GOAWAY properly
tool_help: clarify --max-time unit of time is seconds
curl.1: clarify that options and URLs can be mixed
http2: convert an assert to run-time check
curl_global_sslset: always provide available backends
ftplistparser: keep state between invokes
Curl_memchr: zero length input can't match
examples/sftpuploadresume: typecast fseek argument to long
examples/http2-upload: expand buffer to avoid silly warning
ctype: restore character classification for non-ASCII platforms
mime: avoid NULL pointer dereference risk
cookies: ensure that we have cookies before writing jar
os400.c: fix checksrc warnings
configure: provide --with-wolfssl as an alias for --with-cyassl
cyassl: adapt to libraries without TLS 1.0 support built-in
http2: get rid of another strstr
checksrc: force indentation of lines after an else
cookies: remove unused macro
CURLINFO_PROTOCOL.3: mention the existing defined names
tests: provide 'manual' as a feature to optionally require
travis: enable libssh2 on both macos and Linux
CURLOPT_URL.3: added ENCODING section
wolfssl: Fix non-blocking connect
vtls: don't define MD5_DIGEST_LENGTH for wolfssl
docs: remove extraneous commas in man pages
URL: fix ASCII dependency in strcpy_url and strlen_url
ssh-libssh.c: fix left shift compiler warning
configure: only check for CA bundle for file-using SSL backends
travis: add an mbedtls build
http: don't set the "rewind" flag when not uploading anything
configure: put CURLDEBUG and DEBUGBUILD in lib/curl_config.h
transfer: don't unset writesockfd on setup of multiplexed conns
vtls: use unified "supports" bitfield member in backends
URLs: fix one more http url
travis: add a build using WolfSSL
openssl: change FILE ops to BIO ops
travis: add build using NSS
smb: reject negative file sizes
cookies: accept parameter names as cookie name
http2: getsock fix for uploads
all over: fixed format specifiers
http2: use the correct function pointer typedef

[Dutchman101]

The remote scripting functions still behave as expected (the previous cURL version also had no MTA customisations at all), someone should probably review the other changes bundled into this PR (e05c06b, 84ffe95, 3040720) which are unrelated to updating cURL.

Apart from that, I think this is good to merge.

[ghost]

Generated new project files, compiled the solution, and then tested this. Works fine.

[qaisjp]

The application/json update is fine but I am not sure about the other change. I've cherry-picked that change in e9d4c27 — please can you resubmit your other change (in a PR) so we can decide on that separately? Thanks!

[patrikjuvonen]

@qaisjp Thanks for the feedback. I have now reverted the commits regarding the callRemote errno behavior change. I've opened a new issue at #294.

[ccw808]

Where did the changes in config-linux.h come from?

[patrikjuvonen]

@ccw808 curl's 7.61.0 zip package. I suppose some changes to it might have been made by ./configure?

[ccw808]

config-linux.h is not in the zip package. I think Jusonex originally added it to MTA some time ago.
ENABLE_IPV6 needs to be reverted, not sure about the other changes

[patrikjuvonen]

I've now pushed a commit that disables IPv6 that can be cherry picked: https://github.com/patrikjuvonen/mtasa-blue/commit/1e311911f989b2e9aeaf465f3d8ac2b8bc4c8650

lib/curl_config.h is first generated by configure, is then moved to lib/config-linux.h, and then added to the setup script at lib/curl_setup.h to satisfy the Linux platform.

I initially ran configure and make on this new curl release and updated the old lib/config-linux.h with what lib/curl_config.h had. Curl's configure script has been updated in their releases as more definitions have been added and some have been removed/renamed/changed, so to me it made sense to run configure again and make sure it is up-to-date.

However I don't know what our preferred configure options are/were before (some bad documentation practice).

After the previously mentioned commit the configurations are (--disable-ipv6):

configure: Configured to build curl/libcurl:

  curl version:     7.61.0
  Host setup:       x86_64-pc-linux-gnu
  Install prefix:   /usr/local
  Compiler:         gcc
  SSL support:      no      (--with-{ssl,gnutls,nss,polarssl,mbedtls,cyassl,axtls,winssl,darwinssl} )
  SSH support:      no      (--with-libssh2)
  zlib support:     enabled
  brotli support:   no      (--with-brotli)
  GSS-API support:  no      (--with-gssapi)
  TLS-SRP support:  no      (--enable-tls-srp)
  resolver:         POSIX threaded
  IPv6 support:     no      (--enable-ipv6)
  Unix sockets support: enabled
  IDN support:      no      (--with-{libidn2,winidn})
  Build libcurl:    Shared=yes, Static=yes
  Built-in manual:  enabled
  --libcurl option: enabled (--disable-libcurl-option)
  Verbose errors:   enabled (--disable-verbose)
  SSPI support:     no      (--enable-sspi)
  ca cert bundle:   no
  ca cert path:     
  ca fallback:      
  LDAP support:     no      (--enable-ldap / --with-ldap-lib / --with-lber-lib)
  LDAPS support:    no      (--enable-ldaps)
  RTSP support:     enabled
  RTMP support:     no      (--with-librtmp)
  metalink support: no      (--with-libmetalink)
  PSL support:      no      (libpsl not found)
  HTTP2 support:    disabled (--with-nghttp2)
  Protocols:        DICT FILE FTP GOPHER HTTP IMAP POP3 RTSP SMTP TELNET TFTP

I think we can save some space by disabling some of the protocols, at least? Also what about enabling HTTP2?

[ccw808]

I see your config shows
SSL support: no (--with-{ssl,gnutls,nss,polarssl,mbedtls,cyassl,axtls,winssl,darwinssl} )

So https is disabled?

[jushar]

premake5.lua overrides some options (e.g. enabled mbedtls for Linux and SCHANNEL/SSPI for Windows), so I assume it's fine.