Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
Fix netrc parsing on Windows #258
That might be true (although all the Windows, Linux and MacOSX compilers I know -- GCC, clang, Visual C++, icc, Borland C -- support it, or at least handle it gracefully). However, it would benefit all Windows-based projects using cURL, as cURL would not have to be patched to support CR/LF line endings in
Unfortunately I believe "t" is not defined by POSIX: http://pubs.opengroup.org/onlinepubs/009695399/functions/fopen.html
... instead it says: "If the string is one of the following, the file shall be opened in the indicated mode. Otherwise, the behavior is undefined."
To play really safe, I would say that a little #ifdef dance would be appropriate there.
@orgads Can you confirm this works for you? I read in the git-for-windows issue that someone else tested and it works for them but I would like to know about your specific case. It works for my Cygwin POSIX-only build but it looks like you're using a build that's a POSIX/WinAPI mix.
@bagder I'm thinking of making the same change for curlrc to handle the case where the build is Cygwin POSIX-only but the user's curlrc is CRLF, to ensure it's translated properly. Any objection?
Orgad I thought you were using Cygwin. How are you building? The only way I can reproduce is in Cygwin. The curl and libcurl packages available from the cygwin package manager are what I tested with. You seem to be using a build that's using both POSIX and the Windows API I think, because your netrc is "_netrc" and that underscore style is only for builds in Windows that aren't POSIX-only, which means libcurl uses the Windows API.
I still am not able to reproduce this in msys2 with _netrc though I believe it's likely. So far:
I've installed msys2 using the 64-bit installer. There are 3 environments available in msys2: msys2 (64-bit), mingw32 (mingw-w64 32-bit) and mingw64 (mingw-w64 64-bit). mingw32 has
I've built and installed the mingw-w64 version of curl but removed the patch that reportedly fixes the issue. Without the patch everything is fine in the curl tool.
So next I looked at git. I installed the msys2 git package but that uses .netrc. Indeed the issue is present there but that's to be expected. msys2 targets are built with
So I tried building it from source and that had the same problem. I tried building the git nightly from source but that won't build. They don't have time to support mingw git package from what I was told.
So the mystery remains how @orgads git loads _netrc. Which git are you using?
I think though I have a good idea now what caused this issue. I looked at msysgit's source (what mingw git package is made from) and I can see in the version they use msysgit 18.104.22.168 the default file mode is set to binary:
And basically the same thing eventually made its way upstream since then. In the latest git 2.4.2 the default for all fmode is binary as well.
So that seems to me likely to be the issue, though I still prefer to reproduce to be certain. And it's an interesting issue, I think. What obligation do we have as a shared library if the application overrides the default open mode and changes it to binary? Is the onus on the application or on libcurl? If it's libcurl the remedy for this seems to be change each fopen call where we need to read/write text to explicitly specify
I was able to reproduce what I think is happening by modifying simple.c and adding
mingw-w64 does not support
I'm satisfied that the cause is an application setting the default file mode to binary. I'm ready to move on to a solution but I could really use some input.
Thanks for the follow up. I need input from other people involved in the project. We could handle this by explicitly specifying
I don't want to pepper the libcurl source with ifdefs every time an fopen is needed to read text. I don't know if everyone is going to want to adhere to putting "t" every time we need to read text, either. What I could instead do is replace
Others may have better ideas.
@jay, here's my suggestion on how we could proceed with the code:
#ifdef [the t-needing platforms]
... and then we have all fopens in the code base that read text files done like this (seems to be 9 places in src/ and lib/ right now):
... and the version that writes text like this (I believe 4 places):
- Change fopen calls to use FOPEN_READTEXT instead of "r" or "rt" - Change fopen calls to use FOPEN_WRITETEXT instead of "w" or "wt" This change is to explicitly specify when we need to read/write text. Unfortunately 't' is not part of POSIX fopen so we can't specify it directly. Instead we now have FOPEN_READTEXT, FOPEN_WRITETEXT. Prior to this change we had an issue on Windows if an application that uses libcurl overrides the default file mode to binary. The default file mode in Windows is normally text mode (translation mode) and that's what libcurl expects. Bug: #258 (comment) Reported-by: Orgad Shaneh
@bagder Ok I made those changes in e8423f9 but I left alone the mode settings in the vms wrappers that use fopen. Although this is only a libcurl issue as far as I know I made the changes in the curl tool as well.
I think having to specify
Curl and libcurl 7.43.0 Public curl releases: 147 Command line options: 176 curl_easy_setopt() options: 219 Public functions in libcurl: 58 Contributors: 1291 This release includes the following changes: o Added CURLOPT_PROXY_SERVICE_NAME o Added CURLOPT_SERVICE_NAME o New curl option: --proxy-service-name o Mew curl option: --service-name  o New curl option: --data-raw  o Added CURLOPT_PIPEWAIT  o Added support for multiplexing transfers using HTTP/2, enable this with the new CURLPIPE_MULTIPLEX bit for CURLMOPT_PIPELINING  o HTTP/2: requires nghttp2 1.0.0 or later o scripts: add zsh.pl for generating zsh completion o curl.h: add CURL_HTTP_VERSION_2 This release includes the following bugfixes: o CVE-2015-3236: lingering HTTP credentials in connection re-use  o CVE-2015-3237: SMB send off unrelated memory contents  o nss: fix compilation failure with old versions of NSS  o curl_easy_getinfo.3: document 'internals' in CURLINFO_TLS_SESSION o schannel.c: Fix possible SEC_E_BUFFER_TOO_SMALL error o Curl_ossl_init: load builtin modules  o configure: follow-up fix for krb5-config  o sasl_sspi: Populate domain from the realm in the challenge  o netrc: support 'default' token o README: convert to UTF-8 o cyassl: Implement public key pinning o nss: implement public key pinning for NSS backend o mingw build: add arch -m32/-m64 to LDFLAGS o schannel: Fix out of bounds array  o configure: remove autogenerated files by autoconf o configure: remove --automake from libtoolize call o acinclude.m4: fix shell test for default CA cert bundle/path o schannel: fix regression in schannel_recv  o openssl: skip trace outputs for ssl_ver == 0  o gnutls: properly retrieve certificate status o netrc: Read in text mode when cygwin  o winbuild: Document the option used to statically link the CRT  o FTP: Make EPSV use the control IP address rather than the original host o FTP: fix dangling conn->ip_addr dereference on verbose EPSV o conncache: keep bundles on host+port bases, not only host names o runtests.pl: use 'h2c' now, no -14 anymore o curlver: introducing new version number (checking) macros o openssl: boringssl build brekage, use SSL_CTX_set_msg_callback  o CURLOPT_POSTFIELDS.3: correct variable names  o curl_easy_unescape.3: update RFC reference  o gnutls: don't fail on non-fatal alerts during handshake o testcurl.pl: allow source to be in an arbitrary directory o CURLOPT_HTTPPROXYTUNNEL.3: only works with a HTTP proxy o SSPI-error: Change SEC_E_ILLEGAL_MESSAGE description  o parse_proxy: switch off tunneling if non-HTTP proxy  o share_init: fix OOM crash o perl: remove subdir, not touched in 9 years o CURLOPT_COOKIELIST.3: Add example o CURLOPT_COOKIE.3: Explain that the cookies won't be modified  o CURLOPT_COOKIELIST.3: Explain Set-Cookie without a domain  o FAQ: How do I port libcurl to my OS? o openssl: Use TLS_client_method for OpenSSL 1.1.0+ o HTTP-NTLM: fail auth on connection close instead of looping  o curl_setup: Add macros for FOPEN_READTEXT, FOPEN_WRITETEXT  o curl_getdate.3: update RFC reference o curl_multi_info_read.3: added example o curl_multi_perform.3: added example o curl_multi_timeout.3: added example o cookie: Stop exporting any-domain cookies  o openssl: remove dummy callback use from SSL_CTX_set_verify() o openssl: remove SSL_get_session()-using code o openssl: removed USERDATA_IN_PWD_CALLBACK kludge o openssl: removed error string #ifdef o openssl: Fix verification of server-sent legacy intermediates  o docs: man page indentation and syntax fixes o docs: Spelling fixes o fopen.c: fix a few compiler warnings o CURLOPT_OPENSOCKETFUNCTION: return error at once  o schannel: Add support for optional client certificates o build: Properly detect OpenSSL 1.0.2 when using configure o urldata: store POST size in state.infilesize too  o security:choose_mech remove dead code o rtsp_do: remove dead code o docs: many HTTP URIs changed to HTTPS o schannel: schannel_recv overhaul  This release includes the following known bugs: o see docs/KNOWN_BUGS (http://curl.haxx.se/docs/knownbugs.html) This release would not have looked like this without help, code, reports and advice from friends like these: Alessandro Ghedini, Alexander Dyagilev, Anders Bakken, Anthony Avina, Ashish Shukla, Bert Huijben, Brian Chrisman, Brian Prodoehl, Chris Araman, Dagobert Michelsen, Dan Fandrich, Daniel Melani, Daniel Stenberg, Dmitry Eremin-Solenikov, Drake Arconis, Egon Eckert, Frank Meier, Fred Stluka, Gisle Vanem, Grant Pannell, Isaac Boukris, Jens Rantil, Joel Depooter, Kamil Dudka, Linus Nielsen Feltzing, Linus Nielsen Feltzing Feltzing, Liviu Chircu, Marc Hoersken, Michael Osipov, Oren Souroujon, Orgad Shaneh, Patrick Monnerat, Patrick Rapin, Paul Howarth, Paul Oliver, Rafayel Mkrtchyan, Ray Satiro, Sean Boudreau, Tatsuhiro Tsujikawa, Tomas Tomecek, Viktor Szakáts, Ville Skyttä, Yehezkel Horowitz, (43 contributors) Thanks! (and sorry if I forgot to mention someone) References to bug reports and discussions on issues:  = http://curl.haxx.se/mail/lib-2015-04/0095.html  = curl/curl#206  = curl/curl@5b66860#commitcomment-10473445  = curl/curl#141  = curl/curl#198  = http://curl.haxx.se/mail/lib-2015-04/0199.html  = curl/curl#244  = curl/curl#219  = curl/curl#258  = curl/curl#254  = http://curl.haxx.se/libcurl/c/CURLOPT_PROXY_SERVICE_NAME.html  = http://curl.haxx.se/libcurl/c/CURLOPT_SERVICE_NAME.html  = http://curl.haxx.se/docs/manpage.html#--proxy-service-name  = http://curl.haxx.se/docs/manpage.html#--service-name  = http://curl.haxx.se/libcurl/c/CURLOPT_PIPEWAIT.html  = http://curl.haxx.se/libcurl/c/CURLMOPT_PIPELINING.html  = curl/curl#275  = curl/curl#281  = curl/curl#282  = curl/curl#267  = http://curl.haxx.se/mail/lib-2015-05/0056.html  = http://curl.haxx.se/mail/lib-2015-05/0115.html  = http://curl.haxx.se/mail/lib-2015-05/0137.html  = curl/curl#256  = curl/curl#258 (comment)  = curl/curl#292  = https://rt.openssl.org/Ticket/Display.html?id=3621&user=guest&pass=guest  = http://curl.haxx.se/mail/lib-2015-06/0047.html  = http://curl.haxx.se/mail/lib-2015-06/0019.html  = http://curl.haxx.se/docs/adv_20150617A.html  = http://curl.haxx.se/docs/adv_20150617B.html  = curl/curl#244
- Change fopen calls to use FOPEN_READTEXT instead of "r" or "rt" - Change fopen calls to use FOPEN_WRITETEXT instead of "w" or "wt" This change is to explicitly specify when we need to read/write text. Unfortunately 't' is not part of POSIX fopen so we can't specify it directly. Instead we now have FOPEN_READTEXT, FOPEN_WRITETEXT. Prior to this change we had an issue on Windows if an application that uses libcurl overrides the default file mode to binary. The default file mode in Windows is normally text mode (translation mode) and that's what libcurl expects. Bug: curl#258 (comment) Reported-by: Orgad Shaneh