Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

utils: Add fstream wrapper to allow to pass unicode filename on Windows #13878

Merged
merged 4 commits into from Oct 18, 2018

Conversation

@ken2812221
Copy link
Contributor

@ken2812221 ken2812221 commented Aug 4, 2018

If compiled with mingw, use glibc++ extension stdio_filebuf to open the file by FILE* instead of filename.

In other condition, we can use boost::fstream.

@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented Aug 4, 2018

Reviewers, this pull request conflicts with the following ones:
  • #14372 (msvc: build secp256k1 and leveldb locally by ken2812221)
  • #14303 (rpc: Early call once CWallet::MarkDirty in import calls by promag)
  • #14151 (windows: Fix remaining compiler warnings (MSVC) by practicalswift)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@ken2812221 ken2812221 changed the title utils: Add fstream wrapper to allow to pass unicode filename utils: Add fstream wrapper to allow to pass unicode filename on Windows Aug 5, 2018
@ken2812221 ken2812221 force-pushed the iofstream-custom branch from 0d014b1 to 0dba395 Aug 15, 2018
@alexeyneu
Copy link

@alexeyneu alexeyneu commented Aug 15, 2018

but there's override:

#include <fstream>
int main(int argc, char* argv[])
{	
	std::fstream tr(L"hk");
	return 0;
}

compiles fine.
( msys2 )

c++ json_c.cpp -lws2_32 -static-libgcc -static-libstdc++ -static-libgcc -static-libstdc++ -Wl,-Bstatic -lstdc++ -lpthread

@ken2812221
Copy link
Contributor Author

@ken2812221 ken2812221 commented Aug 15, 2018

This is for mingw

@alexeyneu
Copy link

@alexeyneu alexeyneu commented Aug 15, 2018

untitled 1
@ken2812221 unexpected statement

@ken2812221
Copy link
Contributor Author

@ken2812221 ken2812221 commented Aug 16, 2018

@alexeyneu To be clear, this is for the mingw on Ubuntu 18.04 which is what we use for gitian to build release binaries. If you can test this on Ubuntu, that would be great.

@alexeyneu
Copy link

@alexeyneu alexeyneu commented Aug 16, 2018

msys2 .h files are nothing more then msvc copy-paste. it may not work but linux edition has same headers .
i've used it in msvc with np. this applies to constructor only. this stuff isn't avaible for
.open() and others. last time i've seen there smth own-written was windows edition of watcom c .

@alexeyneu
Copy link

@alexeyneu alexeyneu commented Aug 16, 2018

Which is to say, i dont see much sense in testing it coz it will work .
it's not my PR anyway.

@ryanofsky
Copy link
Contributor

@ryanofsky ryanofsky commented Sep 5, 2018

I don't think I understand how this PR works (does it depend on #13866?), or why reimplementing fstream classes is a good solution compared to alternatives. If this is a bug in boost, wouldn't it be better to fix the bug upstream and maybe patch it locally? Or could we drop the use of boost here and instead use std::ifstream and std::ofstream?

It would be helpful to have a clear description of the bug and possible workarounds.

src/fs.h Outdated
@@ -38,6 +42,41 @@ namespace fsbridge {
void* hFile = (void*)-1; // INVALID_HANDLE_VALUE
#endif
};


#if defined WIN32 && defined __GNUC__
Copy link
Contributor

@ryanofsky ryanofsky Sep 5, 2018

Should this be checking for __GLIBCXX__ instead of __GNUC__? It looks like __gnu_cxx::stdio_filebuf might be present in libstdc++ but not libc++ (https://stackoverflow.com/questions/22624503/how-to-get-a-file-descriptor-from-a-stdbasic-ios-for-clang-on-os-x).

Will different workarounds be needed for MSVC and clang?

@ken2812221
Copy link
Contributor Author

@ken2812221 ken2812221 commented Sep 5, 2018

boost::filesystem::i/ofstream calls std::filebuf internally. So if std::filebuf(wchar_t*) exist, boost will call that function (MSVC). Otherwise it will call std::filebuf(char*) which can lead to encoding issue. After c++17, we can use std::i/ofstream(std::filesystem::path). But before then, we can't.

If building with libstdc++, we can use its extension stdio_filebuf by passing FILE *.

Will different workarounds be needed for MSVC and clang?

I don't know about clang for Windows. But for MSVC, we can use boost::filesystem::i/ofstream.

does it depend on #13866?

Without #13866, it would work. But to solve the encoding issue, it requires #13866.

@ken2812221 ken2812221 force-pushed the iofstream-custom branch from 2e84c0b to f2b00f1 Sep 5, 2018
Copy link
Contributor

@ryanofsky ryanofsky left a comment

utACK b11cf20f8255bf20fc3c80117308fcf77424545a. Only changes since last review were tweaking a comment and updating the tests to use utf8 paths for better coverage. Making the test work required #13877, so that PR is now included here (and hopefully can be merged first).

src/fs.h Outdated
// GNU libstdc++ specific workaround for opening UTF-8 paths on Windows.
//
// On Windows, it is only possible to reliably access multibyte file paths through
// `wchar_t` constructors, not `char` APIs. But because the C++ standard doesn't
Copy link
Contributor

@ryanofsky ryanofsky Sep 21, 2018

Word "constructors" on this line should be changed back to "APIs", because this is just referring in general to how multibyte paths need to be accessed on windows, not to stream object constructors in particular.

src/fs.h Outdated
{
public:
ifstream() = default;
ifstream(const fs::path& p, std::ios_base::openmode mode = std::ios_base::in) { open(p, mode); }
Copy link
Contributor

@practicalswift practicalswift Sep 23, 2018

2018-09-22 21:43:16 cpplint(pr=13878): src/fs.h:65:  Constructors callable with one argument should be marked explicit.  [runtime/explicit] [5]

src/fs.h Outdated
{
public:
ofstream() = default;
ofstream(const fs::path& p, std::ios_base::openmode mode = std::ios_base::out) { open(p, mode); }
Copy link
Contributor

@practicalswift practicalswift Sep 23, 2018

2018-09-22 21:43:16 cpplint(pr=13878): src/fs.h:79:  Constructors callable with one argument should be marked explicit.  [runtime/explicit] [5]

@ken2812221 ken2812221 force-pushed the iofstream-custom branch from b11cf20 to 43c7fbb Sep 26, 2018
@ken2812221
Copy link
Contributor Author

@ken2812221 ken2812221 commented Sep 26, 2018

Rebased and fix all nits from the comment

<ClCompile>
<AdditionalOptions>/utf-8 %(AdditionalOptions)</AdditionalOptions>
</ClCompile>
</ItemDefinitionGroup>
</Project>
Copy link
Contributor

@practicalswift practicalswift Oct 2, 2018

Add missing newline at end of file :-)

Copy link
Contributor Author

@ken2812221 ken2812221 Oct 3, 2018

That seems unnecessary for MSVC project file, they can work without the extra newline. But you can add it in your PR. I don't have strong opinion.

@laanwj
Copy link
Member

@laanwj laanwj commented Oct 18, 2018

utACK 86eb3b3

@laanwj laanwj merged commit 43c7fbb into bitcoin:master Oct 18, 2018
2 checks passed
laanwj added a commit that referenced this issue Oct 18, 2018
…ename on Windows

43c7fbb Make MSVC compiler read the source code using utf-8 (Chun Kuan Lee)
f86a571 tests: Add test case for std::ios_base::ate (Chun Kuan Lee)
a554cc9 Move boost/std fstream to fsbridge (Chun Kuan Lee)
86eb3b3 utils: Add fsbridge fstream function wrapper (Chun Kuan Lee)

Pull request description:

  If compiled with mingw, use glibc++ extension `stdio_filebuf` to open the file by `FILE*` instead of filename.

  In other condition, we can use boost::fstream.

Tree-SHA512: b5dbd83e347fb9b2a0c8b1c2c7bd71a272e839ec0617883b2a0ec12506ae9e825373cf6e95b9bcc91d7edc85bf51580a7716b56a9ecaad776bc3ae61638cb3da
@laanwj laanwj removed this from Blockers in High-priority for review Oct 18, 2018
@ken2812221 ken2812221 deleted the iofstream-custom branch Oct 18, 2018
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Mar 24, 2020
Summary:
```
If compiled with mingw, use glibc++ extension stdio_filebuf to open the
file by FILE* instead of filename.

In other condition, we can use boost::fstream.
```

Backport of core [[bitcoin/bitcoin#13878 | PR13878]] and [[bitcoin/bitcoin#15468 | PR15468]] (related bug fix).

Depends on D5539.

Test Plan:
  ninja check
  make check

Run the Gitian builds.

Build and run for Windows. Run `test_bitcoin` on Windows.

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D5539
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Jun 15, 2021
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Jun 16, 2021
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Jun 24, 2021
PastaPastaPasta added a commit to dashpay/dash that referenced this issue Jun 25, 2021
random-zebra added a commit to PIVX-Project/PIVX that referenced this issue Aug 5, 2021
63e0be6 [Remove] By-pass logprint-scanner restriction. (furszy)
280ced3 utils: Fix broken Windows filelock (Chun Kuan Lee)
be89860 utils: Convert Windows args to utf-8 string (Chun Kuan Lee)
e8cfa6e Call unicode API on Windows (Chun Kuan Lee)
1a02a8a tests: Add test case for std::ios_base::ate (Chun Kuan Lee)
2e57cd4 Move boost/std fstream to fsbridge (furszy)
9d8bcd4 utils: Add fsbridge fstream function wrapper (Chun Kuan Lee)
d59d48d utils: Convert fs error messages from multibyte to utf-8 (ken2812221)
9ef58cc Logging: use "fmterr" variable name for errors instead of general "e" that can be used by any other function. (furszy)
dd94241 utils: Use _wfopen and _wreopen on Windows (Chun Kuan Lee)
3993641 add unicode compatible file_lock for Windows (Chun Kuan Lee)
48349f8 Provide relevant error message if datadir is not writable. (murrayn)

Pull request description:

  As the software is currently using the ANSI encoding on Windows, the user's language settings could affect the proper functioning of the node/wallet, to the point of not be able to open some non-ASCII name files and directories.

  This solves the Windows encoding issues, completing the entire bitcoin#13869 work path (and some other required backports). Enabling for example users that use non-english characters in directories and file names to be accepted.

  Backported PRs:
  * bitcoin#12422.
  * bitcoin#12630.
  * bitcoin#13862.
  * bitcoin#13866.
  * bitcoin#13877.
  * bitcoin#13878.
  * bitcoin#13883.
  * bitcoin#13884.
  * bitcoin#13886.
  * bitcoin#13888.
  * bitcoin#14192.
  * bitcoin#13734.
  * bitcoin#14426.

  This is built on top of other two PRs that i have open #2423 and #2369.
  Solves old issues #940 and #2163.

  TODO:
  * Backport `assert_start_raises_init_error` and `ErrorMatch` in TestNode` (bitcoin#12718)

ACKs for top commit:
  Fuzzbawls:
    ACK 63e0be6
  random-zebra:
    ACK 63e0be6 and merging...

Tree-SHA512: cb1f7c23abb5b7b3af50bba18652cc2cad93fd7c2fca9c16ffd3fee34c4c152a3b666dfa87fe6b44c430064dcdee4367144dcb4a41203c91b0173b805bdb3d7d
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants