Windows: support long paths in open/mkdir/access

[laszlocsomor]

This PR adds support for long paths on Windows in
open(2), mkdir(2), access(2).

Long paths are the ones longer than MAX_PATH (260
characters).

By including io_win32.h we redefine
{open,mkdir,access} to win32_{open,mkdir,access}.

Update 2017-05-04: instead of redefining open/close/etc.
I removed the win32_ prefix from the new function names
and simply import them with a using statement (if _WIN32
is defined). At the same I removed all inclusions of <io.h>
so the code can only use the functions declared in io_win32.h
and imported with using. This is a much safer approach
than the undef-define one, because this allows other
namespaces to declare functions with this name and the
preprocessor won't change those names to win32_open/etc.

These custom functions transform the path argument
to a format expected by the Windows API, then call
_w{open,mkdir,access} which understand these paths
(apparently they wrap the native widechar Windows
API functions).

This PR upstreams the fix for
bazelbuild/bazel#2634

Fixes #2891

[grpc-kokoro]

Thanks for your pull request. The automated tests will run as soon as one of the admins verifies this change is ok for us to run on our infrastructure.

[grpc-kokoro]

Thanks for your pull request. The automated tests will run as soon as one of the admins verifies this change is ok for us to run on our infrastructure.

[laszlocsomor]

@buchgr : FYI

[xfxyjwf]

Please move this to src/google/protobuf/stubs/.... That's where we put all base (and platform related) dependencies.

[laszlocsomor]

Done.

[googlebot]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for the commit author(s). If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.

[laszlocsomor]

292fef7 was also my commit (so the CLA should be fine), just using git-bash instead of my usual git-on-msys, and the author wasn't set up properly.

[laszlocsomor]

Gentle ping.

[xfxyjwf]

The travis test and appveyor test are failing (seems related to this PR). Could you fix them?

[laszlocsomor]

PTAL -- all checks pass and I'm confident there won't be any more changes necessary (unless reviewers request so).

[damienmg]

This only work if checked after the long path prefix. I suggest documenting so if intentional.

[laszlocsomor]

Yes, intentional. Done.

[damienmg]

replace with has_drive_letter || has_longpath_prefix?

[laszlocsomor]

Done.

[damienmg]

All those onliner should probably be declared inline (also my C++ is a bit rusty, this might already be the case for some magic).?

[laszlocsomor]

I think the compiler would auto-inline them where it can. I added the inline declarations anyway but ran into linker errors that I couldn't resolve and didn't want to spend too much time trying.

[laszlocsomor]

gentle ping

[xfxyjwf]

Please squash the commits into one (to reduce noise in revision history).

[xfxyjwf]

Can you use the google::protobuf::internal namespace? That's what we currently use for most other stubs headers.

[laszlocsomor]

Sorry for the long silence. Done, though I moved them to google::protobuf::internal::win32, because the method names are generic enough that they might collide with similarly named functions.

[laszlocsomor]

...if they were in google::protobuf::internal, that is.

[googlebot]

CLAs look good, thanks!

[laszlocsomor]

Re: #2969 (review) :

Please squash the commits into one (to reduce noise in revision history).

Done.

[laszlocsomor]

And rebased on top of upstream/master (9ab7c73).

[laszlocsomor]

Unfortunately ci.bazel.io no longer has the logs from that run (@damienmg : FYI) so I'd just go ahead and merge this PR and fix Bazel breakages if there are any.

[buchgr]

Congrats @laszlocsomor 👍 🎆 🍾

[tri-bao]

Hello,

Does anybody know in which release this fix was included? I try version 3.4.0 (latest version 3.4.1 doesn't have windows compiler binary) on Windows 10 64bit and still get this error. The error like:

D:...\SomeFile.java: No such file or directory

The path is 261-characters long. Shortening the path makes the compiler work.

I checked the source code zip of version 3.4.0, it seems that it includes this patch.

Anybody facing the same problem? What are your solutions apart from shortening the path :)

[laszlocsomor]

I'm not on the Protobuf team, but I assume if the files are in the 3.4.0 zip then they should have been compiled into the release binary.

The files' contents and the import statements are surrounded by #ifdef _MSV_VER ... #endif statements, meaning they'd only be compiled in if built with Visual C++.

@tri-bao Have you tried building your own protoc binary?

A workaround for path lengths is to create a junction that points to the long path, but the junction itself has a short path. Since junctions are just directories with some extra metadata in the filesystem, and the resolution happens in the kernel (I believe), this workaround works both with Command Prompt and Windows Explorer, as well as with all tools that are not longpath-aware.

[tri-bao]

thanks @laszlocsomor , compiling with Visual Studio works! I've created a ticket #3856 for asking why the public precompiled binary was not done with Visual Studio, which makes sense for Windows

[liujisi]

The precompiled protoc was built with mingw32 cross compiled under linux environment for various reasons: mostly for automated building process and eliminating msvc dll dependencies.

Is there anyway to support the long path also under non MSVC? The code used to be guarded by #ifdef _WIN32, and I changed it to _MSV_VER as it created redefinition of those POSIX system APIs - linux-like OS for windows also implements those syscalls. I believe it can be fixed by re-organizing the code. i.e. instead of "using io_win32::mkdir" define our own mkdir wrapper and delegate it to the one in io_win32 when _WIN32 is defined.

[laszlocsomor]

Hey folks, sorry for dropping the ball on this issue.

I opted for defining new implementations in a C++ namespace but using the POSIX function names for the following benefits:

• using io_win32::mkdir drop-in replaces mkdir in the whole compilation unit by just adding that line
• no need to replace mkdir with mkdir_wrapper
• no need to remember calling mkdir_wrapper in new code

@pherl : Does it solve the compilation problem with MinGW if you only #include Linux system headers (such as <sys/stat.h> and <sys/types.h>) if _WIN32 is not defined?

[liujisi]

But the downside would be having to #ifdef _WIN32 and add using directives for each new files requires those posix functions. Having the wrapper made it more explicit and avoid the breakage by including posix headers transitively that we cannot control.

Anyway this should have been fixed in 3.5.0, with changes like: https://github.com/google/protobuf/blob/3.5.x/src/google/protobuf/compiler/objectivec/objectivec_helpers.cc#L64