Forward slashes not recognized in relative symlink targets on Windows

[hakonhagland]

I have built a debug version of perl 5.37.6 on Windows 11 using MinGW-w64 and gcc 11.3.0 from https://winlibs.com/ (using the MSVCRT runtime library). Note that this includes #20271, which fixes the readlink behavior on Windows, see #20460 for more information.

When trying to install Path::Tiny using this perl I get a test failure due to -e operator not working on a symlink. Here is a minimal example:

use feature qw(say);
use strict;
use warnings;
use File::Spec;
use Cwd qw(getcwd);

my $fn = 'foo.txt';
open (my $fh, '>', $fn) or die "Could not open file '$fn': $!";
say $fh "Hello";
close $fh;
my $subdir = "tmp";
if (!-e $subdir) {
    mkdir $subdir or die "Could not create directory '$subdir': $!";
}
my $cwd = getcwd;
#say "cwd = $cwd";
my $link = File::Spec->catfile($subdir, 'bar.txt');
#my $target = File::Spec->catfile(File::Spec->updir, $fn);
#my $target = "..\\$fn";
my $target = "../$fn";
#my $target = "$cwd\\$fn";
#$target =~ s{\\}{/}g;
#say $target;
if (-l $link) {
    unlink $link or die "Could not delete symlink $link: $!";
}
symlink $target, $link or die "Could not create symlink: $!";
if (-e $link) {
    say "-e symlink works";
}
else {
    say "-e symlink does not work";
}
if (-l $link) {
    say "-l symlink works";
}
else {
    say "-l symlink does not work";
}

The output is:

-e symlink does not work
-l symlink works

Expected output is:

-e symlink works
-l symlink works

Note that if I use backward slashes instead of forward slashes when defining the target on line 20, I get the expected output. Similarly, if I use an absolute path for the target (still with forward slashes), I also get the expected output.

Perl configuration

> perl -V
Summary of my perl5 (revision 5 version 37 subversion 6) configuration:
  Commit id: a589f4b93cf3864d7f665d0974e2d36f028b00b6
  Platform:
    osname=MSWin32
    osvers=10.0.22000.1098
    archname=MSWin32-x64-multi-thread
    uname=''
    config_args='undef'
    hint=recommended
    useposix=true
    d_sigaction=undef
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=undef
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='gcc'
    ccflags =' -DWIN32 -DWIN64 -DDEBUGGING -DPERL_TEXTMODE_SCRIPTS -DMULTIPLICITY -DPERL_IMPLICIT_SYS -DUSE_PERLIO -D__USE_MINGW_ANSI_STDIO -fwrapv -fno-strict-aliasing -mms-bitfields'
    optimize='-g -O0'
    cppflags='-DWIN32'
    ccversion=''
    gccversion='11.3.0'
    gccosandvers=''
    intsize=4
    longsize=4
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='long long'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='g++'
    ldflags ='-g -L"c:\perl-debug\lib\CORE" -L"C:\Winlibs64-Gcc11.3-msvcrt\mingw64\lib" -L"C:\Winlibs64-Gcc11.3-msvcrt\mingw64\x86_64-w64-mingw32\lib" -L"C:\Winlibs64-Gcc11.3-msvcrt\mingw64\lib\gcc\x86_64-w64-mingw32\11.3.0"'
    libpth=C:\Winlibs64-Gcc11.3-msvcrt\mingw64\lib C:\Winlibs64-Gcc11.3-msvcrt\mingw64\x86_64-w64-mingw32\lib C:\Winlibs64-Gcc11.3-msvcrt\mingw64\lib\gcc\x86_64-w64-mingw32\11.3.0
    libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    libc=
    so=dll
    useshrplib=true
    libperl=libperl537.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs
    dlext=dll
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags='-shared -g -L"c:\perl-debug\lib\CORE" -L"C:\Winlibs64-Gcc11.3-msvcrt\mingw64\lib" -L"C:\Winlibs64-Gcc11.3-msvcrt\mingw64\x86_64-w64-mingw32\lib" -L"C:\Winlibs64-Gcc11.3-msvcrt\mingw64\lib\gcc\x86_64-w64-mingw32\11.3.0"'


Characteristics of this binary (from libperl):
  Compile-time options:
    DEBUGGING
    HAS_TIMES
    HAVE_INTERP_INTERN
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_HASH_FUNC_SIPHASH13
    PERL_HASH_USE_SBOX32
    PERL_IMPLICIT_SYS
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    PERL_TRACK_MEMPOOL
    PERL_USE_SAFE_PUTENV
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
  Built under MSWin32
  Compiled at Nov  2 2022 00:00:55
  @INC:
    C:/perl-debug/site/lib
    C:/perl-debug/lib

[hakonhagland]

-e symlink does not work

Could this be the problem? See line 1749 in win32.c:

sv_catpvn(new_path, link_target, link_len);

When I set a breakpoint here in gdb I see that new_path is somthing like C:\Users\hakon\perl\test\symlink\ and link_target is ../foo.txt and sv_catpvn joins these two strings into newpath which then becomes: C:\Users\hakon\perl\test\symlink\../foo.txt which is not recognized by DeviceIoControl() at line 1783.

So we cannot have a symlink with a path with backward slashes, at the same time as its target uses forward slashes?

[hakonhagland]

Could this be the problem? See line 1749 in win32.c

Looking closer, it seems like this may not be the problem after all. Maybe rather line 1811 in win32.c?

 handle =
        CreateFileA(path, FILE_READ_ATTRIBUTES,
                    FILE_SHARE_DELETE | FILE_SHARE_READ | FILE_SHARE_WRITE,
                    NULL, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL);

In gdb I see that path is C:\Users\hakon\perl\test\symlink\tmp\bar.txt, but the return value of CreateFileA() depends on whether the target inside bar.txt is ../foo.txt or ..\foo.txt. If it is ../foo.txt it returns INVALID_HANDLE_VALUE.

[sisyphus]

If it is ../foo.txt it returns INVALID_HANDLE_VALUE

I can't find any issue with CreateFileA() and backslash v forwardslash.

According to the attached Inline::C demo, CreateFileA will return INVALID_HANDLE_VALUE if is called with OPEN_EXISTING && the file does not exist.
But if the file exists, then there's no such problem.

CreateFileA will also return INVALID_HANDLE_VALUE if it's called with CREATE_NEW && the file already exists.
Conversely, there's no such issue if the file does not already exist.

But I can't find any evidence that either forwardslashes or backslashes (or a mixture of both) have any bearing on the behaviour.

I've tested the script on both 5.36.0 and 5.37.5 (latest devel release).

Cheers,
Rob

try.txt

[tonycoz]

I've reproduced the behaviour.

From what I can tell the problem is windows, if I stop your test code in the middle, with the symlink created:

C:\Users\Tony\dev\perl\git\perl\win32>type ..\tmp\bar.txt
The system cannot find the file specified.

C:\Users\Tony\dev\perl\git\perl\win32>dir ..\tmp
 Volume in drive C has no label.
 Volume Serial Number is 2824-036D

 Directory of C:\Users\Tony\dev\perl\git\perl\tmp

17/11/2022  03:46 PM    <DIR>          .
17/11/2022  03:46 PM    <DIR>          ..
17/11/2022  03:46 PM    <SYMLINK>      bar.txt [../foo.txt]
               1 File(s)              0 bytes
               2 Dir(s)  214,250,590,208 bytes free

C:\Users\Tony\dev\perl\git\perl\win32>type ..\foo.txt
Hello

C:\Users\Tony\dev\perl\git\perl\win32>

The CreateFileA() call for the symlink has failed, the behaviour of follow_symlinks_to() is irrelevant - the following should only be needed for symlink chains that end in a special file, such as a AF_UNIX socket.

The fix is probably for win32_symlink() to replace any / with \ in the supplied target.

[hakonhagland]

The fix is probably for win32_symlink() to replace any / with \ in the supplied target

@tonycoz Yes that seems like a good idea! Let me know if I should do any further testing.

But I can't find any evidence that either forwardslashes or backslashes (or a mixture of both) have any bearing on the behaviour.

@sisyphus From the perl script you linked to, is it correct that ..\foo.txt is not a symlink? It should be. And it is not the path to the symlink itself that CreateFileA() has problems with. It is the path to the target file that is stored inside the symlink. If that target is a relative path with forward slashes it can make CreateFileA() fail when opening the symlink (not to be confused with opening the target within it).

[sisyphus]

@sisyphus From the perl script you linked to, is it correct that ..\foo.txt is not a symlink?

Yeah, that's correct.
Unfortunately, I took your post out of context and failed to realize that your ..\foo.txt (and ../foo.txt) were symlinks. (My mistake.)
Anyway, @tonycoz to the rescue :-)

On both Windows 11 and Windows 7, I have not been able to build a perl (including blead) that supports symlinks.
For me, the symlink command always fails and sets $! to Operation not permitted.

Thank you, @hakonhagland, for giving me the opportunity to clarify ;-)

Cheers,
Rob

[tonycoz]

On both Windows 11 and Windows 7, I have not been able to build a perl (including blead) that supports symlinks.
For me, the symlink command always fails and sets $! to Operation not permitted.

Under Windows 7 it should work with an administrative/elevated shell.

For Windows 11 you need either an elevated shell, or to enable developer mode in control panel.

For either mklink (the command-line tool) or perl's symlink() should work.

[hakonhagland]

On both Windows 11 and Windows 7, I have not been able to build a perl (including blead) that supports symlinks.

@sisyphus Yes, I forgot to mention that I had turned on "Developer mode" on Windows 11. (Go to the Windows settings app and choose "Update & Security" -> "For developers", and turn on "Developer mode").
As @tonycoz says, I think you need either an elevated shell, or to enable developer mode to have symlinks work on Windows 11.

[sisyphus]

As @tonycoz says, I think you need either an elevated shell, or to enable developer mode to have symlinks work on Windows 11.

Yep - got it. Running in an elevated shell on 11, or as Administrator on 7 is sufficient to enable symlinks for me.
Thanks guys.

Cheers,
Rob