Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`Link.create` only works for directories on windows #33966

Closed
jakemac53 opened this issue Jul 24, 2018 · 13 comments

Comments

@jakemac53
Copy link
Contributor

commented Jul 24, 2018

example program:

import 'dart:io';

main() async {
  var  link = new Link('link.txt');
  await link.create('original.txt');
}

This will "succeed" on both linux and windows, but on windows it will create a directory symlink which doesn't work (since its actually pointing at a file). You end up getting a permissions error if you try and open the symlink.

As far as I can tell windows has supported symlinks for files as well as directories for a while now (at least since vista), so the above should be able to work, although I am not sure how complex it would be to implement.

It looks like the windows file implementation assumes a directory, although this is not documented or validated.

@bkonyi

This comment has been minimized.

Copy link
Contributor

commented Jul 26, 2018

Currently the Link implementation for Windows relies on Junctions which, as you've pointed out, can only be used for directories. We'll have to rewrite the majority of the Link implementation to use symlinks. Is this blocking you on something in particular?

@jakemac53

This comment has been minimized.

Copy link
Contributor Author

commented Jul 27, 2018

It creates a significantly degraded experience for windows users when using the new build_runner package - especially with vm tests due to #33952.

We have to create a merged output directory containing all the generated and original sources before running tests always - and some users are doing this after each build so they can run custom dev servers in that directory.

For the pub package tests for instance the generated directory currently is ~6GB in size, so copying that over each time before running tests is obviously quite slow. Symlinking the files would be much faster.

Note that we can't symlink entire directories because we have to merge files from two different places (source tree and build cache) into the same logical directory.

@bkonyi

This comment has been minimized.

Copy link
Contributor

commented Jul 27, 2018

Ah gotcha. I'll look into how difficult it would be to update the existing implementation and see if I can spare a few cycles to switch to real symlinks in the next week or so.

@bkonyi

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2018

I've spent some time investigating this, and unfortunately I'm not sure that we can move to the Windows symlink implementation. According to this blog post creating a symbolic link requires that either:

  1. The process creating the symbolic link must be running in a console with elevated privileges
  2. The user invoking the process creating the symbolic link must have Windows 10 Developer Mode enabled for those on Insider Preview versions with builds >= 14972 (this is also included in all stable releases since the Creators update).

Both of these options require that the user have admin rights at one point or another, which seems like quite a lot to ask to simply create symbolic links for files. cc'ing @zanderso for his thoughts.

@jakemac53

This comment has been minimized.

Copy link
Contributor Author

commented Jul 31, 2018

Thanks for looking into this @bkonyi, that is quite unfortunate indeed.

Would it be feasible for us to expose an environment variable which informs you if symlinks are supported on the current platform, which would check for developer mode? Then at least we could provide a warning on the command line to our users asking them to enable it.

This would also likely require some sort of optional argument to the Link.create method to tell it to use the newer windows symlinks as opposed to Junctions, which would throw if they weren't supported?

@zanderso

This comment has been minimized.

Copy link
Member

commented Jul 31, 2018

The current implementation happily creates reparse points targeting files even though the underlying system calls don't traverse the reparse points when we try to access them as files. There are workarounds for this both in Dart code and in file_win.cc. In Dart code, you can do things like this:

import 'dart:io';

main() {
  var link = new Link('link.txt');
  link.createSync('original.txt');
  var linkFile = new File('link.txt');
  if (Platform.isWindows && FileSystemEntity.isLinkSync(linkFile.path)) {
    var linkLink = new Link(linkFile.path);
    var linkTarget = linkLink.targetSync();
    var linkTargetFile = new File(linkTarget);
    var contents = linkTargetFile.readAsStringSync();
    print(contents);
  }
}

Another fix is probably to use code like the following in file_win.cc to manually chase the reparse points when we're asked to interpret them as files:

File* File::Open(Namespace* namespc, const char* path, FileOpenMode mode) {
  // Manually chase symbolic links.
  const intptr_t kSymlinkLimit = 31;
  intptr_t links_followed = 0;
  File::Type type = GetType(namespc, path, false);
  while ((type == kIsLink) && (links_followed < kSymlinkLimit)) {
    path = LinkTarget(namespc, path);
    if (path == NULL) {
      return NULL;
    }
    type = GetType(namespc, path, false);
    links_followed++;
  }
  // Check if we reached the symlink limit, and bail if we did.
  if (type == kIsLink) {
    SetLastError(ERROR_TOO_MANY_LINKS);
    return NULL;
  }
  ...
}

We just have to be careful to chase the reparse points in all the required places (and not any extra places). I would prefer a solution like this until the Windows ecosystem is a little more uniform and we can switch over to proper symlinks.

@jakemac53

This comment has been minimized.

Copy link
Contributor Author

commented Jul 31, 2018

The merged output directories we create need to be fully valid, they are intended for use by arbitrary executables, so a workaround like this won't work unfortunately.

@zanderso

This comment has been minimized.

Copy link
Member

commented Aug 20, 2018

@jakemac53 Ack.

I think we should be very reluctant to add Windows specific named parameters and/or to set environment variables in the dart:io implementation. It is a breaking change, but I think we should do the following:

  • Change Link.create to only make new-style symlinks
  • Change the other Link calls to be able to understand both styles of symlink
  • Update Link docs to indicate the need for an elevated console or dev mode on Windows.

I think the impact of the breaking change would be pretty small since Windows developers are already used to using an elevated console or dev mode for programs that make real symlinks. @bkonyi will correct me if I'm misremembering, but I believe this is the approach taken by Go, for example.

/cc @mit-mit for thoughts on the breaking change here.

@jakemac53

This comment has been minimized.

Copy link
Contributor Author

commented Aug 20, 2018

Do you know what the exception we would end up getting looks like on the Dart side of things if you aren't in an elevated console? If it is something that is specific to this error then I think the suggested approach would be totally fine (modulo discussions about the breaking change). If the error is indistinguishable from other IO errors then I would prefer to have some api which can be used to detect support:

  • While docs on the class/methods certainly are good, they aren't as actionable as me being able to explicitly detect the situation and give the user an error message that tells them they need to run in an elevated console.
  • The people experiencing this issue in the wild will probably not be the people who actually wrote the code using the Link apis - and thus they probably haven't read the docs for Link to know they need to do this.
@zanderso

This comment has been minimized.

Copy link
Member

commented Aug 20, 2018

A cursory glance at the Windows docs indicates that symlink creation failure due to lack of privileges will give ERROR_PRIVILEGE_NOT_HELD. I suspect when privileges are held, but there is a regular file system access error, we will continue to get ERROR_ACCESS_DENIED. It should be possible to bubble this up in the OS Error to the caller.

@bkonyi

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2018

Friendly bump. @mit-mit, what are your thoughts?

@zanderso

This comment has been minimized.

Copy link
Member

commented Aug 30, 2018

I discussed offline with @mit-mit, we agreed that we can proceed with this change, but we'll need to make a breaking change announcement on appropriate mailing lists before landing.

@bkonyi

This comment has been minimized.

Copy link
Contributor

commented Jan 29, 2019

Fixed as of this commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.