fix canonical paths windows #389

xmclark · 2018-10-04T04:03:49Z

This PR addresses problems with canonical paths on windows that were reported in #380. I tested this by building a wasm app with a few dependencies on windows.

I would love somebody to check the work and try running it on another windows box targeting another wasm app 😄 .

Implementation:
Remove the usage of std::fs::canonicalize.

Unit test:
I added a unit test based on reported failures:

a project with local transitive dependencies, see Package collision in lockfile #413

ashleygwilliams · 2018-10-04T14:23:10Z

@xmclark thanks so much! looks like the cargo fmt check is failing.

@steveklabnik would you be willing to give this a go on your windows box?

steveklabnik · 2018-10-04T14:29:58Z

@xmclark what's an easy way to reproduce the bug?

xmclark · 2018-10-04T14:55:21Z

Compiler errors on linux/mac. I think I am not correctly conditionally compiling on windows. I am welcome to suggestions, otherwise I will dig into this tonight. I was pretty sure I tried this on my mac after I built it on my windows...but maybe I made a mistake somewhere 🙃

@steveklabnik the easiest way is creating a wasm-pack app that has rust/crate dependencies. Simply calling wasm-pack build might produce a build error because windows doesn't do well with UNC paths.

I will dig into it later this evening after I get over with my day job. Thanks for jumping on this everyone 😂

steveklabnik · 2018-10-04T17:47:00Z

I spun up a new app via cargo-generate, added semver-parser to its Cargo.toml, modified the code to this:

extern crate cfg_if;
extern crate wasm_bindgen;
extern crate semver_parser;

mod utils;

use wasm_bindgen::prelude::*;

#[wasm_bindgen]
extern {
    fn alert(s: &str);
}

#[wasm_bindgen]
pub fn greet() {
    alert(&format!("{:?}", semver_parser::range::parse("1.0.0").unwrap()));
}

and here's the output:

> wasm-pack build

  [1/9] Checking `rustc` version...
  [2/9] Checking crate configuration...
  [3/9] Adding WASM target...
  [4/9] Compiling to WASM...
  [5/9] Creating a pkg directory...
  [6/9] Writing a package.json...
  :-) [WARN]: Field 'description' is missing from Cargo.toml. It is not necessary, but recommended
  :-) [WARN]: Field 'repository' is missing from Cargo.toml. It is not necessary, but recommended
  :-) [WARN]: Field 'license' is missing from Cargo.toml. It is not necessary, but recommended
  [7/9] Copying over your README...
  [8/9] wasm-bindgen already installed...
  [9/9] Running WASM-bindgen...
  :-) Done in 0 seconds
| :-) Your wasm pkg is ready to publish at "\\\\?\\C:\\Users\\Steve Klabnik\\tmp\\wasm-bug\\pkg".

So, I can't reproduce...

ashleygwilliams · 2018-10-04T18:35:26Z

ok so- taking a look at this PR i'm starting to have the suspicion that this is not a wasm-pack bug but potentially something that needs to be fixed either in the plugin or in parcel itself.

the first step to sorting this that i would want to see is a test for wasm-pack that can reproduce the bug you've found. we have our tests run on appveyor and there may very well be a gap in our tests, but fundamentally i think the approach we have for paths on windows is reasonable.

to say a bit more here, we have to dig into the differences between canonicalize and canonicalize_path. my understanding (again could be wrong) is that canonicalize asks your OS for the path and here canonicalize will, on a windows machine, return a UNC path (e.g. \\? which u can see in Steve's output above). canonicalize_path will return a non UNC path, which is to say it will use C:. So, my understanding is that this non UNC path could be wrong, which is why canonicalize is what we use here.

my main curiosity is exactly why the UNC path doesn't work in some cases. my assumption is that there are more constraints be added in the tooling you've layered here, but i genuinely am not sure.

hopefully this was vaguely helpful? again, starting with a test that can reproduce this issue is the best step for moving forward i think. i also would like to dive into the plugin and Parcel code, so if you have links to particular bits of those code bases that could help illuminate how these paths are being used that'd be great!

xmclark · 2018-10-05T02:47:17Z

@steveklabnik @ashleygwilliams thanks for the detailed example. I will be a bit more explicit in my example next time and provide a similar example.

I have tried a few more things on my windows box with both git bash and cmd:

I also was not able to reproduce the error using your example. The default dependencies in the template seem fine on windows.

I am able to reproduce a problem by installing a different package: crc.

I followed the same steps:

Install latest wasm-pack with git url,
cargo generate
Add crc = "1.8.1" to Cargo.toml
wasm-pack build

Mac@DESKTOP-NNBM40V MINGW64 ~/dev/reproduce-wasm-pack-pr-389 (master)
$ wasm-pack build

  [1/9] Checking `rustc` version...
  [2/9] Checking crate configuration...
  [3/9] Adding WASM target...
/ [4/9] Compiling to WASM...
Process exited with exit code: 101: Compilation of your program failed. stderr:

   Compiling crc v1.8.1
error: couldn't read "\\\\?\\C:\\Users\\Mac\\dev\\reproduce-wasm-pack-pr-389\\target\\wasm32-unknown-unknown\\release\\build\\crc-087346432c34fa56\\out/crc16_constants.rs": The filename, directory name, or volume label syntax is incorrect. (os error 123)
 --> C:\Users\Mac\.cargo\registry\src\github.com-1ecc6299db9ec823\crc-1.8.1\src\crc16.rs:8:1
  |
8 | include!(concat!(env!("OUT_DIR"), "/crc16_constants.rs"));
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

error: Could not compile `crc`.

To learn more, run the command again with --verbose.

Running cargo build --target wasm32-unknown-unknown --release --lib will complete successfully.

Mac@DESKTOP-NNBM40V MINGW64 ~/dev/reproduce-wasm-pack-pr-389 (master)
$ cargo build --target wasm32-unknown-unknown --release --lib
   Compiling crc v1.8.1
   Compiling reproduce-wasm-pack-pr-389 v0.1.0 (C:\Users\Mac\dev\reproduce-wasm-pack-pr-389)
warning: function is never used: `set_panic_hook`
  --> src\utils.rs:11:9
   |
11 |         pub fn set_panic_hook() {}
   |         ^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: #[warn(dead_code)] on by default

    Finished release [optimized] target(s) in 0.76s

I'm not sure how I could make this in a unit test. Ideas?

steveklabnik · 2018-10-05T14:21:43Z

So! I can reproduce this now, but it's only the crc crate

> wasm-pack build

  [1/9] Checking `rustc` version...
  [2/9] Checking crate configuration...
  [3/9] Adding WASM target...
- [4/9] Compiling to WASM...
Compilation of your program failed. stderr:

    Updating crates.io index
   Compiling build_const v0.2.1
   Compiling crc v1.8.1
error: couldn't read "\\\\?\\C:\\Users\\Steve Klabnik\\tmp\\wasm-bug\\target\\wasm32-unknown-unknown\\release\\build\\crc-d224ab13f0f73b57\\out/crc16_constants.rs": The filename, directory name, or volume label syntax is incorrect. (os error 123)
 --> C:\Users\Steve Klabnik\.cargo\registry\src\github.com-1ecc6299db9ec823\crc-1.8.1\src\crc16.rs:8:1
  |
8 | include!(concat!(env!("OUT_DIR"), "/crc16_constants.rs"));
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Compiling crc locally, both work:

~\.cargo\registry\src\github.com-1ecc6299db9ec823\crc-1.8.1> cargo +nightly build --target=wasm32-unknown-unknown
   Compiling build_const v0.2.1
   Compiling crc v1.8.1 (C:\Users\Steve Klabnik\.cargo\registry\src\github.com-1ecc6299db9ec823\crc-1.8.1)
    Finished dev [unoptimized + debuginfo] target(s) in 1.48s
~\.cargo\registry\src\github.com-1ecc6299db9ec823\crc-1.8.1> cargo +nightly build
   Compiling crc v1.8.1 (C:\Users\Steve Klabnik\.cargo\registry\src\github.com-1ecc6299db9ec823\crc-1.8.1)
    Finished dev [unoptimized + debuginfo] target(s) in 0.36s

(They also do in release mode, no interesting output though, so I didn't bother including it)

I have no idea what's going on here.

xmclark · 2018-10-06T03:09:14Z

Yeah this is an especially odd one!

I think the main reason is how the std canonicalize function works on windows. It is pretty well documented in this issue.

Calling fs::canonicalize on windows produces a path with a few extra leading characters e.g. \\?C:\blah. This kind of path is not understood by cargo (and many other programs on windows). It is also the same cause for another issue reported recently about the pkg path not being printed correctly.

This PR implements a solution that follows a strategy suggested by a few others in that rust-lang issue.

steveklabnik · 2018-10-06T14:23:05Z

That issue of mine has nothing to do with canonicalization, it has to do with an extra layer of escaping.

…

On Oct 5, 2018, at 11:09 PM, Mackenzie Clark ***@***.***> wrote: Yeah this is an especially odd one! I think the main reason is how the std canonicalize function works on windows. It is pretty well documented in this issue. Calling fs::canonicalize on windows produces a path with a few extra leading characters e.g. \\?C:\blah. This kind of path is not understood by cargo (and many other programs on windows). It is also the same cause for another issue reported recently about the pkg path not being printed correctly. This PR implements a solution that follows a strategy suggested by a few others in that rust-lang issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

xmclark · 2018-10-06T15:10:16Z

Agreed. A separate issue is the display as a UNC path.

If it were un-escaped, it would look like this:
| :-) Your wasm pkg is ready to publish at "\\?\C:\Users\Steve Klabnik\tmp\wasm-bug\pkg"

I would expect it to be printed without the leading \\?\.

Wow! This comment is starting to sound a lot like an issue report... I made this fix as well in this PR. If the project would prefer that be a separate PR, I'm up for removing that extra revision.

ashleygwilliams · 2018-10-08T14:07:17Z

hey! so- this issue PR convo is indeed getting long. @xmclark in order to accept this PR, i need a test case that demonstrates this bug. let me know if you need help or have questions about that!

ashleygwilliams · 2018-10-10T15:30:12Z

@xmclark following up- are you around on a sync channel anywhere? would it be helpful to discuss a bit?

xmclark · 2018-10-13T17:53:02Z

@ashleygwilliams hey sorry for late reply. Long week.

I apologize if there has been some miscommunication on my part. I would like to do whatever I need to help reviewers. I was under the impression that I had provided all the needed information for this to be reviewed. The description references the issue number that this PR fixes. This comment demonstrates how to reproduce the bug.

@technetos was super helpful and got me onto the rust-lang and the rust programming language discords last week. Are those good channels for discussion? Thanks!

steveklabnik · 2018-10-14T20:41:38Z

That comment gives us some kind of reproduction but it doesn’t translate into a test that we could use to ensure that it won’t regress, and it also is “too big” to really sort out what the root issue actually is. In my understanding a unit test is what’s being sought here.

…

On Oct 13, 2018, at 10:53 AM, Mackenzie Clark ***@***.***> wrote: @ashleygwilliams hey sorry for late reply. Long week. I apologize if there has been some miscommunication on my part. I would like to do whatever I need to help reviewers. I was under the impression that I had provided all the needed information for this to be reviewed. The description references the issue number that this PR fixes. This comment demonstrates how to reproduce the bug. @technetos was super helpful and got me onto the rust-lang and the rust programming language discords last week. Are those good channels for discussion? Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

raggy-rs · 2018-10-18T22:39:54Z

src/command/utils.rs

+        }
+    }
+    else {
+        /// Strips UNC from canonical path on Windows.


this comment is missplaced

raggy-rs · 2018-10-18T22:46:05Z

Maybe the repro of #413 could be made into a unit test since it will also get fixed by this PR. The underlying issue there is that when you call cargo from a std::process::Command with a UNC Path cargo cannot handle it.

ashleygwilliams · 2018-10-18T22:53:10Z

@xmclark you can find me on most rust platforms as @ag_dubs if you'd like to talk sync- and no worries on the delay- it's important to fix this but i have also been traveling :)

xmclark · 2018-10-30T02:55:23Z

I'm not sure what's up with the appveyor build. This code is up for review! I have decided on a different implementation which avoids some of the problems with std::canonicalize. I also added two tests that capture the reported fail cases.

fitzgen · 2018-10-30T11:54:32Z

I'm not sure what's up with the appveyor build.

If you rebase on master, the appveyor builds should be fixed now. See #417 for details.

fitzgen · 2018-10-30T12:02:32Z

This PR looks good to me. I'm unfamiliar with the absolutize crate, but if it fixes the problem then that sgtm. Do either of @ashleygwilliams or @alexcrichton want to verify that this is the correct fix?

steveklabnik · 2018-10-30T14:10:30Z

tests/all/utils/fixture.rs

+
+            [dependencies]
+            wasm-bindgen = "=0.2.21"
+            crc = "*"


this is extremely fragile

I'm not sure about this strategy overall

yeah i will need to talk a look at this a bit closer but i am not in favor of * deps in general, and i would prefer a more general use case test

Yeah I also agree, and I'm glad you both point it out.

In this case, I was having a problem installing crc = "1.8.1". I left it as * because pinning the version makes sense today, but if there was a change tomorrow in crc, we wouldn't catch it.

I can pin the version if that is preferred 😄

EDIT:

I think I may have misunderstood Steve's concern. Is the concern with the strategy of testing a specific crate? If I need to revise this test to pin point the exact problem that crc introduces, I can try and investigate. I have some ideas. Just will be more time.

rpjohnst · 2018-10-30T17:44:44Z

Cargo.toml

@@ -21,6 +21,7 @@ indicatif = "0.9.0"
 lazy_static = "1.1.0"
 openssl = { version = '0.10.11', optional = true }
 parking_lot = "0.6"
+path-absolutize = "1.1.1"


Was there a particular reason for using path-absolutize, which implements its own custom logic, rather than just calling GetFullPathName on Windows? I expect the latter would be more robust.

alexcrichton · 2018-10-30T18:01:33Z

I'm unfortunately away for windows for awhile now so I can't verify this, but sorry if this has already been asked by why is canonicalization done? Could the original path just be plumbed around?

xmclark · 2018-11-04T20:41:05Z

@alexcrichton that is a good point. I removed absolutize (and canonicalize) and it just works. It seems that we do not need to canonicalize the path at all.

alexcrichton · 2018-11-06T20:33:32Z

Ok great!

Is this ready to go in that case? Or are there final points to take care of?

xmclark · 2018-11-06T22:00:00Z

Thanks!

I think I could use some feedback on how to fix my unit test that uses the crc-rs crate. The test is valid, but I have received feedback that not pinning the dependency version is bad. I am also a little concerned that we are NOT testing for the underlying problem. I think I understand that fundamental issue, but recreating it in a test fixture would could be really complicated. The crc-rs crate has a custom build step that generates some rust code in the out_dir. When included, the path to the generated file is in the windows UNC path format, and cargo chokes on it. This only happens when building with wasm-pack and this PR fixes that issue.

I am interested in any suggestions on this!

alexcrichton · 2018-11-07T16:25:08Z

I think it's probably ok to not include that test and we can mostly wing it for now. This is a particularly difficult aspect to test, but we've got enough users that it should crop up quickly again if it comes up!

steveklabnik · 2018-11-07T18:24:03Z

👍

…

On Nov 7, 2018, at 8:25 AM, Alex Crichton ***@***.***> wrote: I think it's probably ok to not include that test and we can mostly wing it for now. This is a particularly difficult aspect to test, but we've got enough users that it should crop up quickly again if it comes up! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

xmclark · 2018-11-08T03:32:59Z

Thanks @steveklabnik @alexcrichton! I have removed the test.

alexcrichton · 2018-11-08T03:47:58Z

👍

ashleygwilliams added the needs review label Oct 4, 2018

ashleygwilliams added the needs cargo fmt label Oct 4, 2018

ashleygwilliams added this to the 0.6.0 milestone Oct 4, 2018

xmclark changed the title ~~fix canonical paths windows~~ WIP fix canonical paths windows Oct 5, 2018

xmclark force-pushed the fix-canonical-paths-on-windows branch from b8d90c8 to ef68e5b Compare October 5, 2018 05:18

xmclark changed the title ~~WIP fix canonical paths windows~~ fix canonical paths windows Oct 5, 2018

huangjj27 mentioned this pull request Oct 8, 2018

bugfix(command/build.rs): Cancel Escape #396

Merged

3 tasks

ashleygwilliams added needs tests please add tests to this PR needs reproduction and removed needs cargo fmt needs review labels Oct 9, 2018

ashleygwilliams added the waiting on response label Oct 12, 2018

raggy-rs mentioned this pull request Oct 18, 2018

Package collision in lockfile #413

Closed

raggy-rs reviewed Oct 18, 2018

View reviewed changes

src/command/utils.rs Outdated

}

}

else {

/// Strips UNC from canonical path on Windows.

Copy link

raggy-rs Oct 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment is missplaced

xmclark changed the title ~~WIP - fix canonical paths windows~~ fix canonical paths windows Oct 30, 2018

xmclark force-pushed the fix-canonical-paths-on-windows branch from 87c9ae0 to a2a3520 Compare October 30, 2018 14:05

steveklabnik reviewed Oct 30, 2018

View reviewed changes

rpjohnst reviewed Oct 30, 2018

View reviewed changes

xmclark changed the title ~~fix canonical paths windows~~ WIP - fix canonical paths windows Nov 4, 2018

xmclark force-pushed the fix-canonical-paths-on-windows branch from 0a9c770 to 0c045c2 Compare November 8, 2018 01:36

do not canonicalize the crate path

b3d62e1

xmclark force-pushed the fix-canonical-paths-on-windows branch from 0c045c2 to b3d62e1 Compare November 8, 2018 03:17

xmclark changed the title ~~WIP - fix canonical paths windows~~ fix canonical paths windows Nov 8, 2018

alexcrichton merged commit 91e3293 into rustwasm:master Nov 8, 2018

xmclark mentioned this pull request Nov 22, 2018

Relative path crate dependencies are canonicalized badly on windows #380

Closed

ashleygwilliams added changelog - fix changelog - maintenance and removed needs reproduction needs tests please add tests to this PR waiting on response changelog - fix labels Jan 11, 2019

danwilhelm mentioned this pull request Jan 15, 2019

wasm-pack breaks dependencies that use code generation #460

Closed

konstin mentioned this pull request Apr 18, 2020

README path on windows PyO3/maturin#292

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix canonical paths windows #389

fix canonical paths windows #389

xmclark commented Oct 4, 2018 •

edited

ashleygwilliams commented Oct 4, 2018

steveklabnik commented Oct 4, 2018

xmclark commented Oct 4, 2018

steveklabnik commented Oct 4, 2018

ashleygwilliams commented Oct 4, 2018 •

edited

xmclark commented Oct 5, 2018 •

edited

steveklabnik commented Oct 5, 2018 •

edited

xmclark commented Oct 6, 2018

steveklabnik commented Oct 6, 2018 via email

xmclark commented Oct 6, 2018 •

edited

ashleygwilliams commented Oct 8, 2018

ashleygwilliams commented Oct 10, 2018

xmclark commented Oct 13, 2018

steveklabnik commented Oct 14, 2018 via email

raggy-rs Oct 18, 2018

raggy-rs commented Oct 18, 2018 •

edited

ashleygwilliams commented Oct 18, 2018

xmclark commented Oct 30, 2018

fitzgen commented Oct 30, 2018

fitzgen commented Oct 30, 2018

steveklabnik Oct 30, 2018

ashleygwilliams Oct 30, 2018

xmclark Oct 30, 2018 •

edited

rpjohnst Oct 30, 2018

alexcrichton commented Oct 30, 2018

xmclark commented Nov 4, 2018

alexcrichton commented Nov 6, 2018

xmclark commented Nov 6, 2018

alexcrichton commented Nov 7, 2018

steveklabnik commented Nov 7, 2018 via email

xmclark commented Nov 8, 2018

alexcrichton commented Nov 8, 2018

fix canonical paths windows #389

fix canonical paths windows #389

Conversation

xmclark commented Oct 4, 2018 • edited

ashleygwilliams commented Oct 4, 2018

steveklabnik commented Oct 4, 2018

xmclark commented Oct 4, 2018

steveklabnik commented Oct 4, 2018

ashleygwilliams commented Oct 4, 2018 • edited

xmclark commented Oct 5, 2018 • edited

steveklabnik commented Oct 5, 2018 • edited

xmclark commented Oct 6, 2018

steveklabnik commented Oct 6, 2018 via email

xmclark commented Oct 6, 2018 • edited

ashleygwilliams commented Oct 8, 2018

ashleygwilliams commented Oct 10, 2018

xmclark commented Oct 13, 2018

steveklabnik commented Oct 14, 2018 via email

raggy-rs Oct 18, 2018

Choose a reason for hiding this comment

raggy-rs commented Oct 18, 2018 • edited

ashleygwilliams commented Oct 18, 2018

xmclark commented Oct 30, 2018

fitzgen commented Oct 30, 2018

fitzgen commented Oct 30, 2018

steveklabnik Oct 30, 2018

Choose a reason for hiding this comment

ashleygwilliams Oct 30, 2018

Choose a reason for hiding this comment

xmclark Oct 30, 2018 • edited

Choose a reason for hiding this comment

rpjohnst Oct 30, 2018

Choose a reason for hiding this comment

alexcrichton commented Oct 30, 2018

xmclark commented Nov 4, 2018

alexcrichton commented Nov 6, 2018

xmclark commented Nov 6, 2018

alexcrichton commented Nov 7, 2018

steveklabnik commented Nov 7, 2018 via email

xmclark commented Nov 8, 2018

alexcrichton commented Nov 8, 2018

xmclark commented Oct 4, 2018 •

edited

ashleygwilliams commented Oct 4, 2018 •

edited

xmclark commented Oct 5, 2018 •

edited

steveklabnik commented Oct 5, 2018 •

edited

xmclark commented Oct 6, 2018 •

edited

raggy-rs commented Oct 18, 2018 •

edited

xmclark Oct 30, 2018 •

edited