fresh install fails on Windows: "error: could not rename component file from '... \rust-docs\share/doc/rust/html' ..." #1912
Comments
|
Thank you for generating this report. It does look to me like a probable anti-virus problem. You mentioned in 1723 that you have procmon traces - have a look in that at the time of the error, are there any other processes with handles open anywhere in that subdirectory? I'm happy to look through if you can upload the PML (as a zip/tar) to s3 or seed it as a torrent or some such... |
|
@rbtcollins I have placed the zipped PML log (11M) here: http://www.usn.pp.ru/tmp/rustup-init--procmon.2019-06-24.zip . Unfortunately could not find anything suspicious so far... Please let me know whether the log is good when your time allows... |
|
The log doesn't have enough info in it (or its not what I think it is, or Defender's implementation has changed radically from Windows 8 to 10). So this is the event that fails: But there are no events in the trace from other processes than rustup-init.exe - I suspect you've got a filter on the process name. Check that you aren't filtering for just rustup-init.exe in the filter dialog when you make the trace - something like "Path contains .rustup" would be a good filter - the default filters to exclude procmon.exe itself and so on are fine, but we do want to see activity from other processes within this directory structure. After you've done the trace and saved the file, you can then edit the filter and add filter to exclude events from rustup-init.exe - I'd expect that then you'd see defender or whatever is causing the issue operating on those files. I would find the event in the first place by adding a filter on result for 'ACCESS DENIED' |
|
@rbtcollins I have finally managed to guess the time values to filter out all the events withing 2 sec timeframe around the failure event :) :
All the excludes are the standard ones suggested by procmon, except the one of mine that restricts the time. Please find the resulting zipped PML log (0.5M) here: http://www.usn.pp.ru/tmp/rustup-init--procmon.2019-06-24--two-seconds-around.zip |
|
Ok so thats super interesting, - definitely other processses in the trace, but none operating on .rustup files. I suspect we need to get into kernel tracing at this point - ETW / WPR https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-8.1-and-8/hh448205(v=win.10) - - basically looking for the cause of the access denied. Before doing that, please try building and running master - there's some unpacking IO changes that might make this better - or worse - and would allow me to make the following statement: We make that dir with the same code for both the binaries and docs, so either both should succeed or both should fail, which is why the third-party-process holding a handle open is the most likely scenario. |
|
@rbtcollins Ok, I'll see if I succeed with building the master on the weekend. Meanwhile could you describe your ideas on further troubleshooting in some more detail? What the next steps may look like, and how to proceed? |
|
@s-n-ushakov basically we're at the edge of my ready-to-use knowledge, so I'm going to be vague. But something like: enable kernel event tracing on all file operations, run the failing scenario, disable event tracing, look through the trace which hopefully will have much more detail than procmon, because it wires into the kernel itself rather than purely userspace. |
|
I tried to install rust by chocolatey and got the similar error.
After I turned off my vaccine program, it was successful. |
|
I have the same issue here with the Avira anti-virus. Turning off the "Real-Time Protection" before running rustup-init.exe makes it work. |
|
This failed about 5 times for me and then I found this thread. It worked after I deselected the box in the McAfee Access Protection Properties to "Prevent McAfee services from being stopped" and then stopped the `McAfee McShield" service. It is also known as the "McAfee On-Access Scanner service". This was the error:
|
|
@rbtcollins I have to confess that I failed to do the ETW / WPR exercise so far, as my HDD with Windows died. Sorry... Now I am busy with gradual migration to Ubuntu, so it is not very much likely that I will be able to add more to this investigation, at least in the near future. But it is good to know that numerous colleagues here are reporting their observations of problem correlation with antivirus software... |
|
Hi. |
|
Same. Turning off the McAfee On-Access Scanner fixed the |
|
Badly behaved scan-on-write virus scanners are becoming a great pain to me. I wonder if there's any way we can detect and mitigate this. |
|
If I could disable my anti-virus temporarily I'm sure it would fix this issue. However, at work I am unable to disable the anti-virus program. This makes it impossible for me to use Is it possible to make it so rustup won't give up after the very first attempt at moving? Or give us an option to, instead of moving the files, try copying to destination and deleting from source instead? |
|
Handy to know, thanks @pitaj -- I hope to get a release done soonish that will therefore solve this a bit. |
|
I have the same problem. Corporate laptop with McAfee that can not be turn off. I need this release in order to start learning rust. Please hurry :-) |
|
@pitaj Can you please share the binary you've build from the latest master? I cant install |
|
Found a way: downloaded the latest |
|
Hello, I fix it by deactivating the antivirus while it is installed. |
|
@esbgo97, most corporate laptops do not allow turning off the Antivirus. |
|
I keep having the same lately on corporate Windows 10. There is no 3rd-party antivirus though, just built-in Windows Defender. |
Yay, I can confirm that nuking |
Unfortunately it does not work for me, I am still getting the following I have a corporate with McAfee enabled, and this is preposterous that docs are blocked from being moved (hey, if it was a virus, it would already have infected my computer). |
|
It is a big pain, I agree. McAfee is among the worst offenders :( |
|
If it says "retry renaming" instead of failing then the fix is working. |
|
To me it says "retry renaming" for a while and then it fails on Corporate McAfee enabled, cannot disable. Issue is not restricted to fresh install. I did a |
|
@ChrisGreenaway success! cool; permission denied on the job object, interesting - I'd like to dig into that deeper at some point, at least I have something to go on. If you could poke around and see what permissions you have / need on the default job object in your win10 user session, that would be useful. You might find the effective-limits.rs test suite to be useful too. @ChrisDenton Yes, most of the time, you'd need to be unlucky to race on a file, but its not safe-safe, though we could of course retry on that too. And yes, that behaviour is exactly the problem we're working around. I'm not sure of the distinction you're drawing here between yield and sleep: we're giving up the CPU and trying each time. But McAfee is multi-threaded; there is often no gap between files: etc |
|
@ChrisGreenaway, oh, with 26 retries you should be able to install the entire toolchain at once, no need for profile minimal. |
|
@rbtcollins I mean there should literally be some time after a file handle is a closed and before another is open. Even if they have a tight loop it takes time for the OS to check arguments do security checks, etc even in multithreaded code you can get lucky. My theory is that it may be possible to slot between an open and a close. A very rough example would be: let mut result = Ok(())
// Number of iterations is probably way too large
for _ in 0..100_000 {
result = fs::rename(src, dest);
if let Err(e) = &result {
if e.kind() == io::ErrorKind::PermissionDenied {
// yield this thread's timeslice back to the OS
// so it can run other threads on the same CPU core.
std::thread::yield_now();
continue;
}
}
break;
}
result.chain_err(|| ErrorKind::RenamingFile {
name,
src: PathBuf::from(src),
dest: PathBuf::from(dest),
}) |
|
@rbtcollins - I got that permission denied on the job object error even when running as a machine adminstrator. You asked me to "see what permissions [I] have / need on the default job object in [my] win10 user session". How would I do that? |
|
@ChrisDenton oh, yield_now, which is more-or-less sleep(0), except that there are some guarantees about not causing processor migrations on Windows. No, that is not of any help at all, because McAfee is holding these locks because it is running concurrently with Rust. The least number of hardware threads in a laptop you can feasibly buy today is 4 I think; possibly 2. The traces we have show clearly that multiple mcafee threads are holding handles open across each others open and closes with no windows between them, as I sketched in my last comment. Allowing more another 20ms of mcafee time (which is what yield_now will be more or less as it is backed by SwitchToThread, and desktop Windows scheduler granularity), is basically what we do now, except that by not backing off, you're just probing more often in a wait-loop. But by all means test it ! what we need is empirical data here :). Like @ben-spiller's extraction test, doing experiments is valuable and useful - but please do post the patch or code so we can make sure we all understand what was tested ;) |
|
@ChrisGreenaway process explorer perhaps? I'm not 100% sure, job objects are still a new API for me :). Basically we're calling https://docs.microsoft.com/en-us/windows/win32/api/jobapi2/nf-jobapi2-queryinformationjobobject like so: which should get the default job object, if any. The API says we have to have JOB_OBJECT_QUERY on the handle, but we're using the implied handle. https://docs.microsoft.com/en-us/windows/win32/procthread/job-object-security-and-access-rights documents those, but it's not clear how we wouldn't have access to read extendedlimitinformation on the associated job object,... unless perhaps there's some corporate setup restruction that is ultra limited - but perhaps we can open the job manually with appropriate permissions - anyhow, basically there is an investigation we need to do, ... |
|
Ok, I went away and did some tests. I don't have a McAfee so I made my own. You can find my code here. It seems to work but there could of course be other factors at play that I'm missing (or my AV simulator is broken in some way). I've made a rustup fork ( To be clear, I'm not saying this will definitely work or is the best idea. But I am interested in file handling in Windows and would at least like to rule out this possibility. |
|
@ChrisDenton - I ran |
|
@rbtcollins - is there a test you could add to effective-limits to check the job object? Then I could clone the project and run |
|
@ChrisGreenaway Fair, enough. Thanks for testing. |
|
We've merged an update to the retry count in readiness for 1.22 so hopefully that'll mitigate things as per the discussion above, but I won't close this as it's just papering over the cracks still, until we can resolve things properly. |
|
@ChrisDenton I think your simulator is faulty, sorry. You're dropping the opened files immediately in your simulated av action loops. So they aren't open for x number of ns, they are open for the time it takes to do the syscall. This means it only supports the idea of being able to zip in if and only if the activity was limited to open+close with no reads and no delays in time between them, neither of which are true. For higher fidelity I think you need |
|
I'm getting bit by this bug as well. Corporate laptop, strict policy with McAfee antivirus. |
|
I am also getting bit by this bug. Since it is corporate laptop I cannot remove Mcafee. Is there a way we can make installation wait longer so that Mcafee completes its check |
|
We hope to release a new rustup soon which will help a little; but there's a bug we need to bottom first which is causing problems mentioned above around |
|
The latest version of rustup (out yesterday) has this change in. |
|
Just tried it again now. The installation seems to work great. Despite that it temporarily hangs up / long loops on for several dozen retries...it does succeed in installation now. Thank you so much, you guys are fantastic! |
|
Wonderful, thank you for checking for us. I'll close this now then. |
|
I thought the change that had been merged was an interim fix. @kinnison - said it was just papering over the cracks. Presumably, on different hardware, the timeout may need to be different - although we could just wait to see if anyone encounters the issue. If the current fix is as considered acceptable, then I wonder whether the repeated logging of I had to run |
|
you're quite right, I was overzealous there. |
|
Addressing #2417 may provide a permanent solution to this as well. |
Per rust-lang#1912 McAfee users are still seeing contention on directory renames. Give McAfee more settle time.
|
@rustbot label: +O-windows |
|
I'm closing this as the symptom is well resolved. I'm going to file a new bug about possible future improvements. |
and others
Problem
A fresh install of stable build fails on Windows 8.1 with message "error: could not rename component file from '... \rust-docs\share/doc/rust/html' ...".
Expected behavior would be the install to be successful :)
Steps
%USERPROFILE%\.rustupand%USERPROFILE%\.cargofolders do not existrustup-init.exefile (1.35.0 of 2019-05-23) and run it under normal (non-admin) user without elevating user rights.Possible Solution(s)
A possible temporary workaround may be to run the installer with admin privileges, as per https://stackoverflow.com/questions/52542965/rust-installation-fails-on-windows-subsystem-for-linux-could-not-rename-compone/55373522#55373522 , that proved to work for me.
Notes
Output of
rustup --version:rustup 1.18.3 (435397f48 2019-05-22)Output of
rustup show:The text was updated successfully, but these errors were encountered: