rust: Allow to specify a separate download folder for rust packages #22457
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
By default, the rust downloads are stored in the rustc/cargo subdirectory of the openwrt default download folder. However, this can cause problems if the download directory is not local but is an nfs mount. Fix this by adding a new configuration option to openwrt to change the default download folder for rust packages. Signed-off-by: Florian Eckert <fe@dev.tdt.de>
|
This looks like an issue caused by 0002-rustc-bootstrap-cache.patch - can you try replacing that patch with this: --- a/src/bootstrap/bootstrap.py
+++ b/src/bootstrap/bootstrap.py
@@ -546,7 +546,7 @@ class RustBuild(object):
shutil.rmtree(bin_root)
key = self.stage0_compiler.date
- cache_dst = os.path.join(self.build_dir, "cache")
+ cache_dst = os.getenv('OPENWRT_RUSTC_BOOTSTRAP_CACHE', os.path.join(self.build_dir, "cache"))
rustc_cache = os.path.join(cache_dst, key)
if not os.path.exists(rustc_cache):
os.makedirs(rustc_cache)
--- a/src/bootstrap/download.rs
+++ b/src/bootstrap/download.rs
@@ -202,7 +202,13 @@ impl Config {
Some(other) => panic!("unsupported protocol {other} in {url}"),
None => panic!("no protocol in {url}"),
}
- t!(std::fs::rename(&tempfile, dest_path));
+ match std::fs::rename(&tempfile, dest_path) {
+ Ok(v) => v,
+ Err(_) => {
+ t!(std::fs::copy(&tempfile, dest_path));
+ t!(std::fs::remove_file(&tempfile));
+ }
+ }
}
fn download_http_with_retries(&self, tempfile: &Path, url: &str, help_on_error: &str) {
@@ -520,7 +526,10 @@ impl Config {
key: &str,
destination: &str,
) {
- let cache_dst = self.out.join("cache");
+ let cache_dst = match env::var_os("OPENWRT_RUSTC_BOOTSTRAP_CACHE") {
+ Some(v) => PathBuf::from(v),
+ None => self.out.join("cache"),
+ };
let cache_dir = cache_dst.join(key);
if !cache_dir.exists() {
t!(fs::create_dir_all(&cache_dir));
@@ -647,7 +656,10 @@ download-rustc = false
let llvm_assertions = self.llvm_assertions;
let cache_prefix = format!("llvm-{llvm_sha}-{llvm_assertions}");
- let cache_dst = self.out.join("cache");
+ let cache_dst = match env::var_os("OPENWRT_RUSTC_BOOTSTRAP_CACHE") {
+ Some(v) => PathBuf::from(v),
+ None => self.out.join("cache"),
+ };
let rustc_cache = cache_dst.join(cache_prefix);
if !rustc_cache.exists() {
t!(fs::create_dir_all(&rustc_cache)); |
|
I have tested this patch with |
|
@jefferyto thanks for your reply and your suggested change. If I remove the Another problem has arised with a rust package that was not presented before. So I think this is related to your suggested change. |
If rust/host was already built then nothing should happen when you try to build it a second time. Are
This isn't related to my suggested changed. (The patch is for caching downloads during rust/host build; this error is during build for a package that uses rust/host.) This might be related to having |
jefferyto
added a commit
to jefferyto/openwrt-packages
that referenced
this pull request
Oct 26, 2023
The rust bootstrap downloads files into a "tmp" directory then moves the files into the "cache" directory using std::fs::rename. There are no issues in the original/unpatched case as "tmp" and "cache" are subdirectories in the build directory ($(HOST_BUILD_DIR)/build) and so are nearly guaranteed to be on the same filesystem. 35768bf changed where files are saved/cached (in $(DL_DIR)/rustc). If HOST_BUILD_DIR and DL_DIR are on separate filesystems, then using std::fs::rename to move the files will fail.[1] This updates 0002-rustc-bootstrap-cache.patch to account for this case, i.e. if std::fs::rename fails, fall back to copying the file then removing the original. [1]: openwrt#22457 Fixes: 35768bf ("rust: Cache bootstrap downloads to $(DL_DIR)/rustc") Signed-off-by: Jeffery To <jeffery.to@gmail.com>
|
I am still testing. I will report in the other pullrequest whether it fixes my problem. I am therefore closing the pullrequest. Thanks for your feedback |
neheb
pushed a commit
that referenced
this pull request
Oct 30, 2023
The rust bootstrap downloads files into a "tmp" directory then moves the files into the "cache" directory using std::fs::rename. There are no issues in the original/unpatched case as "tmp" and "cache" are subdirectories in the build directory ($(HOST_BUILD_DIR)/build) and so are nearly guaranteed to be on the same filesystem. 35768bf changed where files are saved/cached (in $(DL_DIR)/rustc). If HOST_BUILD_DIR and DL_DIR are on separate filesystems, then using std::fs::rename to move the files will fail.[1] This updates 0002-rustc-bootstrap-cache.patch to account for this case, i.e. if std::fs::rename fails, fall back to copying the file then removing the original. [1]: #22457 Fixes: 35768bf ("rust: Cache bootstrap downloads to $(DL_DIR)/rustc") Signed-off-by: Jeffery To <jeffery.to@gmail.com>
jefferyto
added a commit
to jefferyto/openwrt-packages
that referenced
this pull request
Oct 31, 2023
The rust bootstrap downloads files into a "tmp" directory then moves the files into the "cache" directory using std::fs::rename. There are no issues in the original/unpatched case as "tmp" and "cache" are subdirectories in the build directory ($(HOST_BUILD_DIR)/build) and so are nearly guaranteed to be on the same filesystem. 35768bf changed where files are saved/cached (in $(DL_DIR)/rustc). If HOST_BUILD_DIR and DL_DIR are on separate filesystems, then using std::fs::rename to move the files will fail.[1] This updates 0002-rustc-bootstrap-cache.patch to account for this case, i.e. if std::fs::rename fails, fall back to copying the file then removing the original. [1]: openwrt#22457 Fixes: 35768bf ("rust: Cache bootstrap downloads to $(DL_DIR)/rustc") Signed-off-by: Jeffery To <jeffery.to@gmail.com> (cherry picked from commit f9f1e02)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Maintainer: @jefferyto
Compile tested: x86_64, -- , OpenWrt master
Run tested: not needed only build changes
Description:
By default, the rust downloads are stored in the rustc/cargo subdirectory of the openwrt default download folder. However, this can cause problems if the download directory is not local but is an nfs mount.
Fix this by adding a new configuration option to openwrt to change the default download folder for rust packages.
My research has shown that the problem is the nfs mount.
Build error if dl ist on a ntfs mount: