Can't build on Windows 10

[Kaduo]

When trying to build the crate with cargo, I get the following error messsage :

error: failed to run custom build command for `openblas-src v0.7.0`
process didn't exit successfully: `C:\Users\(...)\target\debug\build\openblas-src-ea19b54cda88a9a7\build-script-build` (exit code: 101)
--- stdout
Running: `"cp" "-R" "source" "source_x86_64-pc-windows-msvc_tmp"`

--- stderr
thread 'main' panicked at 'Failed: `"cp" "-R" "source" "source_x86_64-pc-windows-msvc_tmp"` (The file can't be found. (os error 2))', C:\Users\(...)\.cargo\registry\src\github.com-1ecc6299db9ec823\openblas-src-0.7.0\build.rs:72:13
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

I am using the latest stable version of rust (1.34.0) on Windows 10.

[Kaduo]

After some toying around, the issue seems to be that the windows shell doesn't have a "cp" command, or a "make" command for that matter, so the shell thinks that these are local executables, hence the "unknown file" error. I'm not sure what I can do to circumvent this issue...

[IvanUkhov]

Yes, I suspected that. Unfortunately, the build script was not written with Windows in mind. Some help from an experienced Windows user is required in order to make appropriate adjustments.

[thecooldrop]

Actually both make and cp commands are functional in windows powershell if you have gnu make and cygwin installed and on path. Though the commands are functional on my shell, I too get the same error but mine is formatted bit differently to "source_x86_64-pc-windows-msvc-tmp". I hope this can be resolved as I am unable to use linux on my system.

[IvanUkhov]

For those interested in researching into the root cause, I’ve set up AppVeyor. Here one can see how it fails for four different configurations with two different errors:

https://ci.appveyor.com/project/IvanUkhov/openblas-src

[thecooldrop]

Changing make to cmake and cp -R command to robocopy source source_x86_64-pc-windows-gnu_tmp /e I get following result for Environment: TARGET=x86_64-pc-windows-msvc https://ci.appveyor.com/project/VanioBegic/openblas-src/builds/24976367/job/2adrcsyx4vaj4yt3 .

The exit code is 1 in this case, which seems to be the problem, since usually anything but 0 is interpreted as error, and I do not know if this is also a case with Rust Command

Update:
This is the exact problem I expected. The status 0 is success and everything else is a failure. This is clearly outlined in Rust documentation. By using the following form of the run function it passes the folder copying stage, but does not manage to find the gfortran.lib

fn run(command: &mut Command) { println!("Running: {:?}", command); match command.status() { Ok(status) => { let mut status_code = false; match status.code(){ Some(1) =>{ status_code = true;} Some(0) =>{ status_code = true;} _ =>{status_code = false;} } if !status_code { panic!("Failed: {:?}({})", command, status); } } Err(error) => { panic!("Failed:{:?} ({})", command, error); } } }

and this compiles to following error https://ci.appveyor.com/project/VanioBegic/openblas-src/builds/24976676/job/6vp5g54ha050w4cc

I am sorry for formatting, I am really new to all of this.

[IvanUkhov]

Thank you for looking into this! Yes, the actual problem is the following:

thread 'main' panicked at 'Failed: `"robocopy" "source" "source_x86_64-pc-windows-msvc_tmp" "/e"` (exit code: 1)', build.rs:68:17

In other words, this rotocopy command returned a nonzero status, and hence everything failed. By changing the run function the way you did, you let this error be ignored.

Since you got to the linking part, does it mean that you were able to compile OpenBLAS? The whole process usually takes around 10 minutes. If it finished in a few seconds, it probably didn’t go well.

In any case, I think the best way to make it work is to compile OpenBLAS in isolation first. That is, just clone their repository and try to go through the whole process from start to finish manually. Then it’ll be a matter of reproducing the same commands in the build script, assuming that all requirements are installed. Here are some instructions:

https://github.com/xianyi/OpenBLAS/wiki/How-to-use-OpenBLAS-in-Microsoft-Visual-Studio

Perhaps linking to any gfortran is not even needed.

[thecooldrop]

As said, this is actually not an error. Robocopy returns exit code 1, when it successfully copies all files. But directly afterwards it returns with note: LINK : fatal error LNK1181: cannot open input file 'gfortran.lib' during compilation of openblas-src

I will try to follow the tutorial given in the link above, and will come back with results.

[IvanUkhov]

Yes, you are right:

https://blogs.technet.microsoft.com/deploymentguys/2008/06/16/robocopy-exit-codes/

[thecooldrop]

Now when I clone the repository and build OpenBLAS manually, how do I link it to rust? Root cause of my problem is actually that I am trying to use ndarray-linalg library, but none of the BLAS crates compiles, and this one seemed most promising. Any tutorials would be helpful.

[IvanUkhov]

So now you’re able to compile it manually, right? Then the natural next step is to integrate it into this crate and make the test here pass. In theory, the only thing that is needed is to link everything properly. You have the library compiled, and now tell Rust to use it in the build script.

1. Change cargo:rustc-link-search to point to the folder with the library.
2. Remove redundant libraries.
3. Tell it the name of the library to take.

[thecooldrop]

So I made some progress on my machine here are the steps I took:

1. From this guide install MSYS2 for your system
2. ALL THE FOLLOWING STEPS PROCEED FROM MSYS SHELL
3. Open MSYS shell
4. Run pacman -S gcc pacman -S make pacman -S gcc-fortran pacman -S git
5. Run git clone https://github.com/xianyi/OpenBLAS
6. Run cd OpenBLAS
7. Run make
8. Run make install

Now after these steps the .a files can be usually found under C:\msys64\opt\OpenBLAS\lib if the installation of MSYS proceeded as given in link. Now if one wants to use this build of OpenBLAS for openblas-src it is necesarry to remove all calls to make from build script, and make corrections for robocopy on windows systems as discussed in previous comments.

Now it is the matter of translating these steps into build-script.

[IvanUkhov]

Perfect! Have you tried to hard-code your manual installation in the build script to check if it is actually able to find and use what is needed? I also assume that all the different parameters that we pass to make in the build script (BINARY, {YES,NO}_CBLAS, {YES,NO}_LAPACKE, and DESTDIR) continue to work properly. Plus the libs, netlib, and shared targets should still exist. I suppose there should be no major difference; it’s the same Makefile that is used.

Assuming the above is the case, I’d proceed as follows. All the prerequisites should be assumed to be installed. We shouldn’t try to do anything in this regard in the build script. It’s better to write it in the Wiki and then make sure people can find it from every package. Then the only thing that has to be changed is the copying of the source directory to a target-specific folder. According to your findings, we have to have two versions of this operation: one for #[cfg(unix)] and one for #[cfg(windows)]. Then one has to extract it into a separate function. Alternatively, one has to write this bit in pure Rust.

What do you think?

[thecooldrop]

Comically enought now cargo build runs to end, but cargo test fails because it cant find a gfortran.lib file, which is not even present on my computer. Even when I comment out the specific println it still fails because it can not find openblas.lib file, which is sadly enough also not present on my computer. I am trying to resolve the issue by trying gnu toolchain.

[IvanUkhov]

Do you not get gfortran.lib installed when you run pacman -S gcc-fortran? What does make compile if not openblas.lib? I’d expect this library to be found after make libs netlib shared.

[thecooldrop]

Actually I do not get gfortran.lib file after running pacman -S gcc-fortran on msys2. What I do get is libgfortran.a and libgfortran.dll.a files located at C:\msys64\usr\lib\gcc\x86_64-pc-msys\7.4.0

About openblas.lib I can not confirm nor deny it, since I did not run make libs netlib shared as used in build script but rather make as described on the OpenBLAS page which you linked to earlier.

Update:
Right now I recompiled the OpenBLAS with make libs netlib shared and got following output after long compilation text :

ranlib ../../../cygopenblas_haswellp-r0.3.7.dev.a make[3]: Leaving directory '/d/OpenBLAS/lapack-netlib/LAPACKE/utils' make[2]: Leaving directory '/d/OpenBLAS/lapack-netlib/LAPACKE' make[1]: Leaving directory '/d/OpenBLAS/lapack-netlib' make[1]: warning: -jN forced in submake: disabling jobserver mode. make[1]: Entering directory '/d/OpenBLAS/exports' perl ./gensymbol win2k x86_64 dummy 0 0 0 0 0 0 "" "" 1 > cygopenblas.def cc -O2 -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=2 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.7.dev\" -mavx2 -DASMNAME=dllinit -DASMFNAME=dllinit_ -DNAME=dllinit_ -DCNAME=dllinit -DCHAR_NAME=\"dllinit_\" -DCHAR_CNAME=\"dllinit\" -DNO_AFFINITY -I.. -c -o dllinit.o -s dllinit.c ranlib ../cygopenblas_haswellp-r0.3.7.dev.a cc -O2 -DMAX_STACK_ALLOC=2048 -Wall -m64 -DF_INTERFACE_GFORT -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=2 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.7.dev\" -mavx2 -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DCHAR_NAME=\"_\" -DCHAR_CNAME=\"\" -DNO_AFFINITY -I.. cygopenblas.def dllinit.o \ -shared -o ../cygopenblas.dll -Wl,--out-implib,../libopenblas.dll.a \ -Wl,--whole-archive ../cygopenblas_haswellp-r0.3.7.dev.a -Wl,--no-whole-archive -L/usr/lib/gcc/x86_64-pc-msys/7.4.0 -L/usr/lib/gcc/x86_64-pc-msys/7.4.0/../../../../x86_64-pc-msys/lib/../lib -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-pc-msys/7.4.0/../../../../x86_64-pc-msys/lib -L/usr/lib/w32api -lgfortran -lquadmath -lm -lmsys-2.0 -lgfortran -lgfortran make[1]: Leaving directory '/d/OpenBLAS/exports'

When I run make install I still get no openblas.lib but rather I get cygopenblas.a and cygopenblas.dll and libopenblas.dll.a files in C:\msys64\opt\OpenBLAS\bin.

In most recent attempt I am trying to compile this repo directly from msys.

[IvanUkhov]

It’s good news that you have libgfortran.a, libgfortran.dll.a, cygopenblas.a, cygopenblas.dll, and libopenblas.dll.a. Those that end with .a are for static linking (see the static Cargo feature in the build script). The one that ends with .dll is for dynamic. The question is, what those ending with *.dll.a are for? Maybe they are the one needed for dynamic linking but named differently for some reason. Here we need a Windows expert. Perhaps rustc can pick it up automatically? Not sure where the .lib convention comes from, and how to influence it.

I’d simplify the build script by hard-coding the path to your manual installation and then play with different linking options.

https://doc.rust-lang.org/cargo/reference/build-scripts.html

[thecooldrop]

It compiles with original build script from MSYS2 shell . I guess we can close the issue, and write in readme or wiki that compilation on windows must be done through MSYS2 shell rather then powershell.

[IvanUkhov]

How come nothing is passing on AppVeyor then?

https://ci.appveyor.com/project/IvanUkhov/openblas-src

Is it a different shell used? Do you know how the configuration file has to be adjusted to make it work?

https://github.com/blas-lapack-rs/openblas-src/blob/master/appveyor.yml

[thecooldrop]

The shell is different. Basically one needs to start this special shell which is installed together with MSYS2 called MSYS2 MinGW 64-bit ( in case of 64 bit enviroment, and 32 bit in case of 32 bit enviroment ) , and then run all the arguments from there. Seems like the way to make that happen in Rust is outlined in following : https://stackoverflow.com/questions/29903129/invoking-shell-scripts-on-windows-under-msys-using-command

[IvanUkhov]

@vaniobegic, I’ve tried doing it from sh, which is shipped with MSYS if I’m not mistaken, but it didn’t seem to help. One has to make rustc be able to find gfortran. How did you do this?

[thecooldrop]

@IvanUkhov As long as you keep getting errors, it is an indicator that a package is not installed in MSYS enviroment. The -lgfortran error is caused because you lack mingw-w64-x86_64-gcc-libgfortran and you can install it by running pacman -S mingw-w64-x86_64-gcc-libgfortran in case that you are using MinGW enviroment, and if that does not run, then run pacman -Ss gfortran to find a lib that you can install. Once installed the -lgfortran error goes away.

UPDATE:

Cautionary note with MSYS is that package repositories sometimes fail in middle of download, so that even if you run the pacman command it may return prematurely. Make sure you run it as many times as needed until the package is installed.

[IvanUkhov]

I close this due to the recent development.