solana-test-validator does not work properly on Windows

[jacobcreech]

Problem

Currently the Solana CLI will install properly on Windows, but the solana-test-validator will run into a few errors. Generally the recommendation has been for users to install WSL to run solana-test-validator, however this is frictional on the developer experience and leads to a lot of developers dropping off.

Current error flow on Windows:
Running solana-test-validator runs into the following:

Error: Failed to start validator. Failed to create ledger at test-ledger: blockstore error

Inspecting the logs:

tar stderr: tar: Can't launch external program: bzip2

After figuring out bzip2 installation on windows, solana-test-validator hangs on initialization process.

Logs:

[2023-04-03T16:28:38.837257000Z INFO  solana_test_validator] Starting validator with: ArgsOs {
        inner: [
            "solana-test-validator",
        ],
    }
[2023-04-03T16:28:38.837284100Z WARN  solana_perf] CUDA is disabled
[2023-04-03T16:28:38.837302300Z INFO  solana_perf] AVX detected
[2023-04-03T16:28:38.837312500Z INFO  solana_perf] AVX2 detected
[2023-04-03T16:28:38.840948300Z INFO  solana_faucet::faucet] Faucet started. Listening on: 0.0.0.0:9900
[2023-04-03T16:28:38.841038000Z INFO  solana_faucet::faucet] Faucet account address: 45U6mutB7zMjjX8NAXfrUYwNpTdYJm51aWnmjnPsBRC3
[2023-04-03T16:28:38.859996300Z INFO  solana_ledger::blockstore] Opening database at "test-ledger\\rocksdb"
[2023-04-03T16:28:39.066944300Z INFO  solana_ledger::blockstore] "test-ledger\\rocksdb" open took 206ms
[2023-04-03T16:28:39.069010700Z INFO  solana_metrics::metrics] metrics disabled: SOLANA_METRICS_CONFIG: environment variable not found
[2023-04-03T16:28:39.069693100Z INFO  solana_metrics::metrics] datapoint: shred_insert_is_full total_time_ms=0i slot=0i last_index=3i num_repaired=0i num_recovered=0i

Another issue related to running solana-test-validator on Windows: solana-labs#34793

Proposed Solution

Make solana-test-validator work on Windows without additional installation of a bash shell or WSL.

[steviez]

Do you have access to a Windows machine? I personally do not; might be able to find one but if you could run the following and share the logs, the additional verbosity might give more insight into where it is getting snagged:

RUST_LOG=trace solana-test-validator

Attaching to the process with WinDbg and viewing thread stacks would be more involved, but probably much more insightful / much quicker path to honing in. I haven't done Windows debugging in years but I don't remember the tool being too bad to work with: https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/

[Nagaprasadvr]

@jacobcreech @steviez i have added Proof of Work here let me know

[Nagaprasadvr]

Pre-requisite dlls and binaries

Binaries -

1. Protoc - https://github.com/protocolbuffers/protobuf/releases/download/v25.3/protoc-25.3-win64.zip
2. OpenSSL-Winx64 - https://slproweb.com/download/Win64OpenSSL-3_2_1.exe

Dlls -

1. libssl - https://www.dllme.com/dll/files/libssl-3-x64
2. libcrypto - https://www.dllme.com/dll/files/libcrypto-3-x64
3. libclang - https://www.dllme.com/dll/files/libclang/
4. clang - https://www.dllme.com/dll/files/clang

After downloading

1. Binaries , I executed them and placed them in C:\Program FIles\ dir
2. Dlls - i placed them in C:\Windows\System32\ dir

System Environment variables

OPENSSL_INCLUDE_DIR=C:\Program Files\OpenSSL-Win64\include
OPENSSL_LIB=C:\Program Files\OpenSSL-Win64\lib
OPENSSL_LIB_DIR=C:\Program Files\OpenSSL-Win64\lib\VC\x64\MT
PROTOC=C:\Program Files\protoc\bin\protoc
LIBCLANG_PATH=C:\Windows\System32\libclang.dll
PATH=C:\Program Files\OpenSSL-Win64\bin

[Nagaprasadvr]

[Nagaprasadvr]

[Nagaprasadvr]

i did the setup

here is the ping status

Logs :

[joncinque]

I've got fixes for some part of the windows issues at #73 to remove the reliance on additional DLLs that aren't part of a fresh windows installation. This doesn't address the tar issue though.

[Nagaprasadvr]

can this fix protoc , libclang , clang deps?

[joncinque]

can this fix protoc , libclang , clang deps?

Those are required for building, but not running

[Nagaprasadvr]

Cool

[steviez]

tar stderr: tar: Can't launch external program: bzip2

This aspect should be resolved as of #3079 since we now use the tar crate instead of whatever tar we can find in the environment.

With Jon's previous change, the number of remaining issues here should be reduced, possibly empty. I'll try to spin up a Windows VM at some point soon to check and see