Huge virtual memory usage on Windows

[kemege]

I have two machines, both running windows and Julia 1.10.3, and only installed two packages: WriteVTK and ArgParse.
After starting up, the physical memory used by Julia on two machines are similar (as seen in the table below), but the amount of virtual memory differs by more than 30 times. I've tried launching the REPL and using julia -e "sleep(3)", but the results are almost the same.
The memory usages are acquired through taskmgr and Process Explorer.

Machine	OS	Processor	Total physical memory	Physical memory used by Julia	Virtual memory used by Julia
Surface Pro 8	Windows 11 22H2	Intel Core i7-1185G7	16GB	192MB	581MB
Workstation	Windows Server 2022 21H2	AMD EPYC 9554	256GB	195MB	16870MB

While I understand that the memory usage can depend on a lot of factors, which can be quite different especially on different machines / processor brands, starting up with virtual memory of more than 16GiB definitely seems abnormal. Is this kind of platform-specific bug, or because of some misconfiguration in my workstation?

By the way, on both machines, Julia is installed through juliaup, and the versioninfo() from the Julia on two machines are:

# On my surface pro 8
Julia Version 1.10.3
Commit 0b4590a550 (2024-04-30 10:59 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, tigerlake)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)

# On my workstation
Julia Version 1.10.3                                                                                                    
Commit 0b4590a550 (2024-04-30 10:59 UTC)                                                                               
Build Info:                                                                                                               
  Official https://julialang.org/ release                                                                               
Platform Info:                                                                                                            
  OS: Windows (x86_64-w64-mingw32)                                                                                        
  CPU: 64 × AMD EPYC 9554 64-Core Processor                                                                              
  WORD_SIZE: 64                                                                                                           
  LIBM: libopenlibm                                                                                                       
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)                                                                                 
Threads: 1 default, 0 interactive, 1 GC (on 256 virtual cores)

[inkydragon]

"Virtual memory used by Julia"

Where did this data come from?
Does this refer to the "Virtual Size" in Process Explorer?

I have 32GB of physical memory.
Windows seems to allocate 2~3TB of virtual memory space for Edge.
Maybe Edge has a much bigger problem than julia?

Maybe related: Unreasonably huge process virtual memory size reported by Process Explorer

[oscardssmith]

In general, virtual memory isn't really meaningful (at least on 64 bit systems). You have roughly infinite address space, available, so as long as you don't bring the memory into resident memory, it's all fairly imaginary.

[kemege]

"Virtual memory used by Julia"

Where did this data come from? Does this refer to the "Virtual Size" in Process Explorer?

I have 32GB of physical memory. Windows seems to allocate 2~3TB of virtual memory space for Edge. Maybe Edge has a much bigger problem than julia?

Maybe related: Unreasonably huge process virtual memory size reported by Process Explorer

No, it's actually the Private Bytes in Process Explorer, which is about 300MB in the screenshot of Edge. In addition, my workstation was initially configured to have total virtual memory of about 32GB, running a few instances of Julia in parallel will result in errors like "Cannot load xxx.dll due to insufficient virtual memory" (I cannot remember the exact message, but it's something like that). After raising the maximum limit of virtual memory to 1TiB, now I can open about 64 instances in parallel without actually doing any calculation.
So I think in my case, Julia seems to use too much virtual memory as soon as it starts up, and uses up the virtual memory at a very fast rate.

In general, virtual memory isn't really meaningful (at least on 64 bit systems). You have roughly infinite address space, available, so as long as you don't bring the memory into resident memory, it's all fairly imaginary.

Within a specific amount (like tens or hundreds of GB), I do agree that virtual memory is not a problem. However, for some unknown reason, on my workstation, the amount of virtual memory is extremely large and prevents other programs (including other instances of Julia) from running.

By the way, is there any way to know how the virtual memory is being used, so that I might be able to compare the usage on both systems, and see what might be wrong?

[giordano]

What if you set the environment variable OPENBLAS_NUM_THREADS=1 before starting julia? Does that change anything?

[kemege]

What if you set the environment variable OPENBLAS_NUM_THREADS=1 before starting julia? Does that change anything?

It works! The amount of used virtual memory has decreased to about 180MB.

After trying some different values of OPENBLAS_NUM_THREADS, it seems that even setting it to 256 (number of logical cores in the Workstation) cannot replicate the usage of 17GB virtual memory, but I think that might be another issue for openBLAS, instead of Julia.

OPENBLAS_NUM_THREADS	Claimed virtual memory (~MB)
1	186
2	318
4	581
8	1106
16	2157
32	4259
64	8462
128	8462
256	8462
No value	18670

Anyway, the problem has been solved, thanks to all you people for helping me with this!