slow performance in MS Windows

[giantmoa]

Hi There,
The following example from the package documentation runs very slow in Windows.

julia> using InMemoryDatasets

julia> ds = Dataset(rand([1.1,2.2,3.4], 100, 100000), :auto);

julia> filewriter("_tmp.csv", ds, buffsize = 2^25, lsize = 500000);

julia> @time ds = filereader("_tmp.csv", buffsize = 2^21, lsize = 2^20, types = fill(Float64, 10^5));
  1.163346 seconds (900.02 k allocations: 180.966 MiB)

julia> @time ds = filereader("_tmp.csv", buffsize = 2^21, lsize = 2^20, guessingrows = 2);
  1.803125 seconds (4.10 M allocations: 289.193 MiB, 2.86% gc time)

It is because of parsing float. It seems that parsing floats in base is very slow in Windows

parse(Float64, "32423") is about 20 times slower than parse(Int, "32423") in Windows (in OSX they are almost the same).

[giantmoa]

more information:

              _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.3 (2022-05-06)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo()
Julia Version 1.7.3
Commit 742b9abb4d (2022-05-06 12:58 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, tigerlake)

julia> using DLMReader

julia> using InMemoryDatasets

julia> ds = Dataset(rand([1.1,2.2,3.4], 100, 100000), :auto);

julia> filewriter("_tmp.csv", ds, buffsize = 2^25, lsize = 500000);

julia>  @time ds = filereader("_tmp.csv", buffsize = 2^21, lsize = 2^20, types = fill(Float64, 10^5));
 41.713920 seconds (9.21 M allocations: 604.244 MiB, 0.23% gc time, 15.31% compilation time)

julia>  @time ds = filereader("_tmp.csv", buffsize = 2^21, lsize = 2^20, types = fill(Float64, 10^5));
 37.499938 seconds (899.67 k allocations: 172.735 MiB)

julia>  @time ds = filereader("_tmp.csv", buffsize = 2^21, lsize = 2^20, types = fill(Float64, 10^5), threads=false);
  5.860094 seconds (899.40 k allocations: 166.560 MiB, 2.01% gc time)

julia> ds = Dataset(rand([11,22,34], 100, 100000), :auto);

julia> filewriter("_tmp2.csv", ds, buffsize = 2^25, lsize = 500000);

julia>  @time ds = filereader("_tmp2.csv", buffsize = 2^21, lsize = 2^20, types = fill(Int64, 10^5));
  1.356547 seconds (913.32 k allocations: 173.512 MiB, 7.86% gc time, 1.27% compilation time)

julia>  @time ds = filereader("_tmp2.csv", buffsize = 2^21, lsize = 2^20, types = fill(Int64, 10^5));
  1.219528 seconds (899.68 k allocations: 172.736 MiB, 3.07% gc time)

julia>  @time ds = filereader("_tmp2.csv", buffsize = 2^21, lsize = 2^20, types = fill(Int64, 10^5), threads=false);
  1.717711 seconds (899.40 k allocations: 166.560 MiB, 1.70% gc time)

julia> Threads.nthreads()
4

julia>

[sl-solution]

Thanks for reporting, would you please report the output for the following code?

julia> using DLMReader
julia> using BenchmarkTools
julia> @btime InlineStrings.Parsers.xparse(Float64, "34334", 1, 5)
julia> @btime InlineStrings.Parsers.xparse(Float64, "343.34", 1, 6)
julia> @btime parse(Float64, "34334")
julia> @btime parse(Float64, "343.34")

[giantmoa]

julia>  using BenchmarkTools

julia> @btime InlineStrings.Parsers.xparse(Float64, "34334", 1, 5)
  16.433 ns (0 allocations: 0 bytes)
Parsers.Result{Float64}(33, 5, 34334.0)

julia> @btime InlineStrings.Parsers.xparse(Float64, "343.34", 1, 6)
  18.337 ns (0 allocations: 0 bytes)
Parsers.Result{Float64}(33, 6, 343.34)

julia> @btime parse(Float64, "34334")
  141.170 ns (0 allocations: 0 bytes)
34334.0

julia> @btime parse(Float64, "343.34")
  393.069 ns (0 allocations: 0 bytes)
343.34

[sl-solution]

This is strange! the parse function in Windows is very slow, however, the stranger thing is that using threads = true worsen it. Would you mind to report the output of the following code;

julia> x = rand(["34234.343", "4345.34", "3534", "-34.43", "0.03"], 10^6)
julia> function parse_float(x)
           res = Vector{Union{Missing, Float64}}(undef, length(x))
           for i in 1:length(x)
               res[i] = parse(Float64, x[i])
           end
           res
       end
julia> function parse_float_parallel(x)
           res = Vector{Union{Missing, Float64}}(undef, length(x))
           Threads.@threads for i in 1:length(x)
               res[i] = parse(Float64, x[i])
           end
           res
       end
julia> using BenchmarkTools
julia> @btime parse_float_parallel(x)
julia> @btime parse_float(x)

[giantmoa]

julia>  @btime parse_float_parallel(x)
  2.951 s (23 allocations: 8.59 MiB)
julia> @btime parse_float(x)
  370.881 ms (2 allocations: 8.58 MiB)
julia> parse_float_parallel(x) == parse_float(x)
true

[sl-solution]

We now use Parsers for floats, it should fix this issue.