Parallel Letter Frequency times out in the web interface

[ryanplusplus]

In my local environment, my solution passes all tests in 1.4 seconds. Of this time, about 70% is spent running the serial solution in test_faster_than_serialized_answer. When this is run on the Exercism infrastructure, my solution times out. For giggles, I tried the example solution from this repo and this also times out. If I make my solution do nothing so that all of the time spent in the Exercism infrastructure is due to the serial solution, it doesn't time out, but obviously it doesn't pass the tests.

It seems that Exercism runners aren't fast enough or don't have enough parallelism to allow a parallel solution to run in the small amount of time left over after accounting for the fixed cost of the serial benchmark solution. Is it actually possible to have a solution succeed when running on the Exercism infrastructure? If not, can we reduce the size of the benchmark input to allow a solution to succeed?

Note that after submitting my failing solution, I saw that no community solutions are listed. Presumably this means that no one has a solution that can run in the time allotted.

[ryanplusplus]

With no change to my solution I'm now able to get it to run without timing out, but my solution fails test_faster_than_serialized_answer despite it succeeding locally. The example solution also fails test_faster_than_serialized_answer despite succeeding locally. Maybe the Exercism runner has only one core?

[kotp]

I have not submitted my solution or at least have not published it. 13 tests pass, one fails, and it is the one concerning benchmark in which we are working on.

[iHiD]

Maybe the Exercism runner has only one core?

I suspect the docker container is only running on one core, yeah.

[PatrickMcSweeny]

I benchmarked this refactor on my local machine and this was the output with the original approach on top, and the new approach on the bottom:

0.000011 0.000003 0.000014 ( 0.000010)
0.000003 0.000001 0.000004 ( 0.000004)

[kotp]

Can you show your benchmark code, please? I suggest using benchmark-ips.

It's reporting is good and you do not have to concern yourselves with the number of iterations, as it is per time rather than iteration.

[PatrickMcSweeny]

require 'benchmark'

time1 = Benchmark.measure do
  texts = Array.new(20, 'a' * 100_000)
end

time2 = Benchmark.measure do
  s = 'a' * 100_00
  texts = Array.new(20, s)
end

puts time1
puts time2

[kotp]

require 'benchmark/ips'

Benchmark.ips do |x|
  x.report("duplicate string") do
    texts = Array.new(20, 'a' * 100_000)
  end

  x.report("shared string") do
    s = 'a' * 100_000
    texts = Array.new(20, s)
  end

  x.compare!
end

__END__
Warming up --------------------------------------
    duplicate string     7.091k i/100ms
       shared string     8.032k i/100ms
Calculating -------------------------------------
    duplicate string     65.369k (±11.9%) i/s   (15.30 μs/i) -    326.186k in   5.076729s
       shared string     54.386k (±11.6%) i/s   (18.39 μs/i) -    273.088k in   5.108371s

Comparison:
    duplicate string:    65368.7 i/s
       shared string:    54386.0 i/s - same-ish: difference falls within error

This is what I got when writing this as a benchmark using ips.

The problem might be too small of a sample to get a good reading, the first time a large string is created, etc. Benchmarking is hard, and I get this result when using Ruby 3.40dev:

require 'benchmark/ips'

Benchmark.ips do |x|
  x.report("duplicate string") do
    texts = Array.new(20, 'a' * 100_000)
  end

  x.report("shared string") do
    s = 'a' * 100_000
    texts = Array.new(20, s)
  end

  x.compare!
end

__END__
Warming up --------------------------------------                                         
    duplicate string     7.063k i/100ms                                                   
       shared string     8.103k i/100ms                                                   
Calculating -------------------------------------                                         
    duplicate string     75.452k (± 8.1%) i/s   (13.25 μs/i) -    381.402k in   5.091754s 
       shared string     65.432k (± 9.1%) i/s   (15.28 μs/i) -    332.223k in   5.124930s 
                                                                                          
Comparison:                                                                               
    duplicate string:    75451.7 i/s                                                      
       shared string:    65432.1 i/s - same-ish: difference falls within error