Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store gateway mmap removal: use a shared pool of buffers across all Decbuf instances #3691

Merged
merged 3 commits into from
Dec 13, 2022
Merged

Store gateway mmap removal: use a shared pool of buffers across all Decbuf instances #3691

merged 3 commits into from
Dec 13, 2022

Commits on Dec 12, 2022

  1. Use a shared pool of buffers across all DecbufFactory instances. 

    This results in a small improvement in performance and heap
consumption:

name                                         old time/op    new time/op    delta
NewStreamBinaryReader/20Names100Values-10       758µs ± 7%     726µs ± 1%    ~     (p=0.111 n=5+4)
NewStreamBinaryReader/20Names500Values-10      2.12ms ± 4%    2.15ms ± 4%    ~     (p=0.548 n=5+5)
NewStreamBinaryReader/20Names1000Values-10     3.49ms ± 2%    3.44ms ± 2%    ~     (p=0.222 n=5+5)
NewStreamBinaryReader/50Names100Values-10      1.54ms ± 4%    1.51ms ± 5%    ~     (p=0.421 n=5+5)
NewStreamBinaryReader/50Names500Values-10      4.33ms ± 1%    4.39ms ± 8%    ~     (p=1.000 n=5+5)
NewStreamBinaryReader/50Names1000Values-10     7.88ms ± 3%    7.84ms ± 3%    ~     (p=0.690 n=5+5)
NewStreamBinaryReader/100Names100Values-10     2.73ms ± 5%    2.57ms ± 5%  -6.15%  (p=0.032 n=5+5)
NewStreamBinaryReader/100Names500Values-10     8.12ms ± 3%    8.02ms ± 2%    ~     (p=0.548 n=5+5)
NewStreamBinaryReader/100Names1000Values-10    15.3ms ± 3%    14.8ms ± 1%  -3.43%  (p=0.016 n=5+5)
NewStreamBinaryReader/200Names100Values-10     4.87ms ± 7%    4.84ms ± 4%    ~     (p=0.841 n=5+5)
NewStreamBinaryReader/200Names500Values-10     15.9ms ± 2%    15.9ms ± 1%    ~     (p=1.000 n=5+5)
NewStreamBinaryReader/200Names1000Values-10    30.0ms ± 2%    30.2ms ± 3%    ~     (p=1.000 n=5+5)

name                                         old alloc/op   new alloc/op   delta
NewStreamBinaryReader/20Names100Values-10      3.36MB ± 0%    3.35MB ± 0%  -0.29%  (p=0.000 n=5+4)
NewStreamBinaryReader/20Names500Values-10      4.00MB ± 0%    3.99MB ± 0%  -0.25%  (p=0.016 n=5+4)
NewStreamBinaryReader/20Names1000Values-10     4.80MB ± 0%    4.79MB ± 0%  -0.21%  (p=0.008 n=5+5)
NewStreamBinaryReader/50Names100Values-10      3.63MB ± 0%    3.62MB ± 0%  -0.27%  (p=0.016 n=4+5)
NewStreamBinaryReader/50Names500Values-10      5.21MB ± 0%    5.20MB ± 0%  -0.19%  (p=0.008 n=5+5)
NewStreamBinaryReader/50Names1000Values-10     7.23MB ± 0%    7.22MB ± 0%  -0.14%  (p=0.008 n=5+5)
NewStreamBinaryReader/100Names100Values-10     4.08MB ± 0%    4.07MB ± 0%  -0.24%  (p=0.008 n=5+5)
NewStreamBinaryReader/100Names500Values-10     7.24MB ± 0%    7.23MB ± 0%  -0.14%  (p=0.008 n=5+5)
NewStreamBinaryReader/100Names1000Values-10    11.3MB ± 0%    11.3MB ± 0%  -0.09%  (p=0.029 n=4+4)
NewStreamBinaryReader/200Names100Values-10     4.97MB ± 0%    4.96MB ± 0%  -0.20%  (p=0.008 n=5+5)
NewStreamBinaryReader/200Names500Values-10     11.6MB ± 0%    11.6MB ± 0%  -0.08%  (p=0.008 n=5+5)
NewStreamBinaryReader/200Names1000Values-10    20.1MB ± 0%    20.1MB ± 0%  -0.05%  (p=0.016 n=4+5)

name                                         old allocs/op  new allocs/op  delta
NewStreamBinaryReader/20Names100Values-10       6.23k ± 0%     6.22k ± 0%  -0.11%  (p=0.029 n=4+4)
NewStreamBinaryReader/20Names500Values-10       30.3k ± 0%     30.3k ± 0%  -0.02%  (p=0.029 n=4+4)
NewStreamBinaryReader/20Names1000Values-10      60.3k ± 0%     60.3k ± 0%  -0.01%  (p=0.029 n=4+4)
NewStreamBinaryReader/50Names100Values-10       15.5k ± 0%     15.5k ± 0%  -0.05%  (p=0.029 n=4+4)
NewStreamBinaryReader/50Names500Values-10       75.6k ± 0%     75.6k ± 0%    ~     (p=0.079 n=4+5)
NewStreamBinaryReader/50Names1000Values-10       151k ± 0%      151k ± 0%  -0.00%  (p=0.029 n=4+4)
NewStreamBinaryReader/100Names100Values-10      31.0k ± 0%     31.0k ± 0%  -0.02%  (p=0.000 n=4+5)
NewStreamBinaryReader/100Names500Values-10       151k ± 0%      151k ± 0%  -0.00%  (p=0.000 n=5+4)
NewStreamBinaryReader/100Names1000Values-10      301k ± 0%      301k ± 0%  -0.00%  (p=0.016 n=4+5)
NewStreamBinaryReader/200Names100Values-10      61.9k ± 0%     61.9k ± 0%  -0.01%  (p=0.008 n=5+5)
NewStreamBinaryReader/200Names500Values-10       302k ± 0%      302k ± 0%  -0.00%  (p=0.008 n=5+5)
NewStreamBinaryReader/200Names1000Values-10      602k ± 0%      602k ± 0%  -0.00%  (p=0.016 n=5+4)
    @charleskorn
    charleskorn committed Dec 12, 2022
    Configuration menu
    Copy the full SHA
    af30987 View commit details
    Browse the repository at this point in the history

  2. Move buffer pool management into FileReader.

    @charleskorn
    charleskorn committed Dec 12, 2022
    Configuration menu
    Copy the full SHA
    9f7d5c9 View commit details
    Browse the repository at this point in the history

  3. Add note about how pooled buffers are cleaned up.

    @charleskorn
    charleskorn committed Dec 12, 2022
    Configuration menu
    Copy the full SHA
    9e85f11 View commit details
    Browse the repository at this point in the history