Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use new context-based API for blosc benchmark to fix hanging #82

Closed
wants to merge 1 commit into from

Conversation

jack-pappas
Copy link
Contributor

I compiled the blosc benchmark program on multiple systems with varying hardware configurations and under both Windows and Linux; then I ran the suite benchmark, specifying various numbers for the thread count (based on the number of cores in a particular system). Each time, the program ran fine for 1 and 2 threads, then hung once it reached the Compression Level 0 step for 3 threads. Strangely, I was able to run the single benchmark just fine for larger numbers of threads (e.g., 4).

Out of curiosity, I modified the bench program so it used the new context-based APIs instead of the older "global" functions and the problem disappeared -- I'm able to run the suite benchmark without issue now. This PR includes the changes I made to the benchmark program to use the new context-based APIs.

@FrancescAlted
Copy link
Member

Yes, I can reproduce that in my Linux box. But as I said before, the '_ctx()' is not the 'new' API, and besides, your PR is breaking the test suite in TravisCI. This should be some artifact introduced after 1.5.0. @crspeller do you have some suggestion on this one?

@FrancescAlted
Copy link
Member

Tested with clang (3.3 and 3.5) and GCC (4.9.1) on Ubuntu 14.10 and all exposes the problem.

@jack-pappas
Copy link
Contributor Author

I spoke too soon, perhaps, when I said these changes fixed the threading problem. After I submitted this PR, I realized I had only tried running the bench program using blosclz. I tried running it with zlib instead and ran into threading issues again. This time, instead of hanging, the benchmark ran into some problem and crashed:

Blosc version: 1.5.3.dev ($Date:: 2014-12-30 #$)
List of supported compressors in this build: blosclz,lz4,lz4hc,snappy,zlib
Supported compression libraries:
  BloscLZ: 1.0.3
  LZ4: 1.5.0
  Snappy: 1.1.1
  Zlib: 1.2.8
Using compressor: zlib
Running suite: suite
--> 1, 2097152, 8, 19, zlib
********************** Run info ******************************
Blosc version: 1.5.3.dev ($Date:: 2014-12-30 #$)
Using synthetic data with 19 significant bits (out of 32)
Dataset size: 2097152 bytes Type size: 8 bytes
Working set: 256.0 MB       Number of threads: 1
********************** Running benchmarks *********************
memcpy(write):        121.9 us, 16410.3 MB/s
memcpy(read):         162.5 us, 12307.7 MB/s
Compression level: 0
comp(write):      162.5 us, 12307.7 MB/s      Final bytes: 2097168  Ratio: 1.00
decomp(read):     121.9 us, 16410.3 MB/s      OK
Compression level: 1
comp(write):     9140.6 us, 218.8 MB/s    Final bytes: 171752  Ratio: 12.21
decomp(read):    2478.1 us, 807.1 MB/s    OK
Compression level: 2
comp(write):     8815.6 us, 226.9 MB/s    Final bytes: 121560  Ratio: 17.25
decomp(read):    2112.5 us, 946.7 MB/s    OK
Compression level: 3
comp(write):     8896.9 us, 224.8 MB/s    Final bytes: 121528  Ratio: 17.26
decomp(read):    2112.5 us, 946.7 MB/s    OK
Compression level: 4
comp(write):     11456.3 us, 174.6 MB/s   Final bytes: 80000  Ratio: 26.21
decomp(read):    2356.3 us, 848.8 MB/s    OK
Compression level: 5
comp(write):     11700.0 us, 170.9 MB/s   Final bytes: 74704  Ratio: 28.07
decomp(read):    2356.3 us, 848.8 MB/s    OK
Compression level: 6
comp(write):     13000.0 us, 153.8 MB/s   Final bytes: 50636  Ratio: 41.42
decomp(read):    2193.8 us, 911.7 MB/s    OK
Compression level: 7
comp(write):     13040.6 us, 153.4 MB/s   Final bytes: 37396  Ratio: 56.08
decomp(read):    2356.3 us, 848.8 MB/s    OK
Compression level: 8
comp(write):     15478.2 us, 129.2 MB/s   Final bytes: 37094  Ratio: 56.54
decomp(read):    2315.6 us, 863.7 MB/s    OK
Compression level: 9
comp(write):     14708.9 us, 136.0 MB/s   Final bytes: 29485  Ratio: 71.13
decomp(read):    2437.5 us, 820.5 MB/s    OK
--> 2, 2097152, 8, 19, zlib
********************** Run info ******************************
Blosc version: 1.5.3.dev ($Date:: 2014-12-30 #$)
Using synthetic data with 19 significant bits (out of 32)
Dataset size: 2097152 bytes Type size: 8 bytes
Working set: 256.0 MB       Number of threads: 2
********************** Running benchmarks *********************
memcpy(write):        121.9 us, 16410.3 MB/s
memcpy(read):         162.5 us, 12307.7 MB/s
Compression level: 0
comp(write):      162.5 us, 12307.5 MB/s      Final bytes: 2097168  Ratio: 1.00
decomp(read):     203.1 us, 9846.2 MB/s   OK
Compression level: 1
comp(write):     3575.0 us, 559.4 MB/s    Final bytes: 171752  Ratio: 12.21
decomp(read):    1462.5 us, 1367.5 MB/s   OK
Compression level: 2
comp(write):     3006.3 us, 665.3 MB/s    Final bytes: 121560  Ratio: 17.25
decomp(read):    1259.4 us, 1588.1 MB/s   OK
Compression level: 3
comp(write):     3168.8 us, 631.2 MB/s    Final bytes: 121528  Ratio: 17.26
decomp(read):    1300.0 us, 1538.5 MB/s   OK
Compression level: 4
comp(write):     5606.3 us, 356.7 MB/s    Final bytes: 80000  Ratio: 26.21
decomp(read):    1381.3 us, 1448.0 MB/s   OK
Compression level: 5
comp(write):     5403.1 us, 370.2 MB/s    Final bytes: 74704  Ratio: 28.07
decomp(read):    1421.9 us, 1406.6 MB/s   OK
Compression level: 6
comp(write):     6256.3 us, 319.7 MB/s    Final bytes: 50636  Ratio: 41.42
decomp(read):    1381.3 us, 1448.0 MB/s   OK
Compression level: 7
comp(write):     6784.4 us, 294.8 MB/s    Final bytes: 37396  Ratio: 56.08
decomp(read):    1543.8 us, 1295.5 MB/s   OK
Compression level: 8
comp(write):     8043.8 us, 248.6 MB/s    Final bytes: 37094  Ratio: 56.54
decomp(read):    1462.5 us, 1367.5 MB/s   OK
Compression level: 9
comp(write):     14665.7 us, 136.4 MB/s   Final bytes: 29485  Ratio: 71.13
decomp(read):    2478.1 us, 807.1 MB/s    OK
--> 3, 2097152, 8, 19, zlib
********************** Run info ******************************
Blosc version: 1.5.3.dev ($Date:: 2014-12-30 #$)
Using synthetic data with 19 significant bits (out of 32)
Dataset size: 2097152 bytes Type size: 8 bytes
Working set: 256.0 MB       Number of threads: 3
********************** Running benchmarks *********************
memcpy(write):        121.9 us, 16410.3 MB/s
memcpy(read):         162.5 us, 12307.7 MB/s
Compression level: 0
comp(write):      203.1 us, 9846.0 MB/s   Final bytes: 2097168  Ratio: 1.00
decomp(read):     203.1 us, 9846.2 MB/s   OK
Compression level: 1
comp(write):     4590.6 us, 435.7 MB/s    Final bytes: 171752  Ratio: 12.21
decomp(read):    1259.4 us, 1588.1 MB/s   OK
Compression level: 2
comp(write):     5565.6 us, 359.3 MB/s    Final bytes: 121560  Ratio: 17.25
decomp(read):    1137.5 us, 1758.2 MB/s   OK
Compression level: 3
comp(write):     5525.0 us, 362.0 MB/s    Final bytes: 121528  Ratio: 17.26
decomp(read):    1096.9 us, 1823.4 MB/s   OK
Compression level: 4
comp(write):     5321.9 us, 375.8 MB/s    Final bytes: 80000  Ratio: 26.21
decomp(read):    1340.6 us, 1491.8 MB/s   OK
Compression level: 5
comp(write):     5484.4 us, 364.7 MB/s    Final bytes: 74704  Ratio: 28.07
decomp(read):    1340.6 us, 1491.8 MB/s   OK
Compression level: 6
comp(write):     7596.9 us, 263.3 MB/s    Final bytes: 50636  Ratio: 41.42
decomp(read):    1665.6 us, 1200.7 MB/s   OK
Compression level: 7
comp(write):     7596.9 us, 263.3 MB/s    Final bytes: 37396  Ratio: 56.08
decomp(read):    1665.6 us, 1200.7 MB/s   OK
Compression level: 8
comp(write):     8287.5 us, 241.3 MB/s    Final bytes: 37094  Ratio: 56.54
Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error allocating memory!Error 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants