Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changing the number of threads during appends blocks. #31

Closed
albertosm27 opened this issue Nov 23, 2017 · 0 comments
Closed

Changing the number of threads during appends blocks. #31

albertosm27 opened this issue Nov 23, 2017 · 0 comments

Comments

@albertosm27
Copy link
Member

The program below shows the problem. When it starts with a number of threads larger than 1 it can not go back to a another number of threads larger than 1. However it can cycle correctly between the original number of threads and 1, and back.

`/*
Copyright (C) 2015 Francesc Alted
http://blosc.org
License: BSD (see LICENSE.txt)

Example program demonstrating use of the delta filter from C code.

To compile this program:

$ gcc -O delta_schunk_ex.c -o delta_schunk_ex -lblosc

To run:

$ ./delta_schunk_ex
Blosc version info: 2.0.0a4.dev ($Date:: 2016-08-04 #$)
Compression ratio: 762.9 MB -> 7.6 MB (100.7x)
Compression time: 0.222 s, 3437.4 MB/s
Decompression time: 0.162 s, 4714.4 MB/s
Successful roundtrip!

*/

#include <stdio.h>
#include <assert.h>
#include <context.h>
#include "blosc.h"

#define KB 1024.
#define MB (1024KB)
#define GB (1024
MB)

#define CHUNKSIZE (200 * 1000)
#define NCHUNKS 500
#define NTHREADS 4

int main() {
static int64_t data[CHUNKSIZE];
static int64_t data_dest[CHUNKSIZE];
const size_t isize = CHUNKSIZE * sizeof(int64_t);
int dsize = 0;
int64_t nbytes, cbytes;
blosc2_cparams cparams = BLOSC_CPARAMS_DEFAULTS;
blosc2_dparams dparams = BLOSC_DPARAMS_DEFAULTS;
blosc2_schunk* schunk;
int i;
int j = 0;
int nchunk;
size_t nchunks;
blosc_timestamp_t last, current;
double ttotal;

printf("Blosc version info: %s (%s)\n",
BLOSC_VERSION_STRING, BLOSC_VERSION_DATE);

/* Initialize the Blosc compressor */
blosc_init();

/* Create a super-chunk container */
cparams.typesize = 8;
cparams.filters[0] = BLOSC_DELTA;
cparams.compcode = BLOSC_BLOSCLZ;
cparams.clevel = 9;
cparams.nthreads = NTHREADS;
dparams.nthreads = NTHREADS;
schunk = blosc2_new_schunk(cparams, dparams);

struct blosc2_context_s * cctx = schunk->cctx;
blosc_set_timestamp(&last);
for (nchunk = 1; nchunk <= NCHUNKS; nchunk++) {
for (i = 0; i < CHUNKSIZE; i++) {
data[i] = i * (int64_t)nchunk;
}
if (j % 2 != 0) {
cctx->nthreads = 1; // if different from 1 the program blocks
} else {
cctx->nthreads = NTHREADS;
}
nchunks = blosc2_append_buffer(schunk, isize, data);
printf("Compressed");
j++;
assert(nchunks == nchunk);
}
/* Gather some info */
nbytes = schunk->nbytes;
cbytes = schunk->cbytes;
blosc_set_timestamp(&current);
ttotal = blosc_elapsed_secs(last, current);
printf("Compression ratio: %.1f MB -> %.1f MB (%.1fx)\n",
nbytes / MB, cbytes / MB, (1. * nbytes) / cbytes);
printf("Compression time: %.3g s, %.1f MB/s\n",
ttotal, nbytes / (ttotal * MB));

/* Retrieve and decompress the chunks (0-based count) */
blosc_set_timestamp(&last);
for (nchunk = NCHUNKS-1; nchunk >= 0; nchunk--) {
dsize = blosc2_decompress_chunk(schunk, (size_t)nchunk,
(void *)data_dest, isize);
}
if (dsize < 0) {
printf("Decompression error. Error code: %d\n", dsize);
return dsize;
}
blosc_set_timestamp(&current);
ttotal = blosc_elapsed_secs(last, current);
printf("Decompression time: %.3g s, %.1f MB/s\n",
ttotal, nbytes / (ttotal * MB));

/* Check integrity of the first chunk */
for (i = 0; i < CHUNKSIZE; i++) {
if (data_dest[i] != (uint64_t)i) {
printf("Decompressed data differs from original %d, %zd!\n",
i, data_dest[i]);
return -1;
}
}

printf("Successful roundtrip!\n");

/* Free resources */
blosc2_destroy_schunk(schunk);

return 0;
}`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant