Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compressor: Add Brotli Compressor #19549

Merged
merged 1 commit into from Jan 6, 2018

Conversation

Projects
None yet
4 participants
@bi-shun
Copy link
Contributor

commented Dec 15, 2017

Signed-off-by: BI SHUN KE aionshun@livemail.tw

.gitmodules Outdated
@@ -55,3 +55,6 @@
[submodule "src/rapidjson"]
path = src/rapidjson
url = https://github.com/ceph/rapidjson
[submodule "src/brotli"]
path = src/brotli
url = https://github.com/google/brotli.git

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 15, 2017

Contributor

awesome, but i'd suggest add brotli using ExternalProject_Add() and download brotli on the fly, see also https://github.com/ceph/ceph/blob/master/src/os/CMakeLists.txt#L122

@bi-shun bi-shun force-pushed the bi-shun:brotli-compressor branch 2 times, most recently from c44e005 to c7cd84d Dec 16, 2017

@bi-shun

This comment has been minimized.

Copy link
Contributor Author

commented Dec 19, 2017

@tchaikov I have modified as you suggested, can you review again, thanks.

@@ -719,6 +719,7 @@ add_subdirectory(include)
add_subdirectory(librados)
add_subdirectory(libradosstriper)


This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 24, 2017

Contributor

drop this change.

@@ -18,7 +19,8 @@ endif()
set(ceph_compressor_libs
ceph_snappy
ceph_zlib
ceph_zstd)
ceph_zstd

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 24, 2017

Contributor

could you avoid using tab in CMake scripts?

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 25, 2017

Contributor

@bi-shun this comment is not addressed.

@@ -31,6 +31,7 @@ const char * Compressor::get_comp_alg_name(int a) {
#ifdef HAVE_LZ4
case COMP_ALG_LZ4: return "lz4";
#endif
case COMP_ALG_BROTLI: return "brotli";

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 24, 2017

Contributor

should guard this case statement with #ifdef HAVE_BROTLI. as "brotli" could be disabled at compile-time.

@@ -46,6 +47,8 @@ boost::optional<Compressor::CompressionAlgorithm> Compressor::get_comp_alg_type(
if (s == "lz4")
return COMP_ALG_LZ4;
#endif
if (s == "brotli")

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 24, 2017

Contributor

ditto.

@@ -37,6 +37,7 @@ class Compressor {
#ifdef HAVE_LZ4
COMP_ALG_LZ4 = 4,
#endif
COMP_ALG_BROTLI = 5,

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 24, 2017

Contributor

ditto.

size_t available_out = 0;
uint8_t* next_out = NULL;
if (!s) {
BrotliEncoderDestroyInstance(s);

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 24, 2017

Contributor

no need to destroy s, if it's not created at all.

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 25, 2017

Contributor

@bi-shun this comment is not addressed.

BrotliEncoderSetParameter(s, BROTLI_PARAM_QUALITY, (uint32_t)9);
BrotliEncoderSetParameter(s, BROTLI_PARAM_LGWIN, 22);
unsigned char* c_in;
for (std::list<buffer::ptr>::const_iterator i = in.buffers().begin(); i != in.buffers().end();) {

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 24, 2017

Contributor

nit, use auto

out.append(ptr, 0, have);
} while (available_out == 0);
if (BrotliEncoderIsFinished(s)) {
BrotliEncoderDestroyInstance(s);

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 24, 2017

Contributor

why not just break here?

have = MAX_LEN - available_out;
out.append(ptr, 0, have);
} while (available_out == 0);
if (BrotliDecoderIsFinished(s)) {

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 24, 2017

Contributor

could just break here.

COMMAND "true"
ALWAYS 1)

add_library(brotlicommon STATIC IMPORTED)

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 24, 2017

Contributor

nit, could name it brotli::common and use a loop to set the dependency and the properties of these libraries.

@bi-shun bi-shun force-pushed the bi-shun:brotli-compressor branch 3 times, most recently from 7033bfa to 081a3f8 Dec 25, 2017

@bi-shun

This comment has been minimized.

Copy link
Contributor Author

commented Dec 25, 2017

@tchaikov I have to change the compressor plugin as optional and modify for you suggestions, can you review again, thanks.

@@ -330,7 +330,11 @@ INSTANTIATE_TEST_CASE_P(
#endif
"zlib/noisal",
"snappy",
"zstd"));
"zstd"
#ifdef HAVE_BROTLI

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 25, 2017

Contributor

nit, could add this line before "zstd", less code churn, and better looking this way =)

COMMAND "true"
ALWAYS 1)

set(bortli_libs brotlicommon brotlidec brotlienc)

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 25, 2017

Contributor

s/brotlicommon/brotli::common/
s/brotlidec/brotli::dec/
s/brotlienc/brotli::enc/

option(WITH_BROTLI "Brotli compression support" OFF)
if(WITH_BROTLI)
set(HAVE_BROTLI TRUE)
endif(WITH_BROTLI)

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 25, 2017

Contributor

remove WITH_BROTLI from endif(), as it's a very short if block.

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 28, 2017

Contributor

@bi-shun this one.

set(HAVE_BROTLI TRUE)
endif(WITH_BROTLI)


This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 25, 2017

Contributor

remove this empty line.

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 28, 2017

Contributor

@bi-shun and this one.

@@ -15,6 +15,10 @@ if (HAVE_LZ4)
add_subdirectory(lz4)
endif()

if (HAVE_BROTLI)

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 25, 2017

Contributor

nit, remove the space after if.

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 28, 2017

Contributor

@bi-shun here.

@bi-shun

This comment has been minimized.

Copy link
Contributor Author

commented Dec 25, 2017

@tchaikov I have fixed the problems.

@bi-shun

This comment has been minimized.

Copy link
Contributor Author

commented Dec 28, 2017

@tchaikov Cloud you review again, thanks.

@@ -24,6 +28,10 @@ if (HAVE_LZ4)
list(APPEND ceph_compressor_libs ceph_lz4)
endif()

if (HAVE_BROTLI)

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 28, 2017

Contributor

ditto.

@tchaikov

This comment has been minimized.

Copy link
Contributor

commented Dec 28, 2017

@bi-shun please let me know once all comments are addressed.

@bi-shun bi-shun force-pushed the bi-shun:brotli-compressor branch from 8a67338 to f03292e Dec 28, 2017

@bi-shun

This comment has been minimized.

Copy link
Contributor Author

commented Dec 28, 2017

@tchaikov I merge the branch commits into one, and all comments are addressed, thanks.

@bi-shun bi-shun force-pushed the bi-shun:brotli-compressor branch 3 times, most recently from f4b09f3 to db1b72f Dec 28, 2017

int BrotliCompressor::compress(const bufferlist &in, bufferlist &out)
{
BrotliEncoderState* s = BrotliEncoderCreateInstance(nullptr, nullptr, nullptr);
auto sg = make_scope_guard([&s] { BrotliEncoderDestroyInstance(s); });

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 28, 2017

Contributor

#19549 (comment) , please move this after if (!s) {} block.

size_t available_in = 0;
const uint8_t* next_in = nullptr;
size_t available_out = 0;
uint8_t* next_out = nullptr;

This comment has been minimized.

Copy link
@tchaikov
}
BrotliEncoderSetParameter(s, BROTLI_PARAM_QUALITY, (uint32_t)9);
BrotliEncoderSetParameter(s, BROTLI_PARAM_LGWIN, 22);
unsigned char* c_in;

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 28, 2017

Contributor

no need to have this variable, IMHO.

unsigned char* c_in;
for (auto i = in.buffers().begin(); i != in.buffers().end();) {
c_in = (unsigned char*) i->c_str();
size_t len = i->length();

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 28, 2017

Contributor

why do we need this variable?

bufferptr ptr = buffer::create_page_aligned(max_comp_size);
next_out = (unsigned char*)ptr.c_str();
available_out = max_comp_size;
if (!BrotliEncoderCompressStream(s,finish,&available_in, &next_in, &available_out, &next_out, nullptr)) {

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 28, 2017

Contributor

add a space after , please

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 28, 2017

Contributor

please wrap at 80 chars.

add_library(ceph_brotli SHARED ${brotli_sources})
add_dependencies(ceph_brotli ${CMAKE_SOURCE_DIR}/src/ceph_ver.h)
target_include_directories(ceph_brotli PRIVATE "${CMAKE_BINARY_DIR}/src/brotli/c/include")
List(REVERSE bortli_libs)

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 28, 2017

Contributor

why reverse the libs list?

This comment has been minimized.

Copy link
@bi-shun

bi-shun Dec 28, 2017

Author Contributor

Because depend sequence.

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 29, 2017

Contributor

nit, if brotli::dec and brotli::enc depend on brotli::common, why not ordering it the right way at https://github.com/ceph/ceph/pull/19549/files#diff-66716f6999869d6458bd6a9251638b70R25 ?

#ifdef HAVE_BROTLI
"brotli",
#endif
"zstd"

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 28, 2017

Contributor

drop this change.

@bi-shun bi-shun force-pushed the bi-shun:brotli-compressor branch 4 times, most recently from 1c3e2f1 to 7a4dd35 Dec 28, 2017

@bi-shun bi-shun force-pushed the bi-shun:brotli-compressor branch from 7a4dd35 to a496ad7 Dec 29, 2017

@bi-shun

This comment has been minimized.

Copy link
Contributor Author

commented Dec 29, 2017

@tchaikov Thank you very much for your helpful comments and corrections. I tried to correct according to your comments . Could you review again, thanks.

@tchaikov
Copy link
Contributor

left a comment

thanks! lgtm, would be great if you can address the comments.

add_library(ceph_brotli SHARED ${brotli_sources})
add_dependencies(ceph_brotli ${CMAKE_SOURCE_DIR}/src/ceph_ver.h)
target_include_directories(ceph_brotli PRIVATE "${CMAKE_BINARY_DIR}/src/brotli/c/include")
List(REVERSE bortli_libs)

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 29, 2017

Contributor

nit, if brotli::dec and brotli::enc depend on brotli::common, why not ordering it the right way at https://github.com/ceph/ceph/pull/19549/files#diff-66716f6999869d6458bd6a9251638b70R25 ?

target_include_directories(ceph_brotli PRIVATE "${CMAKE_BINARY_DIR}/src/brotli/c/include")
List(REVERSE bortli_libs)
target_link_libraries(ceph_brotli ${bortli_libs})
set_target_properties(ceph_brotli PROPERTIES VERSION 2.0.0 SOVERSION 2)

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 29, 2017

Contributor

I don't think we need to version the plugin.

COMMAND "true"
ALWAYS 1)

set(bortli_libs brotli::common brotli::dec brotli::enc)

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 29, 2017

Contributor

nit, could avoid the string manipulation, and dedup the brotli:: prefix, see


if(WITH_EMBEDDED)
add_library(cephd_compressor_brotli STATIC ${brotli_sources})
target_include_directories(cephd_compressor_brotli PRIVATE "${CMAKE_BINARY_DIR}/src/brotli/c/include")

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 29, 2017

Contributor

remove the extra space.


virtual int factory(CompressorRef *cs, std::ostream *ss)
{
if (compressor == 0) {

This comment has been minimized.

Copy link
@tchaikov

tchaikov Dec 29, 2017

Contributor

could just put

if (compressor) {
// ...
}

or s/0/nullptr/ if you prefer to be explicit.

@tchaikov

This comment has been minimized.

Copy link
Contributor

commented Dec 29, 2017

the change looks good, could you please squash these two commits into a single one?

Compressor: Add Brotli Compressor
Signed-off-by: BI SHUN KE <aionshun@livemail.tw>

@bi-shun bi-shun force-pushed the bi-shun:brotli-compressor branch from a496ad7 to f0aa226 Dec 29, 2017

@bi-shun

This comment has been minimized.

Copy link
Contributor Author

commented Dec 29, 2017

@tchaikov Sure, last the comments, I have corrected and squash it into one, thanks.

@liewegas liewegas changed the title Compressor: Add Brotli Compressor compressor: Add Brotli Compressor Dec 29, 2017

@yuriw

This comment has been minimized.

Copy link
Contributor

commented Jan 2, 2018

@yuriw yuriw merged commit 392314d into ceph:master Jan 6, 2018

5 checks passed

Docs: build check OK - docs built
Details
Signed-off-by all commits in this PR are signed
Details
Unmodified Submodules submodules for project are unmodified
Details
make check make check succeeded
Details
make check (arm64) make check succeeded
Details

@bi-shun bi-shun deleted the bi-shun:brotli-compressor branch Jan 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.