Permalink
Browse files

Update snappy to version 1.0.5.

  • Loading branch information...
1 parent df1a181 commit af0f4f9483fc9aef4d3af914d80d56b560272dcb @jrfonseca jrfonseca committed Mar 14, 2012
View

Large diffs are not rendered by default.

Oops, something went wrong.
View
@@ -1,3 +1,34 @@
+Snappy v1.0.5, February 24th 2012:
+
+ * More speed improvements. Exactly how big will depend on
+ the architecture:
+
+ - 3–10% faster decompression for the base case (x86-64).
+
+ - ARMv7 and higher can now use unaligned accesses,
+ and will see about 30% faster decompression and
+ 20–40% faster compression.
+
+ - 32-bit platforms (ARM and 32-bit x86) will see 2–5%
+ faster compression.
+
+ These are all cumulative (e.g., ARM gets all three speedups).
+
+ * Fixed an issue where the unit test would crash on system
+ with less than 256 MB address space available,
+ e.g. some embedded platforms.
+
+ * Added a framing format description, for use over e.g. HTTP,
+ or for a command-line compressor. We do not have any
+ implementations of this at the current point, but there seems
+ to be enough of a general interest in the topic.
+ Also make the format description slightly clearer.
+
+ * Remove some compile-time warnings in -Wall
+ (mostly signed/unsigned comparisons), for easier embedding
+ into projects that use -Wall -Werror.
+
+
Snappy v1.0.4, September 15th 2011:
* Speeded up the decompressor somewhat; typically about 2–8%
View
@@ -76,11 +76,11 @@ your calling file, and link against the compiled library.
There are many ways to call Snappy, but the simplest possible is
- snappy::Compress(input, &output);
+ snappy::Compress(input.data(), input.size(), &output);
and similarly
- snappy::Uncompress(input, &output);
+ snappy::Uncompress(input.data(), input.size(), &output);
where "input" and "output" are both instances of std::string.
@@ -1,5 +1,5 @@
Snappy compressed format description
-Last revised: 2011-08-09
+Last revised: 2011-10-05
This is not a formal specification, but should suffice to explain most
@@ -21,8 +21,8 @@ The stream starts with the uncompressed length (up to a maximum of 2^32 - 1),
stored as a little-endian varint. Varints consist of a series of bytes,
where the lower 7 bits are data and the upper bit is set iff there are
more bytes to be read. In other words, an uncompressed length of 64 would
-be stored as 0x40, and an uncompressed length of 2097151 (0x1FFFFF)
-would be stored as 0xFF 0xFF 0x7F.
+be stored as 0x40, and an uncompressed length of 2097150 (0x1FFFFE)
+would be stored as 0xFE 0xFF 0x7F.
2. The compressed stream itself
@@ -0,0 +1,124 @@
+Snappy framing format description
+Last revised: 2011-12-15
+
+This format decribes a framing format for Snappy, allowing compressing to
+files or streams that can then more easily be decompressed without having
+to hold the entire stream in memory. It also provides data checksums to
+help verify integrity. It does not provide metadata checksums, so it does
+not protect against e.g. all forms of truncations.
+
+Implementation of the framing format is optional for Snappy compressors and
+decompressor; it is not part of the Snappy core specification.
+
+
+1. General structure
+
+The file consists solely of chunks, lying back-to-back with no padding
+in between. Each chunk consists first a single byte of chunk identifier,
+then a two-byte little-endian length of the chunk in bytes (from 0 to 65535,
+inclusive), and then the data if any. The three bytes of chunk header is not
+counted in the data length.
+
+The different chunk types are listed below. The first chunk must always
+be the stream identifier chunk (see section 4.1, below). The stream
+ends when the file ends -- there is no explicit end-of-file marker.
+
+
+2. File type identification
+
+The following identifiers for this format are recommended where appropriate.
+However, note that none have been registered officially, so this is only to
+be taken as a guideline. We use "Snappy framed" to distinguish between this
+format and raw Snappy data.
+
+ File extension: .sz
+ MIME type: application/x-snappy-framed
+ HTTP Content-Encoding: x-snappy-framed
+
+
+3. Checksum format
+
+Some chunks have data protected by a checksum (the ones that do will say so
+explicitly). The checksums are always masked CRC-32Cs.
+
+A description of CRC-32C can be found in RFC 3720, section 12.1, with
+examples in section B.4.
+
+Checksums are not stored directly, but masked, as checksumming data and
+then its own checksum can be problematic. The masking is the same as used
+in Apache Hadoop: Rotate the checksum by 15 bits, then add the constant
+0xa282ead8 (using wraparound as normal for unsigned integers). This is
+equivalent to the following C code:
+
+ uint32_t mask_checksum(uint32_t x) {
+ return ((x >> 15) | (x << 17)) + 0xa282ead8;
+ }
+
+Note that the masking is reversible.
+
+The checksum is always stored as a four bytes long integer, in little-endian.
+
+
+4. Chunk types
+
+The currently supported chunk types are described below. The list may
+be extended in the future.
+
+
+4.1. Stream identifier (chunk type 0xff)
+
+The stream identifier is always the first element in the stream.
+It is exactly six bytes long and contains "sNaPpY" in ASCII. This means that
+a valid Snappy framed stream always starts with the bytes
+
+ 0xff 0x06 0x00 0x73 0x4e 0x61 0x50 0x70 0x59
+
+The stream identifier chunk can come multiple times in the stream besides
+the first; if such a chunk shows up, it should simply be ignored, assuming
+it has the right length and contents. This allows for easy concatenation of
+compressed files without the need for re-framing.
+
+
+4.2. Compressed data (chunk type 0x00)
+
+Compressed data chunks contain a normal Snappy compressed bitstream;
+see the compressed format specification. The compressed data is preceded by
+the CRC-32C (see section 3) of the _uncompressed_ data.
+
+Note that the data portion of the chunk, i.e., the compressed contents,
+can be at most 65531 bytes (2^16 - 1, minus the checksum).
+However, we place an additional restriction that the uncompressed data
+in a chunk must be no longer than 32768 bytes. This allows consumers to
+easily use small fixed-size buffers.
+
+
+4.3. Uncompressed data (chunk type 0x01)
+
+Uncompressed data chunks allow a compressor to send uncompressed,
+raw data; this is useful if, for instance, uncompressible or
+near-incompressible data is detected, and faster decompression is desired.
+
+As in the compressed chunks, the data is preceded by its own masked
+CRC-32C (see section 3).
+
+An uncompressed data chunk, like compressed data chunks, should contain
+no more than 32768 data bytes, so the maximum legal chunk length with the
+checksum is 32772.
+
+
+4.4. Reserved unskippable chunks (chunk types 0x02-0x7f)
+
+These are reserved for future expansion. A decoder that sees such a chunk
+should immediately return an error, as it must assume it cannot decode the
+stream correctly.
+
+Future versions of this specification may define meanings for these chunks.
+
+
+4.5. Reserved skippable chunks (chunk types 0x80-0xfe)
+
+These are also reserved for future expansion, but unlike the chunks
+described in 4.4, a decoder seeing these must skip them and continue
+decoding.
+
+Future versions of this specification may define meanings for these chunks.
@@ -68,5 +68,4 @@ char* UncheckedByteArraySink::GetAppendBuffer(size_t len, char* scratch) {
return dest_;
}
-
}
@@ -60,6 +60,7 @@ class Sink {
// The default implementation always returns the scratch buffer.
virtual char* GetAppendBuffer(size_t length, char* scratch);
+
private:
// No copying
Sink(const Sink&);
@@ -86,10 +86,9 @@ using namespace std;
// version (anyone who wants to regenerate it can just do the call
// themselves within main()).
#define DEFINE_bool(flag_name, default_value, description) \
- bool FLAGS_ ## flag_name = default_value;
+ bool FLAGS_ ## flag_name = default_value
#define DECLARE_bool(flag_name) \
- extern bool FLAGS_ ## flag_name;
-#define REGISTER_MODULE_INITIALIZER(name, code)
+ extern bool FLAGS_ ## flag_name
namespace snappy {
@@ -179,6 +178,8 @@ class LogMessageVoidify {
// Potentially unaligned loads and stores.
+// x86 and PowerPC can simply do these loads and stores native.
+
#if defined(__i386__) || defined(__x86_64__) || defined(__powerpc__)
#define UNALIGNED_LOAD16(_p) (*reinterpret_cast<const uint16 *>(_p))
@@ -189,6 +190,47 @@ class LogMessageVoidify {
#define UNALIGNED_STORE32(_p, _val) (*reinterpret_cast<uint32 *>(_p) = (_val))
#define UNALIGNED_STORE64(_p, _val) (*reinterpret_cast<uint64 *>(_p) = (_val))
+// ARMv7 and newer support native unaligned accesses, but only of 16-bit
+// and 32-bit values (not 64-bit); older versions either raise a fatal signal,
+// do an unaligned read and rotate the words around a bit, or do the reads very
+// slowly (trip through kernel mode). There's no simple #define that says just
+// “ARMv7 or higher”, so we have to filter away all ARMv5 and ARMv6
+// sub-architectures.
+//
+// This is a mess, but there's not much we can do about it.
+
+#elif defined(__arm__) && \
+ !defined(__ARM_ARCH_5__) && \
+ !defined(__ARM_ARCH_5T__) && \
+ !defined(__ARM_ARCH_5TE__) && \
+ !defined(__ARM_ARCH_5TEJ__) && \
+ !defined(__ARM_ARCH_6__) && \
+ !defined(__ARM_ARCH_6J__) && \
+ !defined(__ARM_ARCH_6K__) && \
+ !defined(__ARM_ARCH_6Z__) && \
+ !defined(__ARM_ARCH_6ZK__) && \
+ !defined(__ARM_ARCH_6T2__)
+
+#define UNALIGNED_LOAD16(_p) (*reinterpret_cast<const uint16 *>(_p))
+#define UNALIGNED_LOAD32(_p) (*reinterpret_cast<const uint32 *>(_p))
+
+#define UNALIGNED_STORE16(_p, _val) (*reinterpret_cast<uint16 *>(_p) = (_val))
+#define UNALIGNED_STORE32(_p, _val) (*reinterpret_cast<uint32 *>(_p) = (_val))
+
+// TODO(user): NEON supports unaligned 64-bit loads and stores.
+// See if that would be more efficient on platforms supporting it,
+// at least for copies.
+
+inline uint64 UNALIGNED_LOAD64(const void *p) {
+ uint64 t;
+ memcpy(&t, p, sizeof t);
+ return t;
+}
+
+inline void UNALIGNED_STORE64(void *p, uint64 v) {
+ memcpy(p, &v, sizeof v);
+}
+
#else
// These functions are provided for architectures that don't support
@@ -226,6 +268,20 @@ inline void UNALIGNED_STORE64(void *p, uint64 v) {
#endif
+// This can be more efficient than UNALIGNED_LOAD64 + UNALIGNED_STORE64
+// on some platforms, in particular ARM.
+inline void UnalignedCopy64(const void *src, void *dst) {
+ if (sizeof(void *) == 8) {
+ UNALIGNED_STORE64(dst, UNALIGNED_LOAD64(src));
+ } else {
+ const char *src_char = reinterpret_cast<const char *>(src);
+ char *dst_char = reinterpret_cast<char *>(dst);
+
+ UNALIGNED_STORE32(dst_char, UNALIGNED_LOAD32(src_char));
+ UNALIGNED_STORE32(dst_char + 4, UNALIGNED_LOAD32(src_char + 4));
+ }
+}
+
// The following guarantees declaration of the byte swap functions.
#ifdef WORDS_BIGENDIAN
@@ -46,7 +46,7 @@
#define SNAPPY_MAJOR 1
#define SNAPPY_MINOR 0
-#define SNAPPY_PATCHLEVEL 4
+#define SNAPPY_PATCHLEVEL 5
#define SNAPPY_VERSION \
((SNAPPY_MAJOR << 16) | (SNAPPY_MINOR << 8) | SNAPPY_PATCHLEVEL)
@@ -353,7 +353,6 @@ int ZLib::CompressAtMostOrAll(Bytef *dest, uLongf *destLen,
// compression.
err = deflate(&comp_stream_, flush_mode);
- const uLong source_bytes_consumed = *sourceLen - comp_stream_.avail_in;
*sourceLen = comp_stream_.avail_in;
if ((err == Z_STREAM_END || err == Z_OK)
@@ -397,7 +396,6 @@ int ZLib::CompressChunkOrAll(Bytef *dest, uLongf *destLen,
int ZLib::Compress(Bytef *dest, uLongf *destLen,
const Bytef *source, uLong sourceLen) {
int err;
- const uLongf orig_destLen = *destLen;
if ( (err=CompressChunkOrAll(dest, destLen, source, sourceLen,
Z_FINISH)) != Z_OK )
return err;
@@ -135,7 +135,7 @@ namespace File {
while (!feof(fp)) {
char buf[4096];
size_t ret = fread(buf, 1, 4096, fp);
- if (ret == -1) {
+ if (ret == 0 && ferror(fp)) {
perror("fread");
exit(1);
}
Oops, something went wrong.

0 comments on commit af0f4f9

Please sign in to comment.