Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update snappy examples in wiki for snappy 7.x.x (async) #436

Closed
thomastoye opened this issue May 15, 2023 · 1 comment
Closed

Update snappy examples in wiki for snappy 7.x.x (async) #436

thomastoye opened this issue May 15, 2023 · 1 comment

Comments

@thomastoye
Copy link
Contributor

thomastoye commented May 15, 2023

BlockDecoder

The wiki for BlockDecoder (link) has an example for using snappy. It uses Buffer#slice, which is deprecated. It also mixes up async (callback-based) and sync (throwing an Error) error handling. I propose to update the example in the wiki to

const crc32 = require('buffer-crc32');
const snappy = require('snappy');

const blockDecoder =   new avro.streams.BlockDecoder({
    codecs: {
      null: (buf, cb) => cb(null, buf),
      snappy: async (buf, cb) => {
        // Avro appends checksums to compressed blocks.
        const len = buf.length
        const checksum = buf.subarray(len - 4, len)

        try {
          const inflated = await snappy.uncompress(buf.subarray(0, len - 4))

          if (!checksum.equals(crc32(inflated))) {
            // We make sure that the checksum matches.
            throw new Error('invalid checksum')
          }

          cb(null, inflated)
        } catch (err) {
          cb(err)
        }
      },
    },
  })

BlockEncoder

The wiki for BlockEncoder (link) relies on the old snappy sync behaviour. snappy.compress is now an async call (snappy 7.x.x). I propose to update the example to

const blockEncoder =   new avro.streams.BlockEncoder(someType, {
    writeHeader: 'auto',
    codec: 'snappy',
    codecs: {
      null: (buf, cb) => {
        cb(null, buf)
      },
      snappy: async (buf, cb) => {
        try {
          // Avro requires appending checksums to compressed blocks.
          const checksum = crc32(buf)
          const deflated = await snappy.compress(buf)

          const block = Buffer.alloc(deflated.length + 4)
          deflated.copy(block)
          checksum.copy(block, deflated.length)
          cb(null, block)
        } catch (err) {
          cb(err)
        }
      },
    },
  })

(Note that both examples also include the null codec, which is technically not needed for the example, but helps as an extra example on how to implement codecs)

@mtth
Copy link
Owner

mtth commented Jun 3, 2023

Thanks @thomastoye; examples updated.

@mtth mtth closed this as completed Jun 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants