Skip to content

Critical bug fixes, Buffer support, TypeScript
Compare
Choose a tag to compare
@divmgl divmgl released this 28 Dec 08:30
· 20 commits to master since this release
v2.0.1
27a2869

If you're using jackd in production, you should consider upgrading to this version. This is a major release of jackd that fixes some critical bugs and introduces Buffer support.

For existing users in production

I did my best to try to avoid breaking changes, but unfortunately you'll still at the very least need to change one line of code. If you're hard-pressed for time, don't care and/or you just want to upgrade to version 2 without changing your project, you can instantiate the jackd version 2 client with the following option enabled:

const beanstalkd = new Jackd({ useLegacyStringPayloads: true })

This will retain all of the existing functionality where job reservations were returning payloads as ASCII strings.

const { id, payload } = await beanstalkd.reserve() // => { id: "123", payload: "my random job" }

However, you probably should not do this. More details in the following sections.

Release notes

  • Bug fix: Strings and objects passed into put are no longer encoded into ASCII strings.
  • Bug fix: Reserving a job no longer decodes incoming payloads as ASCII strings.
  • Bug fix: jobs that have an \r\n no longer cause instability.
  • put now accepts Buffers. Buffers are sent as-is and no encoding is performed. Strings and objects are still converted to byte arrays, but jackd now uses Node.js's UTF-8 encoding by default.
  • reserve now returns Buffer payloads.

Critical bug fixes

Payloads are no longer encoded/decoded as ASCII

From its inception, jackd was designed to be an ASCII library. Indeed, as stated literally in the first line of the beanstalkd protocol docs, the beanstalkd protocol communicates with clients using ASCII encoding and line breaks:

The beanstalk protocol runs over TCP using ASCII encoding.

Further below however, there's a clarification:

The protocol contains two kinds of data: text lines and unstructured chunks of data. Text lines are used for client commands and server responses. Chunks are used to transfer job bodies and stats information. Each job body is an opaque sequence of bytes. The server never inspects or modifies a job body and always sends it back in its original form. It is up to the clients to agree on a meaningful interpretation of job bodies.

Unfortunately, jackd did not adhere to this section of the protocol and was encoding and decoding job payloads as ASCII. In v1, every command sent was encoded in ASCII, including job payloads:

JackdClient.prototype.write = function(string) {
  assert(string)

  return new Promise((resolve, reject) => {
    this.socket.write(string, 'ascii', err => (err ? reject(err) : resolve()))
  })
}

Note the 'ascii' encoding in socket.write. Similarly, all of the incoming data to the socket was encoded into ASCII: https://github.com/getjackd/jackd/blob/v1.2.6/src/index.js#L33

There are obvious, and potentially catastrophic, problems with this. Firstly, if you provided jackd with strings containing characters outside of the ASCII encoding, such as special UTF-8 characters that are not alphanumeric, the payload would be irreversibly corrupted. Secondly, if you received any job with a payload that wasn't ASCII encoded, the job was still decoded using ASCII. This meant that if you sent any other kind of encoded data, such as binary data, this data is corrupted and likely deleted right after (depending on your setup).

If you've been using jackd to send over simple UUIDs, integers and other simple payloads, you may not have noticed that there is a problem. In fact, you may not have lost any data. Additionally, if you've run into this problem during your testing you may have used base64 or hex encoding to send binary information over the wire. You no longer need to worry about this with jackd version 2. jackd now accepts Buffers and will send your data as-is. Similarly, payloads from jobs reserved with jackd will now come through as Buffers.

This is a breaking change. You'll need to go through all of your payloads and change them from payload to payload.toString(). On a brighter note, because jackd was rewritten in TypeScript, you'll get notified out of the box if you use TypeScript in your project.

Payloads with new lines will no longer cause instability and data loss

jackd also had a big architectural problem related to the way it processed messages. As per the beanstalkd protocol, there are two types of data: text lines and chunks of data. Both text lines and chunks of data are delimited by \r\n. However, commands that return chunks of data will inform the client how many bytes the following message will contain. For instance, the reserve job will reply something like this:

RESERVED 2 19\r\n

Where the first number is the job ID and the second number is the number of bytes that will be contained in the following message. This lets clients know how many more bytes to consume before the payload is consumed and processed.

Unfortunately, jackd was not using this byte count to consume subsequent messages, even though it did have support for chunks. The original architecture allowed commands to be designated as multipart commands. While these commands would create two handlers for incoming messages, the messages were always processed the same: using delimiters.

Of course, this always resulted in passing tests because chunks are also delimited by new lines. But as soon as a job payload that is ASCII encoded contained \r\n, jackd would begin executing all of the subsequent messages in that same data frame incorrectly. This is potentially catastrophic due to the data loss that occurs when commands are executed incorrectly.

This issue is now fixed and the message parsing architecture is significantly improved. jackd now fully leverages the serial nature of beanstalkd and keeps a queue of the incoming data buffers, the messages that are successfully processed, and the commands that have been executed by the caller. Reserving a job now takes bytes into account and subsequent messages are no longer parsed by looking at delimiter alone. There is now also a test for this.

Improvements

Buffers

As a result of all of this, jackd now supports Buffers as a first-class citizen. This means that you can send in all kinds of different data and jackd won't encode it at all.

const id = await beanstalkd.put(Buffer.from('my-awesome-job', 'utf-8'))

Native TypeScript

jackd has had TypeScript data bindings for some while thanks to the community. However, after realizing a huge refactor was in order, I bit the bullet and converted jackd over to TypeScript.

As a result of this, if you have TypeScript in your project you'll be alerted to all of the breaking changes when you upgrade to jackd@2. Additionally you may see some richer type annotations in some places.

What's next?

There are still some rough spots in the library that could probably use some polishing. If you encounter any bugs at all during this release, do not hesitate to open an issue.

Assets 2