FileChannelJournalSegmentReader throws half buffers away

[Zelldon]

Description

During our investigation of the engine latency we found out that the atomix journal is a big hotspot performance wise. We had a deeper look at the code and found the readNext method.

We have to say before that per Zeebe default the maxEntrySize is 4 MB. The reader will allocate a buffer which is 8MB big. Our assumption is that it wants to do a read a head, but actually it doesn't because it clears the buffer positions etc.

  /** Reads the next entry in the segment. */
  @SuppressWarnings("unchecked")
  private void readNext() {
    // Compute the index of the next entry in the segment.
    final long index = getNextIndex();

    try {
      // Read more bytes from the segment if necessary.
      if (memory.remaining() < maxEntrySize) {
        final long position = channel.position() + memory.position();
        channel.position(position);
        memory.clear(); // <=== this resets to position 0, limit to capacity
        channel.read(memory);  // <=== we read now again 8 MB
        channel.position(position);
        memory.flip();
      }

      // Mark the buffer so it can be reset if necessary.
      memory.mark();

      try {
        // Read the length of the entry.
        final int length = memory.getInt();

        // If the buffer length is zero then return.
        if (length <= 0 || length > maxEntrySize) {
          memory.reset().limit(memory.position());
          nextEntry = null;
          return;
        }

        // Read the checksum of the entry.
        final long checksum = memory.getInt() & 0xFFFFFFFFL;

        // Compute the checksum for the entry bytes.
        final Checksum crc32 = new CRC32();
        crc32.update(memory.array(), memory.position(), length);

        // If the stored checksum equals the computed checksum, return the entry.
        if (checksum == crc32.getValue()) {
          final int limit = memory.limit();
          memory.limit(memory.position() + length);
          final E entry = namespace.deserialize(memory);
          memory.limit(limit);
          nextEntry = new Indexed<>(index, entry, length);
        } else {
          memory.reset().limit(memory.position());
          nextEntry = null;
        }
      } catch (final BufferUnderflowException e) {
        memory.reset().limit(memory.position());
        nextEntry = null;
      }
    } catch (final IOException e) {
      throw new StorageException(e);
    }
  }

This means we will read 8 MB process from these ~4 MBuntil our remaining is less then maxEntrySize, so after we reached half of the buffer. Then we will read again 8MB where the other remaining is thrown away and will end at the beginning of the buffer, because it is re-read.

This problem is more an issue when the maxEntrySize and realEntrySize is very different, which is our case. We using 4MB to support big deployments but actually most of our records are less then 1 kb. If the sizes would be nearer then we would probably also read most of the buffer, but this is not our use case so we should fix this!