problem with PooledMemoryManager

[kentsang77]

I have used HeapMemoryManager for a long time and everything works fine.
Last week we tried grizzly version 2.3.30 and found that message decoder doesn't work at all.
Last working grizzly version was 2.3.24.
After investigation, we found the default MemoryManager changed to PooledMemoryManager at 2.3.26 and there is a strange issue when allocating a large encoding buffer (say 4M):

final MemoryManager memoryManager = obtainMemoryManager(storage);
final Buffer output = memoryManager.allocate(1024 * 1024 * 4); // ok if 4k but not 4m

The symptom is : only null bytes can be read at decoder.
I created a all in 1 class junit for your investigation.

StringMessageFilterTest.zip

[rlubke]

@kentsang77 You're aware how to change the default memory manager back to HeapMemoryManager?

[kofemann]

Try start jvm with

-Dorg.glassfish.grizzly.DEFAULT_MEMORY_MANAGER=org.glassfish.grizzly.memory.HeapMemoryManager

option

[rlubke]

The root of the issue is that with allocations that span multiple buffers, as is the case with the large allocation, the ByteBuffer you obtain from the Buffer isn't shared between the two.

The documentation for Buffer.toByteBuffer() states:

• Converts this Buffer to a {@link ByteBuffer}.
  • If this Buffer is not composite - then returned
  • {@link ByteBuffer}'s content is a shared subsequence of this buffer's
  • content, with {@link CompositeBuffer} this is not guaranteed.
  • The position of the returned {@link ByteBuffer} is not guaranteed to be 0,
  • the capacity of the returned {@link ByteBuffer} is not guaranteed to be
  • equal to the capacity of this Buffer.
  • It is guaranteed that the result of the returned ByteBuffer's
  • {@link ByteBuffer#remaining()} call will be equal (limit - position).
  • The Buffer's and ByteBuffer's position, limit, and mark values are not
  • guaranteed to be independent, so it's recommended to save and restore
  • position, limit values if it is planned to change them or
  • {@link ByteBuffer#slice()} the returned {@link ByteBuffer}.

I have a question about the test case.

You do the following:

` final Buffer output = memoryManager.allocate(allocationSize);
final ByteBuffer byteBuffer = output.toByteBuffer().slice();

        final byte[] byteRepresentation;
        try {
            if (stringTerminator != null) {
                input = input + stringTerminator;
            }

            byteRepresentation = input.getBytes(charset.name());

            if (stringTerminator == null) {
                byteBuffer.putInt(byteRepresentation.length);
            }
            byteBuffer.put(byteRepresentation);
            output.limit(byteBuffer.position());

            LOGGER.log(Level.INFO, "encode byteBuffer = {0}", Arrays.toString(Arrays.copyOfRange(byteBuffer.array(), 0, 100)));

        } catch (Exception e) {
            return TransformationResult.createErrorResult(0,
                    "Exception during construction of byteBuffer, Exception = " + e.getMessage());
        }

`

Is there a reason you're converting to a ByteBuffer prior to manipulating? Was this only to demonstrate the problem or do you do this in production?

[kentsang77]

It is currently doing at production.
All of my service messages are using jdk's ByteBuffer for serialization and deserialization.
I cannot change that to grizzly Buffer.

[rlubke]

Then the option you have is a) Set the default memory manager to HeapMemoryManager or b) if the Buffer is composite, copy the ByteBuffer content back to the Buffer.

The latter option isn't ideal for performance reasons.

[kentsang77]

My solution is to use a PooledMemoryManagerFactory to init a PooledMemoryManager with large enough DEFAULT_BASE_BUFFER_SIZE (where allocationSize < DEFAULT_BASE_BUFFER_SIZE ).

Personally I think if the PooledMemoryManager cannot handle large buffer size, it should throw exception. At least not just returning a weired empty buffer.