Maybe the max buffer capacity should be customized

[cichengzi]

Feature Description (功能描述)

In some extreme big data query tasks, if the amount of data I need to export is very large, MAX_BUFFER_CAPACITY will limit the export. This parameter is located in BytesBuffer under org.apache.hugegraph.backend.serializer in hugegraph-core. It defaults to 128MB and does not support customization. It is desirable to enable customization through configuration files.

[imbajin]

@dosu could u analysis & answer the question?

[dosubot]

This is a valid feature request. Here's the analysis:

Current Behavior:
MAX_BUFFER_CAPACITY is hardcoded as 128 * 1024 * 1024 (128 MB) in BytesBuffer.java. It acts as a hard ceiling in two places:

1. Buffer creation — throws IllegalArgumentException if the requested capacity exceeds 128 MB [1]
2. Dynamic resizing via require() — fails when the expanded capacity would exceed 128 MB [2]

This means any single serialization/deserialization operation that produces data larger than 128 MB will fail, which can block large data exports.

Developer Intent:
The code already contains // TODO: support user-defined configuration comments for related constants (ID_LEN_MAX, BYTES_LEN_MAX) [3], indicating the team recognized the need but hasn't implemented it yet.

Feasibility:
HugeGraph's CoreOptions class already defines several similar configurable memory limits [4] — e.g., MAX_MEMORY_CAPACITY (1 GB default), TASK_INPUT_SIZE_LIMIT (16 MB), and TASK_RESULT_SIZE_LIMIT (16 MB). Adding a new option for MAX_BUFFER_CAPACITY would follow the same pattern:

public static final ConfigOption<Integer> SERIALIZER_BUFFER_MAX_CAPACITY =
    new ConfigOption<>(
        "serializer.buffer_max_capacity",
        "Maximum capacity of the serialization buffer in bytes.",
        rangeInt(64 * 1024 * 1024, 512 * 1024 * 1024), // example: 64MB–512MB
        128 * 1024 * 1024 // default 128MB
    );

Implementation steps would include:

1. Add a ConfigOption in CoreOptions
2. Modify BytesBuffer to read the configured value (likely cached at initialization for performance in hot paths like require())
3. Update references in LZ4 compression utilities that also use this constant

This seems like a reasonable enhancement given the existing pattern and the TODO comments already in the code.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[LegendPei]

@imbajin I would like to try it,I will address this issue through the plan outlined by dosubot

[imbajin]

@LegendPei Thanks, I assigned this issue to you.

I agree this is a reasonable scoped improvement, but let's keep it simple and avoid over-designing it. Please focus on making the single BytesBuffer max capacity configurable while keeping the current default behavior.

Suggested scope:

• Add a CoreOptions config such as serializer.buffer_max_capacity, defaulting to the current 128MB.
• Use the configured value in BytesBuffer capacity checks and related LZ4Util logic.
• Keep a reasonable upper bound and make error messages show the effective limit.
• Add focused unit tests and update the sample config/docs if needed.

Note: this option should be treated as the max capacity of one serialization buffer, not as a general solution for huge exports. Large exports should still rely on paging/streaming when possible.