Recommended ArcadeDB Configuration for High-Concurrency Production Workload

[HarinathMaddela]

Hello ArcadeDB Team,

I have a system where multiple jobs can run simultaneously, and these jobs may generate a heavy workload with many parallel requests(Read/Write) accessing the ArcadeDB database.

Currently, I have configured my Docker-based ArcadeDB-26.10.1 deployment with the following settings:

ARCADEDB_OPTS_MEMORY=-Xms4G -Xmx4G
arcadedb.server.mode=production
arcadedb.pageCacheSize=1073741824
arcadedb.txWalFlush=1
arcadedb.bucketDefaultPageSize=65536

The system is currently assigned 8 CPU cores.

I would like to understand whether this configuration is suitable for a production environment with a high level of parallel and potentially heavy database activity.

I have observed that, at times, the CPU utilization reaches almost 100% across all 8 CPU cores. Since the expected workload may increase further, I would like to understand the recommended configuration and sizing for this type of workload.

Could you please advise on the following?

Are the above ArcadeDB and JVM settings appropriate for a production system with heavy concurrent workloads?
Are there any specific configuration parameters that you would recommend tuning for high parallelism?
Is 4 GB of JVM heap and 1 GB page cache sufficient, or should these be increased based on the workload?
How many CPU cores would you generally recommend for a system handling multiple concurrent, CPU-intensive database jobs?
Is there a recommended CPU/memory ratio or sizing guideline for ArcadeDB that we can use when planning for increased load?
Since 8 CPU cores is the maximum I can allocate to this system, and I would actually prefer to reduce the CPU allocation if possible(maybe 4 cores), what would you recommend as a reasonable CPU configuration for this workload? If the CPUs are consistently highly utilized, should I first tune any ArcadeDB/JVM/database settings to improve CPU efficiency, or is the high CPU utilization expected for this type of parallel workload?

Any recommendations regarding CPU, memory, page cache, WAL, concurrency, and other relevant production settings would be greatly appreciated.

Thank you again for your continued support.

Regards,
Harinath

[keremoktener]

One concrete correction: in 26.10.1 the page-cache setting is arcadedb.maxPageRAM, measured in MB. arcadedb.pageCacheSize isn't a recognized setting. For your intended 1 GiB cache, use:

arcadedb.maxPageRAM=1024

The 26.10.1 configuration source defaults this to one quarter of the maximum heap, so with -Xmx4G you may already have approximately that cache size. Correcting the setting doesn't by itself explain the CPU saturation.

Also, the page cache is inside the JVM heap, so your configuration isn't 4 GiB heap plus another 1 GiB cache. Leave container memory headroom for non-heap allocations.

Before reducing from eight cores to four, benchmark your representative workload at bounded client concurrency, recording throughput, p95 latency, GC time and transaction retries. Check expensive queries with EXPLAIN, particularly full scans and missing indexes. High CPU alone doesn't tell you whether the time is useful query execution, GC or contention.

What is the container's memory limit, and could you share representative queries, the read/write mix, and your throughput/latency target? Those would make a sizing recommendation much more concrete.

[Das-Rabindra]

@keremoktener Currently, we are running ArcadeDB with the following configuration:

IMAGE=alineconsulting/arcadedb:26.10.1-061026

JAVA_OPTS_VAL='-XX:+UseZGC -XX:+ZGenerational -Djava.net.preferIPv4Stack=true -Darcadedb.server.rootPassword=***** -Darcadedb.server.plugins=Postgres:com.arcadedb.postgres.PostgresProtocolPlugin,Bolt:com.arcadedb.bolt.BoltProtocolPlugin -Darcadedb.postgres.port=5432 -Darcadedb.bolt.port=7687'

SETTINGS_VAL='-Darcadedb.server.mode=production -Darcadedb.txWalFlush=1 -Darcadedb.maxPageRAM=1024 -Darcadedb.bucketDefaultPageSize=65536 -Darcadedb.dumpConfigAtStartup=true -Darcadedb.studio.enabled=true'

docker run \
  --name="arcade" \
  --cpus="8" \
  --memory="8589934592" \
  --restart unless-stopped \
  --volume /data/limadata/arcadedb/replication:/home/arcadedb/replication \
  --volume /data/limadata/arcadedb/databases:/home/arcadedb/databases \
  --volume /data/limadata/arcadedb/config:/home/arcadedb/config \
  --volume /data/limadata/arcadedb/log:/home/arcadedb/log \
  --volume /data/limadata/arcadedb/backups:/home/arcadedb/backups \
  --env "ARCADEDB_OPTS_MEMORY=-Xms4G -Xmx4G" \
  --env "JAVA_OPTS=$JAVA_OPTS_VAL" \
  --env "ARCADEDB_SETTINGS=$SETTINGS_VAL" \
  --network=bridge \
  --workdir=/home/arcadedb \
  -p 2424:2424 \
  -p 2480:2480 \
  -p 5433:5432 \
  -p 7687:7687 \
  --expose=27017 --expose=50051 --expose=6379 --expose=8182 --expose=9998 --expose=9999 \
  --runtime=runc \
  --detach=true \
  "$IMAGE" ./bin/server.sh

The container is limited to 8 GiB RAM and 8 CPUs, with the JVM heap configured as:

-Xms4G -Xmx4G

We have also configured:

-Darcadedb.maxPageRAM=1024

Could you please confirm whether this is an appropriate configuration for our 8 GiB container, particularly the combination of -Xmx4G and maxPageRAM=1024?

Also, are there any other ArcadeDB memory-related settings or configuration changes you would recommend for this setup to avoid unnecessary memory pressure while maintaining good query performance?

Thank you.

[keremoktener]

Thanks for the details. A 4 GiB heap with maxPageRAM=1024 looks like a reasonable starting point for an 8 GiB container. That 1 GiB cache sits inside the heap; the remaining container memory still needs to cover native allocations, thread stacks, direct buffers and filesystem cache.

I’d keep those limits initially and measure under representative load before increasing the cache or reducing CPU. In 26.10.1, queryMaxHeapRAM also defaults to about 2048 MB with this heap. Keep that protection enabled, though it only covers tracked query buffers, not every allocation.

Since this is a custom image, which Java version does it use? A sample with concurrency, read/write mix, throughput, p95 latency, heap usage after GC and peak container memory would help narrow down whether the pressure is coming from queries, GC or something else.

[Das-Rabindra]

Thanks for the clarification.

We checked the Java version inside our custom ArcadeDB 26.10.1 image. It is:

openjdk 21.0.12.1 2026-08-18 LTS
OpenJDK Runtime Environment Temurin-21.0.12.1+1
OpenJDK 64-Bit Server VM Temurin-21.0.12.1+1-LTS

Our custom image is based directly on:

FROM arcadedata/arcadedb:26.10.1

The only customization is adding shadow and su-exec and using an entrypoint to align the arcadedb UID/GID with the mounted host directories. We are not changing the Java runtime.

The current runtime configuration is:

Java: OpenJDK/Temurin 21.0.12.1 LTS
Container memory: 8 GiB
CPU: 8 cores
JVM: -Xms4G -Xmx4G
GC: ZGC + Generational ZGC
Page RAM: 1024 MB
queryMaxHeapRAM: default

We will keep these limits initially and collect representative workload metrics including concurrency, read/write mix, throughput, p95 latency, post-GC heap usage, and peak container memory.

Regarding queryMaxHeapRAM, we will leave the default enabled initially rather than overriding it, so we can establish a clean baseline.

Thanks for the guidance.

[keremoktener]

Sounds good, thanks for confirming. Feel free to share the measurements here once you have them.

[Das-Rabindra]

dump&logs.zip
OOM

[HarinathMaddela]

Hi ArcadeDB Team,

My workload can consist of a combination of read and update queries, with multiple queries potentially running in parallel. In a real-world scenario, this could be similar to multiple users accessing the same feature simultaneously, resulting in several concurrent database operations.

I am attaching a few sample queries for reference — two are read queries and the others are update queries.

Query1

MATCH (a1:asset_info_master {asset_type: 'TABLE', asset_role_group: 'system_admin', created_user: 'harinath'}) WITH COLLECT(a1) AS r1 OPTIONAL MATCH (:role_master {role_group_id: 'system_admin'})-[:lineage_edge]->(a2:asset_info_master {asset_type: 'TABLE'}) WITH r1, COLLECT(DISTINCT a2) AS r2 OPTIONAL MATCH (:role_group_master {_key: 'system_admin'})-[:lineage_edge]->(a3:asset_info_master {asset_type: 'TABLE'}) WITH r1, r2, COLLECT(DISTINCT a3) AS r3 OPTIONAL MATCH (p:public_asset {asset_type: 'TABLE'}) OPTIONAL MATCH (a4:asset_info_master {_key: p._key}) WITH r1, r2, r3, COLLECT(DISTINCT a4) AS r4 WITH r1 + r2 + r3 + r4 AS combined UNWIND combined AS n WITH DISTINCT n WHERE n.asset_type = 'TABLE' OPTIONAL MATCH (n)<-[:lineage_edge]-(v:asset_info_master) WHERE v.data.vendor IS NOT NULL WITH n, COLLECT(DISTINCT v.data.vendor)[0] AS vendor WITH n { .*, isOwner: n.created_user = 'harinath', access: 'DELETE', vendor: vendor } AS asset WHERE  asset.asset_type IN ["TABLE"] ORDER BY   asset.project_order ASC , asset.catalogue_order ASC  RETURN properties(asset) AS result

Query2

SELECT name FROM schema:types WHERE name LIKE :prefix

Query3

UPDATE DSP_GRAPH_123  SET data = [{'result': {'asset_project': 'master', 'asset_used_for': null, 'data': {'linkedColumns': null, 'columnSourceMapping': {'country': null, 'unitsold': null, 'unitcost': null, 'created_user': null, 'orderid': null, 'orderdate': null, 'modified_date': null, 'unitprice': null, 'orderpriority': null, 'totalcost': null, 'totalprofit': null, 'modified_user': null, 'totalrevenue': null, 'itemtype': null, 'saleschannel': null, 'product_id': null, 'br_status': null, 'system_data_source': null, 'id': null, 'created_date': null, 'region': null, 'shipdate': null}, 'executionOrderForSmartColumnsLookups': [], 'dataChangeLog': false, 'columnOrders': ['product_id', 'region', 'country', 'itemtype', 'saleschannel', 'orderpriority', 'orderdate', 'orderid', 'shipdate', 'unitsold', 'unitprice', 'unitcost', 'totalrevenue', 'totalcost', 'totalprofit', 'system_data_source', 'created_user', 'created_date', 'br_status', 'modified_user', 'modified_date', 'id'], 'asset_sub_type': 'LIMATABLE', 'oneTimeColumns': ['system_data_source', 'created_user', 'created_date'], 'tableName': 'performance_test_lkpsrc', 'groupName': 'master', 'sequentialLinkedColumns': null, 'assetTableName': 'performance_test_lkpsrc', 'autoDelete': null, 'automaticColumns': ['br_status', 'modified_user', 'modified_date'], 'sequenceNameVariableNames': {}, 'connectionName': 'limadbconn', 'additionalColumnInfo': {'country': {'displayName': 'country', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'unitsold': {'displayName': 'unitsold', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'unitcost': {'displayName': 'unitcost', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'created_user': {'displayName': 'created_user', 'validations': {'allowNull': true}}, 'orderid': {'displayName': 'orderid', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'orderdate': {'displayName': 'orderdate', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'modified_date': {'displayName': 'modified_date', 'validations': {'allowNull': true}}, 'unitprice': {'displayName': 'unitprice', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'orderpriority': {'displayName': 'orderpriority', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'totalcost': {'displayName': 'totalcost', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'totalprofit': {'displayName': 'totalprofit', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'modified_user': {'displayName': 'modified_user', 'validations': {'allowNull': true}}, 'totalrevenue': {'displayName': 'totalrevenue', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'itemtype': {'displayName': 'itemtype', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'saleschannel': {'displayName': 'saleschannel', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'product_id': {'displayName': 'product_id', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': false}}, 'br_status': {'displayName': 'br_status', 'validations': {'allowNull': true}}, 'system_data_source': {'displayName': 'system_data_source', 'validations': {'allowNull': true}}, 'id': {'displayName': 'id', 'validations': {'allowNull': false}}, 'created_date': {'displayName': 'created_date', 'validations': {'allowNull': true}}, 'region': {'displayName': 'region', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}, 'shipdate': {'displayName': 'shipdate', 'description': '', 'validations': {'functionalColumnConfig': null, 'allowNull': true}}}, 'projectName': 'master', 'dbCatalog': 'system_admin', 'temporal': false, 'isVersionTable': null}, 'asset_id': 'a1cf1702-25d2-424d-b630-b5ba0370dd93', '_key': 'system_admin:TABLE@performance_test_lkpsrc', 'asset_location': null, 'modified_user': 'harinath', 'public': true, 'asset_name': 'performance_test_lkpsrc', 'asset_type': 'TABLE', 'asset_description': null, 'revisionStatus': 'ACTIVE', 'asset_role_group': 'system_admin', 'is_active': 'ACTIVE', 'catalogue_order': 1, 'created_user': 'harinath', 'owner_user_id': 'harinath', 'display_name': 'performance_test_lkpsrc', 'modified_date': '2026-10-07 11:07:03', 'asset_sub_type': 'LIMATABLE', 'revisionLabel': 'base', 'security_level': null, 'asset_catalogue': 'master', 'created_date': '2026-10-07 11:07:03', '_id': 'asset_info_master/system_admin:TABLE@performance_test_lkpsrc', 'project_order': 1, 'asset_extras': null, 'access': 'DELETE', 'isOwner': true}}], success = true WHERE _key = 'KEY_DSP_GRAPH_123'

As part of my testing, I executed Query 1 concurrently 50 times across multiple threads. During this test, I observed an OutOfMemory (OOM) condition. There was also a significant increase in system load at the same time, and the OOM occurred while the system was under this high concurrent workload.

I wanted to share this additional information to provide more context around the workload and help understand whether the observed behavior is related to the query structure, concurrency, or ArcadeDB configuration.

Thanks,
Harinath

[lvca]

This fix is not in main branch.

[lvca]

Opened an issue for his: #9404

[lvca]

Please check #9404 (comment)

[tae898]

We looked at where Query1's CPU goes, on a synthetic copy of this shape (10,000 asset_info_master vertices, about 27,000 lineage_edge edges, Query1 returning about 5,000 rows), embedded, on 26.10.2 and on current main. The matching is cheap. Most of the time is in the three WITH r1, r2, COLLECT(DISTINCT ...) steps: each groups by the lists collected before it, and that list key is canonicalized and hashed again for every row, so the cost grows with rows times list length. Filed as #10254 with a small reproduction.

In the same copy, the four sources computed as UNION branches (RETURN n._key) gave the same node set in 10.91 ms, against 170.65 ms for the collect chain. Indexes on _key and asset_type changed Query1 little (252.24 ms without, 258.48 ms with, JDK 25), and eight concurrent copies of the query showed no lock contention (no monitor waits in a profile), so under load it is this per-row work, repeated by each concurrent query, that fills the cores.