Greptime
Replies: 3 comments 37 replies
|
Would you mind running Currently, we can't use index to accelerate |
You mentioned that you are in v1.0.0-rc.1 above. Our latest release is v1.1.2. So I guessed you are still using that version.
The format switching isn't automatic because it needs to flush your existing data. Since your instance is under high memory pressure, you may need to retry the alteration if it was killed. Because switching format needs to flush all in memory data. If switching wasn't successful, you can stop the ingestion first.
No, this is a default configuration entry. But I found another way to check all your table settings. You can check all your tables' create options with this SQL. And you will see the merge mode in it. |
|
You can check if switching the format can fix this after all tables are in the Our docker image would output the logs to It'd be helpful to share your config file (the TOML file). If you changed settings like |
|
Oh, I missed one pprof file So I guess compaction may be the root cause. You can try this configuration For example: You can set experimental_compaction_on_exhausted to "fail". |
|
I might not have mentioned it earlier, but I did upgrade to 1.1.1 in the last week or two. Do you have a secure place I could upload the logs? Unzipped they are 1.9G. And that is just some of yesterday and today. I had had debug logging turned on too long had to clean out over 100gb of logs when I switched to local storage. My custom.toml file: default_timezone = "America/Los_Angeles"
max_in_flight_write_bytes = "8GB"
## The maximum current queries allowed to be executed. Zero means unlimited.
max_concurrent_queries = 0
## Enable telemetry to collect anonymous usage data. Enabled by default.
enable_telemetry = false
[storage]
data_home = "/greptimedb_data"
type = "File"
[logging]
dir = "/greptimedb_data/logs"
log_format = "json"
[[region_engine]]
[region_engine.mito]
max_concurrent_scan_files = 600
scan_memory_limit = "80%"
scan_memory_on_exhausted = "wait(15s)" The docker compose service has this set to start greptime. command:
- "standalone"
- "start"
- "--config-file"
- "/etc/greptime/custom.toml"
- "--user-provider"
- "static_user_provider:file:/etc/greptime/users"
- "--http-addr"
- "0.0.0.0:4000"
- "--grpc-bind-addr"
- "0.0.0.0:4001"
- "--mysql-addr"
- "0.0.0.0:4002"
- "--postgres-addr"
- "0.0.0.0:4003"Here are the results of that query for table info you gave me. A couple of my data tables are not flat yet, so I'll fix that. I'll check the docs for this in a bit, but what is the merge_mode for? For context, what kind of cpu/ram usage would you expect to see when around 50 vms are sending between 2000 and 5000 rows/second of metrics and logs to greptimedb? When I've had the monitoring sidecar shipping data to the monitorng db actually able to gather the data,, those are the numbers I've seen in the "Total Ingestion Rate" metric on the grafana dashboard you all shared. Also, for shipping data to greptimedb, are there tools that you see do better than other tools? I'm currently using both telegraf and vector, but I'm open to others if they'd do a better job. (I'm using telegraf for metrics, since last a I checked vector doesn't gather docker container specific metrics, and vector for logs, since telegraf's log reading didn't work all that well when I tried it.) As always, thanks for the help figuring this out! |
|
You can check this document for how different merge modes work. |
|
Ok, I think we have made progress. I modified my config file: default_timezone = "America/Los_Angeles"
max_in_flight_write_bytes = "8GB"
## The maximum current queries allowed to be executed. Zero means unlimited.
max_concurrent_queries = 0
## Enable telemetry to collect anonymous usage data. Enabled by default.
enable_telemetry = false
[storage]
data_home = "/greptimedb_data"
type = "File"
[logging]
dir = "/greptimedb_data/logs"
log_format = "json"
[[region_engine]]
[region_engine.mito]
num_workers = 2
max_background_flushes = 1
max_background_compactions = 1
max_background_purges = 1
max_concurrent_scan_files = 600
scan_memory_limit = "80%"
scan_memory_on_exhausted = "wait(15s)"
experimental_compaction_memory_limit = "12GB"
experimental_compaction_on_exhausted = "fail"I'm seeing fewer usage spikes, and when the spike is too large for too long, I see the compaction job canceled. The newest memory profile in the attached zip shows what I currently see when the ram is spiking. |
|
Consolidating labels may not be a problem. We can use some parquet tools to find the columns that have large uncompressed sizes after checking the ssts_manifest table. |
|
Er, where did table_id 1244 come from? I don't actually have a table with that id. The logs_docker_alpha table id is 1068. Anyway I did get the results of the ssts_manifest query for logs_docker_alpha exported to csv. Do you have a place I can upload the file that isn't going to leave it public for who knows how long? I tend to lean heavily on the paranoia side of things when it comes to this kind of data.... |
|
Oh, and looking at my logs, I see a lot of "Failed to merge compaction output for region" messages. As well as some "Failed to acquire memory for compaction" messages. I'm guessing those are from the settings? So the goal of looking up the ssts manifest data is to figure out how to fix things so those limits are not hit? Since I added those settings, greptime has not ran out of memory. So I've been able to doe some visualization of the ssts_manifest. What should I look for? |
|
Sorry, it's 1068. I copied the example from my dashboard. The ssts_manifest result usually doesn't contain sensitive data. You can DM the CSV to me on slack. We can see if it's possible to adjust the table options to avoid hitting the memory limit. |
|
Hi, could you also DM me a zip of the current manifest directory? The manifest directory is under We haven’t exposed a field in the ssts_manifest table, so I have to check the manifest files directly. |
|
We found some files have very large blobs, so it took a lot of memory to compact them. So this is expected. You can keep the compaction memory limit to avoid OOM. We also implemented a fix to skip large tasks early if possible recently. I think it'd be helpful to expose max_row_group_uncompressed_size to the ssts_manifest. |
|
I started this in slack, but it should probably go here:
I found the docs on the topic. Would it help reduce memory usage if I changed the max_output_file_size for the tables with large .parquet files to something smaller? Then could I use the The idea would be that I would set the max size to 25MiB. Then I would run the Is that possible? |
|
Ok, I can't share the actual column names, but the tables with large .parquet files are all from Vector's docker_logs source. The default output of that source adds a column for every label on a container/service. The original table has over 400 label columns. Since I use traefik, there are a lot of columns that have no values except for specific services. This did not make parsing out useful information from the logs very easy. So I used vrl (I think I posted it earlier) to transform those hundreds of labels into json text contained in just 3 columns. This obviously means that the label itself is stored alongside the value when there is a value. All that has me wondering if that is why my memory usage started spiking so high. Is there a way to check how much data is used per row in tables? As in I could find a log entry for a specific service in the old table, and compare it's size to the same kind of entry in the new table. |
|
Only files within a few time spans (UTC) have large row size.
You can inspect the files in the table directory with some parquet tools (e.g. using Python). Here are some files that have very large uncompressed sizes:
The average row size is about 500KB. You can compare them with some of the latest parquet files. Please share your findings with us. |
|
Ok, so, I'm not a python expert, so I used a bit of AI to generate the skeleton of the following script. I then found a few random files from the different docker related tables, plus a list of files from a recent "Failed to merge compaction output..." error log. End results: import pyarrow.parquet as pq
paths = [
"/path/to/db_storage/1068/1068_0000000000/fd5ceb97-70ae-4a6b-86c9-f444625f0ddb.parquet",
"/path/to/db_storage/1068/1068_0000000000/25027428-ae2f-4d30-a925-677ffcef796a.parquet",
"/path/to/db_storage/1068/1068_0000000000/9e4fd839-7c85-426f-b1d9-f5f66a2583e2.parquet",
"/path/to/db_storage/1068/1068_0000000000/108015bc-7590-47ae-a580-f08f31b8a096.parquet",
]
print("---- Known Large Rows in docker_logs_alpha ----")
for path in paths:
pf = pq.ParquetFile(path)
num_rows = pf.metadata.num_rows
total_uncompressed = sum(
pf.metadata.row_group(i).total_byte_size
for i in range(pf.metadata.num_row_groups)
)
avg_row = total_uncompressed / num_rows if num_rows else 0
print(
f"{path}: "
f"rows={num_rows}, "
f"uncompressed={total_uncompressed}, "
f"avg_row={avg_row:.2f}"
)
print("--- Most Recent 4 Files in docker_logs_alpha --- ")
newer_paths = [
"/path/to/db_storage/1068/1068_0000000000/71f53874-96e9-44b2-b089-c13745dbbd93.parquet",
"/path/to/db_storage/1068/1068_0000000000/cbf1b5c8-3c16-448b-9bab-7372641fb2ea.parquet",
"/path/to/db_storage/1068/1068_0000000000/6505ea75-7a2c-4fbb-8d1e-451bcb96a0b8.parquet",
"/path/to/db_storage/1068/1068_0000000000/3af6e3c5-09da-4ed3-8949-8abdbdd6ebbe.parquet"
]
for path in newer_paths:
pf = pq.ParquetFile(path)
num_rows = pf.metadata.num_rows
total_uncompressed = sum(
pf.metadata.row_group(i).total_byte_size
for i in range(pf.metadata.num_row_groups)
)
avg_row = total_uncompressed / num_rows if num_rows else 0
print(
f"{path}: "
f"rows={num_rows}, "
f"uncompressed={total_uncompressed}, "
f"avg_row={avg_row:.2f}"
)
print("--- Most Recent 4 Files in docker_logs --- ")
older_paths = [
"/path/to/db_storage/1025/1025_0000000000/bd800fbf-66a3-46ba-aa86-b2af46635521.parquet",
"/path/to/db_storage/1025/1025_0000000000/d896aad4-a215-48c7-8381-9953a7bf2c75.parquet",
"/path/to/db_storage/1025/1025_0000000000/ec443856-1537-416d-bab6-fc6ff01e7a3d.parquet",
"/path/to/db_storage/1025/1025_0000000000/95e97a76-1889-4868-b843-e579df36eb2e.parquet"
]
for path in older_paths:
pf = pq.ParquetFile(path)
num_rows = pf.metadata.num_rows
total_uncompressed = sum(
pf.metadata.row_group(i).total_byte_size
for i in range(pf.metadata.num_row_groups)
)
avg_row = total_uncompressed / num_rows if num_rows else 0
print(
f"{path}: "
f"rows={num_rows}, "
f"uncompressed={total_uncompressed}, "
f"avg_row={avg_row:.2f}"
)
print("--- Most Recent 4 Files in docker_logs_customapp --- ")
customapp_paths = [
"/path/to/db_storage/1076/1076_0000000000/765648d9-c9b0-4043-92b8-b0655fd35f76.parquet",
"/path/to/db_storage/1076/1076_0000000000/4dc7c17e-8044-4c75-9597-ad2a21c54c89.parquet",
"/path/to/db_storage/1076/1076_0000000000/d7ce70c1-a65f-40f4-96b7-78e975749652.parquet",
"/path/to/db_storage/1076/1076_0000000000/3c04d381-e290-416d-b426-f69a09fd6649.parquet"
]
for path in customapp_paths:
pf = pq.ParquetFile(path)
num_rows = pf.metadata.num_rows
total_uncompressed = sum(
pf.metadata.row_group(i).total_byte_size
for i in range(pf.metadata.num_row_groups)
)
avg_row = total_uncompressed / num_rows if num_rows else 0
print(
f"{path}: "
f"rows={num_rows}, "
f"uncompressed={total_uncompressed}, "
f"avg_row={avg_row:.2f}"
)
print("--- 4 Files in docker_logs_customapp from 7/8 --- ")
customapp_paths = [
"/path/to/db_storage/1076/1076_0000000000/d6629045-b755-45e1-be26-1d489ecd8c59.parquet",
"/path/to/db_storage/1076/1076_0000000000/e6eaab8e-08ae-400d-b5e6-0bb07218efb1.parquet",
"/path/to/db_storage/1076/1076_0000000000/32231fe6-78ba-4d2c-a9fa-d4ab75665faa.parquet",
"/path/to/db_storage/1076/1076_0000000000/3d844908-0055-430d-8b79-7e00d1bbfb8a.parquet"
]
for path in customapp_paths:
pf = pq.ParquetFile(path)
num_rows = pf.metadata.num_rows
total_uncompressed = sum(
pf.metadata.row_group(i).total_byte_size
for i in range(pf.metadata.num_row_groups)
)
avg_row = total_uncompressed / num_rows if num_rows else 0
print(
f"{path}: "
f"rows={num_rows}, "
f"uncompressed={total_uncompressed}, "
f"avg_row={avg_row:.2f}"
)
print("--- Files from a recent failed to merge error log --- ")
recent_compaction_error_paths = [
"/path/to/db_storage/1076/1076_0000000000/8717ab87-3cf2-4b65-9ce2-5698f864a134.parquet",
"/path/to/db_storage/1076/1076_0000000000/159e6361-334e-4eb6-bfa3-84aba9b0a3bc.parquet",
"/path/to/db_storage/1076/1076_0000000000/d82353cd-8213-4ac4-b61d-c7a02bde4a65.parquet",
"/path/to/db_storage/1076/1076_0000000000/542fff44-2589-48a1-bb8f-75106acea1cb.parquet",
"/path/to/db_storage/1076/1076_0000000000/ce453567-70dd-41cb-bde8-8646df592eb9.parquet",
"/path/to/db_storage/1076/1076_0000000000/aee62771-bef0-4cc1-b23b-796317a12639.parquet",
"/path/to/db_storage/1076/1076_0000000000/5542a15a-f2d2-4804-9368-b101afd76463.parquet",
"/path/to/db_storage/1076/1076_0000000000/e46d64c3-0962-48c2-9c80-2b8f00e63e4a.parquet",
"/path/to/db_storage/1076/1076_0000000000/4bd35c1e-668c-4249-b518-99f0d0db8b73.parquet",
"/path/to/db_storage/1076/1076_0000000000/371d341d-f229-47d0-88a5-b820d0c102a4.parquet",
"/path/to/db_storage/1076/1076_0000000000/772b462a-1bf1-43bc-a536-09d0bee8c6de.parquet",
"/path/to/db_storage/1076/1076_0000000000/9192475f-486c-41a0-88bb-8a826002e84e.parquet",
"/path/to/db_storage/1076/1076_0000000000/4600bcaa-a746-4b65-92b3-9ae80cc203db.parquet",
"/path/to/db_storage/1076/1076_0000000000/6a236cf5-4cdb-4618-acd5-603a94592c80.parquet",
"/path/to/db_storage/1076/1076_0000000000/ef520d65-e5fe-4512-b478-6c2620c43c6a.parquet",
"/path/to/db_storage/1076/1076_0000000000/99485395-b56f-48e2-b87f-b30c6e60bed4.parquet",
"/path/to/db_storage/1076/1076_0000000000/a4d47d72-8416-406a-bf08-0973754ca6ae.parquet",
"/path/to/db_storage/1076/1076_0000000000/cbd6ed91-ae9f-490a-bd84-290b22b8ceb8.parquet",
"/path/to/db_storage/1076/1076_0000000000/dd79465d-28ee-4db4-aa31-df671f063805.parquet",
"/path/to/db_storage/1076/1076_0000000000/260ab033-1722-4d85-9691-6abaf1987de8.parquet",
"/path/to/db_storage/1076/1076_0000000000/1a5f506d-b22c-4ec2-a9fc-3f3d7a5b89bc.parquet"
]
for path in recent_compaction_error_paths:
pf = pq.ParquetFile(path)
num_rows = pf.metadata.num_rows
total_uncompressed = sum(
pf.metadata.row_group(i).total_byte_size
for i in range(pf.metadata.num_row_groups)
)
avg_row = total_uncompressed / num_rows if num_rows else 0
print(
f"{path}: "
f"rows={num_rows}, "
f"uncompressed={total_uncompressed}, "
f"avg_row={avg_row:.2f}"
)Script output: File info (ls -al) for the files in the log message: -rw-r--r-- 1 root root 2842063 Jun 24 13:45 /path/to/db_storage/1076/1076_0000000000/8717ab87-3cf2-4b65-9ce2-5698f864a134.parquet
-rw-r--r-- 1 root root 45109849 Jun 24 14:02 /path/to/db_storage/1076/1076_0000000000/159e6361-334e-4eb6-bfa3-84aba9b0a3bc.parquet
-rw-r--r-- 1 root root 6381494 Jun 24 13:45 /path/to/db_storage/1076/1076_0000000000/d82353cd-8213-4ac4-b61d-c7a02bde4a65.parquet
-rw-r--r-- 1 root root 10252679 Jun 24 13:45 /path/to/db_storage/1076/1076_0000000000/542fff44-2589-48a1-bb8f-75106acea1cb.parquet
-rw-r--r-- 1 root root 7425633 Jun 24 13:45 /path/to/db_storage/1076/1076_0000000000/ce453567-70dd-41cb-bde8-8646df592eb9.parquet
-rw-r--r-- 1 root root 6422739 Jun 24 13:45 /path/to/db_storage/1076/1076_0000000000/aee62771-bef0-4cc1-b23b-796317a12639.parquet
-rw-r--r-- 1 root root 7121328 Jun 24 13:45 /path/to/db_storage/1076/1076_0000000000/5542a15a-f2d2-4804-9368-b101afd76463.parquet
-rw-r--r-- 1 root root 18400364 Jun 24 13:45 /path/to/db_storage/1076/1076_0000000000/e46d64c3-0962-48c2-9c80-2b8f00e63e4a.parquet
-rw-r--r-- 1 root root 6211650 Jun 24 13:46 /path/to/db_storage/1076/1076_0000000000/4bd35c1e-668c-4249-b518-99f0d0db8b73.parquet
-rw-r--r-- 1 root root 6269157 Jun 24 13:46 /path/to/db_storage/1076/1076_0000000000/371d341d-f229-47d0-88a5-b820d0c102a4.parquet
-rw-r--r-- 1 root root 6196110 Jun 24 13:46 /path/to/db_storage/1076/1076_0000000000/772b462a-1bf1-43bc-a536-09d0bee8c6de.parquet
-rw-r--r-- 1 root root 6369456 Jun 24 13:46 /path/to/db_storage/1076/1076_0000000000/9192475f-486c-41a0-88bb-8a826002e84e.parquet
-rw-r--r-- 1 root root 5632370 Jun 24 13:46 /path/to/db_storage/1076/1076_0000000000/4600bcaa-a746-4b65-92b3-9ae80cc203db.parquet
-rw-r--r-- 1 root root 15954293 Jun 24 13:47 /path/to/db_storage/1076/1076_0000000000/6a236cf5-4cdb-4618-acd5-603a94592c80.parquet
-rw-r--r-- 1 root root 18199800 Jun 24 13:47 /path/to/db_storage/1076/1076_0000000000/ef520d65-e5fe-4512-b478-6c2620c43c6a.parquet
-rw-r--r-- 1 root root 7177937 Jun 24 13:47 /path/to/db_storage/1076/1076_0000000000/99485395-b56f-48e2-b87f-b30c6e60bed4.parquet
-rw-r--r-- 1 root root 7365241 Jun 24 13:47 /path/to/db_storage/1076/1076_0000000000/a4d47d72-8416-406a-bf08-0973754ca6ae.parquet
-rw-r--r-- 1 root root 11443635 Jun 24 13:49 /path/to/db_storage/1076/1076_0000000000/cbd6ed91-ae9f-490a-bd84-290b22b8ceb8.parquet
-rw-r--r-- 1 root root 28286103 Jun 24 13:50 /path/to/db_storage/1076/1076_0000000000/dd79465d-28ee-4db4-aa31-df671f063805.parquet
-rw-r--r-- 1 root root 15743225 Jun 24 13:50 /path/to/db_storage/1076/1076_0000000000/260ab033-1722-4d85-9691-6abaf1987de8.parquet
-rw-r--r-- 1 root root 20739410 Jun 24 13:53 /path/to/db_storage/1076/1076_0000000000/1a5f506d-b22c-4ec2-a9fc-3f3d7a5b89bc.parquetThe error log: {
"timestamp": "2026-07-21T16:45:14.488083Z",
"level": "WARN",
"fields":
{
"message": "Failed to merge compaction output for region: 4621384810496(1076, 0), inputs: [8717ab87-3cf2-4b65-9ce2-5698f864a134,159e6361-334e-4eb6-bfa3-84aba9b0a3bc,d82353cd-8213-4ac4-b61d-c7a02bde4a65,542fff44-2589-48a1-bb8f-75106acea1cb,ce453567-70dd-41cb-bde8-8646df592eb9,aee62771-bef0-4cc1-b23b-796317a12639,5542a15a-f2d2-4804-9368-b101afd76463,e46d64c3-0962-48c2-9c80-2b8f00e63e4a,4bd35c1e-668c-4249-b518-99f0d0db8b73,371d341d-f229-47d0-88a5-b820d0c102a4,772b462a-1bf1-43bc-a536-09d0bee8c6de,9192475f-486c-41a0-88bb-8a826002e84e,4600bcaa-a746-4b65-92b3-9ae80cc203db,6a236cf5-4cdb-4618-acd5-603a94592c80,ef520d65-e5fe-4512-b478-6c2620c43c6a,99485395-b56f-48e2-b87f-b30c6e60bed4,a4d47d72-8416-406a-bf08-0973754ca6ae,cbd6ed91-ae9f-490a-bd84-290b22b8ceb8,dd79465d-28ee-4db4-aa31-df671f063805,260ab033-1722-4d85-9691-6abaf1987de8,1a5f506d-b22c-4ec2-a9fc-3f3d7a5b89bc]",
"err": "0: Failed to compute arrow arrays, at src/mito2/src/read/flat_merge.rs:87:22\n1: OffsetOverflowError(2150538869)"
},
"target": "mito2::compaction::compactor"
}So, looking at all that, is my current data just fine, but my older data, where the sizes are so much larger, is never going to get properly compacted? |
|
You can also inspect the large data if you want to know what the content is.
Yes. But the uncompressed data is already large enough, so it is fine not to compact it. You can ignore these warnings. |
Uh oh!
There was an error while loading. Please reload this page.
I finally have time to work on my Greptime stuff again. Specifically I need to pull out log data about a service so we can figure out what to do with it.
The issue is that as soon as I start querying more than 2 days of data, the query times out. Since I do keep running into this issue when examining other data as well, I'd like to learn how to optimize/fix/etc whatever I need to in order to do what I want.
Setup is greptime/greptimedb:v1.0.0-rc.1 docker image running on a dedicated vm with 24gb of ram and 4 vcpu.
Storage is local pointed at an iscsi volume living on a flash array.
Grafana is 12.3 via a container on a different host. Both vms/hosts are in the same network/hypervisor.
I have a traefik proxy instance in front of GreptimeDB.
In grafana, my queries are:
I just added indexes on a couple columns, and that helped a bit. It wasn't working for more than a few minutes worth of data before, and after it started working up to 2 days worth.
I am applying some transformations in grafana, but my timeout errors don't occur via my browser crashing, so it shouldn't be the processing of those, right?
Anyone have any tips?
All reactions