Skip to content

[Bug] [BanyanDB] Write data error after updated Group schema #12871

@wankai123

Description

@wankai123

Search before asking

  • I had searched in the issues and found no similar issues.

Apache SkyWalking Component

BanyanDB (apache/skywalking-banyandb)

What happened

When after updated Group schema to:

    gmMinuteShardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_SHARD_NUM:3}
    gmMinuteSIDays: ${SW_STORAGE_BANYANDB_GM_MINUTE_SI_DAYS:2}
    gmMinuteTTLDays: ${SW_STORAGE_BANYANDB_GM_MINUTE_TTL_DAYS:10}

OAP can't write new measure data into the DB and read old data, throw errors:

2024-12-17 08:39:43,716 org.apache.skywalking.banyandb.v1.client.MeasureBulkWriteProcessor 93 [grpc-default-executor-0] ERROR [] - [10.2.0-SNAPSHOT-3223a2c] Error occurs in flushing measures
io.grpc.StatusRuntimeException: INTERNAL: directory is exist
	at io.grpc.Status.asRuntimeException(Status.java:532) ~[grpc-api-1.68.1.jar:1.68.1]
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) [grpc-stub-1.68.1.jar:1.68.1]
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) [grpc-api-1.68.1.jar:1.68.1]
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) [grpc-api-1.68.1.jar:1.68.1]
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) [grpc-api-1.68.1.jar:1.68.1]
	at org.apache.skywalking.banyandb.v1.client.grpc.channel.ChannelManager$NetworkExceptionAwareClientCall$1.onClose(ChannelManager.java:211) [banyandb-java-client-0.8.0-rc0.jar:0.8.0-rc0]
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564) [grpc-core-1.68.1.jar:1.68.1]
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72) [grpc-core-1.68.1.jar:1.68.1]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729) [grpc-core-1.68.1.jar:1.68.1]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710) [grpc-core-1.68.1.jar:1.68.1]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.68.1.jar:1.68.1]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.68.1.jar:1.68.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

The BanyanDB server also throw errors:

2024-12-17 08:37:20 {"level":"error","module":"METRIC-SERVICE.SCHEDULER.METRICS-COLLECTOR","name":"metrics-collector","panic":"invalid memory address or nil pointer dereference","stack":"goroutine 3221 [running]:\nruntime/debug.Stack()\n\t/opt/hostedtoolcache/go/1.23.3/x64/src/runtime/debug/stack.go:26 +0x5e\ngithub.com/apache/skywalking-banyandb/pkg/timestamp.(*task).run.func1.1()\n\t/home/runner/work/skywalking-banyandb/skywalking-banyandb/pkg/timestamp/scheduler.go:203 +0xd3\npanic({0x14812e0?, 0x26f8fe0?})\n\t/opt/hostedtoolcache/go/1.23.3/x64/src/runtime/panic.go:785 +0x132\ngithub.com/apache/skywalking-banyandb/banyand/measure.(*snapshot).decRef(0xc00335c700?)\n\t/home/runner/work/skywalking-banyandb/skywalking-banyandb/banyand/measure/snapshot.go:73 +0x17\npanic({0x14812e0?, 0x26f8fe0?})\n\t/opt/hostedtoolcache/go/1.23.3/x64/src/runtime/panic.go:785 +0x132\ngithub.com/apache/skywalking-banyandb/banyand/measure.(*tsTable).Collect(0xc0000aaee0, {0x1b42ca0?, 0xc000b13800})\n\t/home/runner/work/skywalking-banyandb/skywalking-banyandb/banyand/measure/metrics.go:300 +0x9a\ngithub.com/apache/skywalking-banyandb/banyand/internal/storage.(*database[...]).collect(0x1b76e80)\n\t/home/runner/work/skywalking-banyandb/skywalking-banyandb/banyand/internal/storage/tsdb.go:163 +0x2ea\ngithub.com/apache/skywalking-banyandb/banyand/observability.(*Collector).collect(0x2?)\n\t/home/runner/work/skywalking-banyandb/skywalking-banyandb/banyand/observability/collector.go:56 +0xd4\ngithub.com/apache/skywalking-banyandb/banyand/observability.(*metricService).Serve.func1({0xc004ea9810?, 0xc003556070?, 0xc004ea9810?}, 0xc0054b5f00?)\n\t/home/runner/work/skywalking-banyandb/skywalking-banyandb/banyand/observability/service.go:152 +0x2f\ngithub.com/apache/skywalking-banyandb/pkg/timestamp.(*task).run.func1(0xc001848460, 0xc0054b5ec8)\n\t/home/runner/work/skywalking-banyandb/skywalking-banyandb/pkg/timestamp/scheduler.go:208 +0xaf\ngithub.com/apache/skywalking-banyandb/pkg/timestamp.(*task).run(0xc001848460)\n\t/home/runner/work/skywalking-banyandb/skywalking-banyandb/pkg/timestamp/scheduler.go:209 +0x3f0\ngithub.com/apache/skywalking-banyandb/pkg/timestamp.(*Scheduler).Register.func1()\n\t/home/runner/work/skywalking-banyandb/skywalking-banyandb/pkg/timestamp/scheduler.go:98 +0x47\ncreated by github.com/apache/skywalking-banyandb/pkg/timestamp.(*Scheduler).Register in goroutine 3204\n\t/home/runner/work/skywalking-banyandb/skywalking-banyandb/pkg/timestamp/scheduler.go:97 +0x5f1\n","time":"2024-12-17T00:37:20Z","message":"panic"}
2024-12-17 08:37:33 {"level":"info","module":"MEASURE","group":"minute","time":"2024-12-17T00:37:33Z","message":"closing the previous tsdb"}
2024-12-17 08:37:33 {"level":"info","module":"MINUTE.SCHEDULER.RETENTION","name":"retention","time":"2024-12-17T00:37:33Z","message":"closed"}
2024-12-17 08:37:33 {"level":"info","module":"MEASURE","group":"minute","time":"2024-12-17T00:37:33Z","message":"creating a new tsdb"}
2024-12-17 08:37:33 {"level":"warn","module":"MEASURE","err":{"ExistingCollector":{},"NewCollector":{}},"time":"2024-12-17T00:37:33Z","message":"watching the events"}
2024-12-17 08:39:43 {"level":"panic","module":"STORAGE.FILESYSTEM","path":"/tmp/measure-data/measure/minute/seg-20241217","time":"2024-12-17T00:39:43Z","message":"directory is exist"}

What you expected to happen

Can read and write data after updating Group schema.

How to reproduce

  1. Write measure data
  2. Update following Group schema:
    gmMinuteShardNum: ${SW_STORAGE_BANYANDB_GM_MINUTE_SHARD_NUM:3}
    gmMinuteSIDays: ${SW_STORAGE_BANYANDB_GM_MINUTE_SI_DAYS:2}
    gmMinuteTTLDays: ${SW_STORAGE_BANYANDB_GM_MINUTE_TTL_DAYS:10}
  1. Write/Read measure data

Anything else

No response

Are you willing to submit a pull request to fix on your own?

  • Yes I am willing to submit a pull request on my own!

Code of Conduct

Metadata

Metadata

Assignees

Labels

bugSomething isn't working and you are sure it's a bug!databaseBanyanDB - SkyWalking native database

Type

No type
No fields configured for issues without a type.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions