Implement Write-ahead Logging #261

HHoflittlefish777 · 2023-03-19T04:14:29Z

HHoflittlefish777 · 2023-03-19T04:19:52Z

There are some things that need help:

During disk flushing, in order to avoid blocking writes, I cloned the buffer so that I can continue to accept write requests while flushing the disk. However, during asynchronous callbacks, all channels will be notified. At this time, write requests that only write to the buffer should not receive this request, but they receive it.how can I resolve this issue？

codecov-commenter · 2023-03-19T04:20:35Z

Codecov Report

Merging #261 (56ae9db) into main (3c3ef3c) will decrease coverage by 4.32%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #261      +/-   ##
==========================================
- Coverage   44.25%   39.94%   -4.32%     
==========================================
  Files          93      100       +7     
  Lines        9814    10868    +1054     
==========================================
- Hits         4343     4341       -2     
- Misses       5053     6109    +1056     
  Partials      418      418

Impacted Files	Coverage Δ
pkg/run/channel_closer.go	`0.00% <0.00%> (ø)`
pkg/run/channel_group_closer.go	`0.00% <0.00%> (ø)`
pkg/wal/wal.go	`0.00% <0.00%> (ø)`

... and 5 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

hanahmily · 2023-03-20T00:09:47Z

There are some things that need help:

During disk flushing, in order to avoid blocking writes, I cloned the buffer so that I can continue to accept write requests while flushing the disk. However, during asynchronous callbacks, all channels will be notified. At this time, write requests that only write to the buffer should not receive this request, but they receive it.how can I resolve this issue？

You should have switched to a new flushChannel on switching the buffer. Theoretically, a flush channel sends a notice which indicates its corresponding buffer gets flushed. Based on that, the flush channel seems a field in the buffer structure instead.

hanahmily · 2023-03-20T00:10:22Z

@HHoflittlefish777 Please fix test failures.

…b into wal

HHoflittlefish777 · 2023-03-20T07:11:08Z

There are some things that need help:

During disk flushing, in order to avoid blocking writes, I cloned the buffer so that I can continue to accept write requests while flushing the disk. However, during asynchronous callbacks, all channels will be notified. At this time, write requests that only write to the buffer should not receive this request, but they receive it.how can I resolve this issue？

You should have switched to a new flushChannel on switching the buffer. Theoretically, a flush channel sends a notice which indicates its corresponding buffer gets flushed. Based on that, the flush channel seems a field in the buffer structure instead.

I have two channel field in the Log struct, should I move them into buffer struct?

hanahmily · 2023-03-23T00:31:20Z

I have two channel field in the Log struct, should I move them into buffer struct?

Make sense to me. I submitted the Buffer patch(#262), in which you could find a similar structure. I hope you could get some inspiration from it.

pkg/wal/wal.go

hanahmily

I took a quick look at the implementation. There are several channels to sync operations. The std testing framework can't handle this pattern appropriately. I recommend :

Using https://github.com/onsi/ginkgo as other packages to test async operations.
https://github.com/uber-go/goleak will help check the goroutine leaks

There is a comprehensive test case which you could get some inspiration from:
https://github.com/apache/skywalking-banyandb/blob/359fd2a84a525351a8fc0e2313b31f7e601f88e7/banyand/tsdb/buffer_test.go

pkg/wal/wal_test.go

…b into wal

pkg/wal/wal_test.go

pkg/wal/wal.go

hanahmily · 2023-04-03T13:09:14Z

pkg/wal/wal.go

+			timer := time.NewTimer(bufferBatchInterval * time.Millisecond)
+			select {
+			case request := <-log.writeChannel:
+				bufferSize += seriesIDLength + timestampLength + len(request.data)


The length of seriesID is not fixed.

hanahmily · 2023-04-03T13:20:30Z

pkg/wal/wal.go

+					buf := log.buffer
+					// Clear buffer to receive Log request.
+					log.newBuffer()
+					if log.closeFlag {


You can't archive synchronization through a local varable. To safely share data between goroutines is by using channel or context.

hanahmily · 2023-04-03T13:34:54Z

pkg/wal/wal.go

+	}()
+}
+
+func (log *Log) asyncBatchflush(buffer buffer, flushCh chan string) {


Suggested change

func (log *Log) asyncBatchflush(buffer buffer, flushCh chan string) {

func (log *Log) asyncBatchFlush(buffer buffer, flushCh chan string) {

hanahmily · 2023-04-03T13:47:49Z

pkg/wal/wal.go

@@ -43,18 +128,450 @@ type WAL interface {
 	// Write a logging entity.
 	// It will return immediately when the data is written in the buffer,
 	// The returned function will be called when the entity is flushed on the persistent storage.
-	Write(seriesID common.SeriesID, timestamp time.Time, data []byte) (func(), error)
+	Write(seriesID []byte, timestamp time.Time, data []byte) error


Why do you drop off the returned function to get flushed event?

hanahmily · 2023-04-03T14:25:20Z

pkg/wal/wal.go

+		}
+
+		// Compression and flush.
+		compressionData := snappy.Encode(nil, bytesBuffer.Bytes())


Instead of compressing the entire segment, it is recommended to separately encode/compress timestamps and values.

The pkg/encoding/xor package offers methods for encoding uint64. These can be used to encode/decode the UNIX epoch (nanosecond) of a timestamp.

For quick compression of values, snappy can be utilized.

By following this approach, there is no need for bytesBuffer during flushing. Instead, data can be written directly to workSegment.file.

wu-sheng · 2023-05-07T01:38:40Z

This PR seems not to update for weeks. Is this still being processed?

hanahmily · 2023-05-07T02:50:36Z

This PR seems not to update for weeks. Is this still being processed

Certainly, I will processed with it later, after completing load testing.

hailin0 · 2023-05-17T02:53:52Z

I want to try to continue, Thanks

HHoflittlefish777 · 2023-05-18T03:03:01Z

@hailin0 Thanks for you interest in it, but WAL doesn't have much work to continue with, just fix the review. There is still a lot of work to be done in BanyanDB, and you can do some other work you like.

hanahmily · 2023-05-18T10:09:47Z

@hailin0 Please address the issues outlined in the review. If you encounter any difficulties or have questions, feel free to reach out for assistance.

@HHoflittlefish777
The remaining tasks will be assigned to @hailin0, who is interested in contributing to the project. Completing this PR will serve as an excellent introduction for him to become familiar with the project.

HHoflittlefish777 · 2023-05-18T14:45:21Z

That is fine, Looking forward to your further contributions to BanyanDB @hailin0.

hailin0 · 2023-05-21T13:04:03Z

thanks, i am doing

hanahmily · 2023-06-07T03:32:28Z

@hailin0 any update?

…-banyandb into pr/261

pkg/run/channel_closer.go

pkg/run/channel_closer_test.go

…-banyandb into pr/261

hanahmily

Would you add benchmark to the write and rotate operations?

CHANGES.md

api/common/id.go

hailin0 · 2023-07-06T14:47:43Z

Would you add benchmark to the write and rotate operations?

Yes, I would add benchmark testing

…-banyandb into pr/261

hailin0 · 2023-07-16T16:48:20Z

mem

cpu

profile.zip

hanahmily

LGTM. Thank you to everyone who participated.

Implement Write-ahead Logging

d6030fe

Merge branch 'main' into wal

cb7cff8

HHoflittlefish777 and others added 3 commits March 20, 2023 07:02

fix lint

022980f

Merge branch 'apache:main' into wal

110fc04

Merge branch 'wal' of github.com:HHoflittlefish777/skywalking-banyand…

fd5c06e

…b into wal

HHoflittlefish777 added 2 commits March 20, 2023 09:52

fix test

e6208a4

fix format

35ab556

hanahmily requested changes Mar 23, 2023

View reviewed changes

pkg/wal/wal.go Outdated Show resolved Hide resolved

HHoflittlefish777 and others added 2 commits March 24, 2023 10:59

fix review

d3bd85a

Merge branch 'main' into wal

359fd2a

hanahmily requested changes Mar 27, 2023

View reviewed changes

HHoflittlefish777 added 2 commits April 2, 2023 07:05

test:use ginkgo for test

ddd6aca

Merge branch 'wal' of github.com:HHoflittlefish777/skywalking-banyand…

14cd31e

…b into wal

hanahmily requested changes Apr 3, 2023

View reviewed changes

Merge branch 'main' into wal

e2dbebe

Merge branch 'main' into wal

dd08984

hailin0 and others added 3 commits July 2, 2023 22:19

Fix review & add channel closer

8eef779

Merge branch 'wal' of https://github.com/HHoflittlefish777/skywalking…

f0d21a4

…-banyandb into pr/261

Merge branch 'main' into wal

22b336f

hanahmily requested changes Jul 3, 2023

View reviewed changes

pkg/run/channel_closer.go Outdated Show resolved Hide resolved

pkg/run/channel_closer_test.go Show resolved Hide resolved

hailin0 and others added 7 commits July 3, 2023 19:02

fix channel closer

6c746e7

Merge branch 'wal' of https://github.com/HHoflittlefish777/skywalking…

aa1c414

…-banyandb into pr/261

fix channel closer

a1ab7f0

add channel group closer

d2f5a5d

fix channel closer

4edbd57

update changelog

2b56073

Merge branch 'main' into wal

4762af1

hanahmily requested changes Jul 5, 2023

View reviewed changes

CHANGES.md Outdated Show resolved Hide resolved

api/common/id.go Outdated Show resolved Hide resolved

hailin0 added 3 commits July 6, 2023 22:01

Rename SeriesIDV2 to GlobalSeriesID

c86be30

Merge branch 'main' into wal

2b30841

fix changelog

3082256

hailin0 and others added 4 commits July 9, 2023 10:07

Merge branch 'main' into wal

b052855

add file sync option

f33d582

Merge branch 'wal' of https://github.com/HHoflittlefish777/skywalking…

39b64b0

…-banyandb into pr/261

Merge branch 'main' into wal

96b9a89

hailin0 added 7 commits July 17, 2023 00:53

Add benchmark

795af78

fix

4e22a1f

fix lint

50d9efa

fix lint

2baba3f

fix lint

63bfdfe

fix ci check

56ae9db

add readme

eb74615

hanahmily approved these changes Jul 17, 2023

View reviewed changes

hanahmily merged commit 482831f into apache:main Jul 17, 2023
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Write-ahead Logging #261

Implement Write-ahead Logging #261

HHoflittlefish777 commented Mar 19, 2023

HHoflittlefish777 commented Mar 19, 2023 •

edited

codecov-commenter commented Mar 19, 2023 •

edited

hanahmily commented Mar 20, 2023

hanahmily commented Mar 20, 2023

HHoflittlefish777 commented Mar 20, 2023 •

edited

hanahmily commented Mar 23, 2023

hanahmily left a comment

hanahmily Apr 3, 2023

hanahmily Apr 3, 2023

hanahmily Apr 3, 2023

hanahmily Apr 3, 2023

hanahmily Apr 3, 2023

wu-sheng commented May 7, 2023

hanahmily commented May 7, 2023

hailin0 commented May 17, 2023

HHoflittlefish777 commented May 18, 2023

hanahmily commented May 18, 2023

HHoflittlefish777 commented May 18, 2023

hailin0 commented May 21, 2023

hanahmily commented Jun 7, 2023

hanahmily left a comment

hailin0 commented Jul 6, 2023

hailin0 commented Jul 16, 2023 •

edited

hanahmily left a comment

	func (log *Log) asyncBatchflush(buffer buffer, flushCh chan string) {
	func (log *Log) asyncBatchFlush(buffer buffer, flushCh chan string) {

Implement Write-ahead Logging #261

Implement Write-ahead Logging #261

Conversation

HHoflittlefish777 commented Mar 19, 2023

HHoflittlefish777 commented Mar 19, 2023 • edited

codecov-commenter commented Mar 19, 2023 • edited

Codecov Report

hanahmily commented Mar 20, 2023

hanahmily commented Mar 20, 2023

HHoflittlefish777 commented Mar 20, 2023 • edited

hanahmily commented Mar 23, 2023

hanahmily left a comment

Choose a reason for hiding this comment

hanahmily Apr 3, 2023

Choose a reason for hiding this comment

hanahmily Apr 3, 2023

Choose a reason for hiding this comment

hanahmily Apr 3, 2023

Choose a reason for hiding this comment

hanahmily Apr 3, 2023

Choose a reason for hiding this comment

hanahmily Apr 3, 2023

Choose a reason for hiding this comment

wu-sheng commented May 7, 2023

hanahmily commented May 7, 2023

hailin0 commented May 17, 2023

HHoflittlefish777 commented May 18, 2023

hanahmily commented May 18, 2023

HHoflittlefish777 commented May 18, 2023

hailin0 commented May 21, 2023

hanahmily commented Jun 7, 2023

hanahmily left a comment

Choose a reason for hiding this comment

hailin0 commented Jul 6, 2023

hailin0 commented Jul 16, 2023 • edited

hanahmily left a comment

Choose a reason for hiding this comment

HHoflittlefish777 commented Mar 19, 2023 •

edited

codecov-commenter commented Mar 19, 2023 •

edited

HHoflittlefish777 commented Mar 20, 2023 •

edited

hailin0 commented Jul 16, 2023 •

edited