Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf(common): Optimize null_bitmap with Set64 #8941

Merged
merged 28 commits into from Apr 6, 2023
Merged

perf(common): Optimize null_bitmap with Set64 #8941

merged 28 commits into from Apr 6, 2023

Conversation

kwannoel
Copy link
Contributor

@kwannoel kwannoel commented Apr 3, 2023 •
edited

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

This Pr uses Set64 to optimize null bitmap to avoid overhead of vector.

Breaking change: Hard limit of 64 group keys now. Breaking change fixed in next PR: #9026

#8802

Checklist For Contributors

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • I have demonstrated that backward compatibility is not broken by breaking changes and created issues to track deprecated features to be removed in the future. (Please refer to the issue)
  • All checks passed in ./risedev check (or alias, ./risedev c)

Checklist For Reviewers

  • I have requested macro/micro-benchmarks as this PR can affect performance substantially, and the results are shown.

Documentation

  • My PR DOES NOT contain user-facing changes.
Click here for Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove the others.

  • Installation and deployment
  • Connector (sources & sinks)
  • SQL commands, functions, and operators
  • RisingWave cluster configuration changes
  • Other (please specify in the release note below)

Release note

@kwannoel

This comment was marked as outdated.

Sign in to view
st1page
st1page reviewed Apr 3, 2023
View reviewed changes
Copy link
Contributor

@st1page st1page left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I think it should not influence the performance 😕

src/common/src/hash/key.rs Outdated Show resolved Hide resolved
@kwannoel kwannoel force-pushed the kwannoel/optimize-bitmap-u8 branch from b68d133 to 9d76adf Compare April 3, 2023 08:30
@kwannoel

This comment was marked as outdated.

Sign in to view
@kwannoel

This comment was marked as outdated.

Sign in to view
@kwannoel kwannoel force-pushed the kwannoel/optimize-bitmap-u8 branch from d519b07 to f750b08 Compare April 5, 2023 05:03
@kwannoel
Copy link
Contributor Author

kwannoel commented Apr 5, 2023 •
edited

Latest bench of 1024 rows, vector serialization, has performance across the board. I guess I was benchmarking on wrong commit...

Here's the breakdown of improvements:

  • Mixed: ~15%
  • KeySerialized: ~15%
  • Fixed size key: ~70%.

I think it's good to go. Future PR can optimize further. The Mixed case seems to be forced to use KeySerialized. Perhaps we can come up with a better key encoding for it.

Additionally I have no idea why medium case, which has 8 bigint keys uses KeySerialized rather than Key64. Will figure that out separately too.

Full


> cargo bench --bench bench_hash_key_encoding -- "1024 rows, vec ser" 1>main-results.log 2>&1

   Compiling risingwave_common v1.0.0-alpha (/Users/noelkwan/projects/risingwave/src/common)
    Finished bench [optimized] target(s) in 24.10s
     Running benches/bench_hash_key_encoding.rs (target/release/deps/bench_hash_key_encoding-fbf1b9e7c3ee200e)
Benchmarking vec ser 1024 rows, Serial Key64, Pr[null]=0
Benchmarking vec ser 1024 rows, Serial Key64, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, Serial Key64, Pr[null]=0: Collecting 100 samples in estimated 5.0494 s (490k iterations)
Benchmarking vec ser 1024 rows, Serial Key64, Pr[null]=0: Analyzing
vec ser 1024 rows, Serial Key64, Pr[null]=0
                        time:   [10.442 µs 10.492 µs 10.545 µs]
                        change: [-71.624% -71.453% -71.289%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Benchmarking vec ser 1024 rows, Serial Key64, Pr[null]=0.01
Benchmarking vec ser 1024 rows, Serial Key64, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, Serial Key64, Pr[null]=0.01: Collecting 100 samples in estimated 5.0214 s (485k iterations)
Benchmarking vec ser 1024 rows, Serial Key64, Pr[null]=0.01: Analyzing
vec ser 1024 rows, Serial Key64, Pr[null]=0.01
                        time:   [10.284 µs 10.310 µs 10.338 µs]
                        change: [-72.985% -72.823% -72.654%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild

Benchmarking vec ser 1024 rows, Serial Key64, Pr[null]=0.1
Benchmarking vec ser 1024 rows, Serial Key64, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, Serial Key64, Pr[null]=0.1: Collecting 100 samples in estimated 5.0065 s (485k iterations)
Benchmarking vec ser 1024 rows, Serial Key64, Pr[null]=0.1: Analyzing
vec ser 1024 rows, Serial Key64, Pr[null]=0.1
                        time:   [10.257 µs 10.283 µs 10.312 µs]
                        change: [-73.100% -72.915% -72.726%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

Benchmarking vec ser 1024 rows, int32 Key32, Pr[null]=0
Benchmarking vec ser 1024 rows, int32 Key32, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, int32 Key32, Pr[null]=0: Collecting 100 samples in estimated 5.0186 s (722k iterations)
Benchmarking vec ser 1024 rows, int32 Key32, Pr[null]=0: Analyzing
vec ser 1024 rows, int32 Key32, Pr[null]=0
                        time:   [6.9406 µs 6.9467 µs 6.9530 µs]
                        change: [-80.854% -80.794% -80.723%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) high mild
  5 (5.00%) high severe

Benchmarking vec ser 1024 rows, int32 Key32, Pr[null]=0.01
Benchmarking vec ser 1024 rows, int32 Key32, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, int32 Key32, Pr[null]=0.01: Collecting 100 samples in estimated 5.0002 s (712k iterations)
Benchmarking vec ser 1024 rows, int32 Key32, Pr[null]=0.01: Analyzing
vec ser 1024 rows, int32 Key32, Pr[null]=0.01
                        time:   [7.0223 µs 7.0288 µs 7.0356 µs]
                        change: [-80.121% -80.004% -79.881%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) high mild
  4 (4.00%) high severe

Benchmarking vec ser 1024 rows, int32 Key32, Pr[null]=0.1
Benchmarking vec ser 1024 rows, int32 Key32, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, int32 Key32, Pr[null]=0.1: Collecting 100 samples in estimated 5.0104 s (656k iterations)
Benchmarking vec ser 1024 rows, int32 Key32, Pr[null]=0.1: Analyzing
vec ser 1024 rows, int32 Key32, Pr[null]=0.1
                        time:   [7.6363 µs 7.6419 µs 7.6476 µs]
                        change: [-78.889% -78.722% -78.546%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  7 (7.00%) high mild
  1 (1.00%) high severe

Benchmarking vec ser 1024 rows, int64 Key64, Pr[null]=0
Benchmarking vec ser 1024 rows, int64 Key64, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, int64 Key64, Pr[null]=0: Collecting 100 samples in estimated 5.0358 s (490k iterations)
Benchmarking vec ser 1024 rows, int64 Key64, Pr[null]=0: Analyzing
vec ser 1024 rows, int64 Key64, Pr[null]=0
                        time:   [10.245 µs 10.276 µs 10.309 µs]
                        change: [-73.574% -73.401% -73.215%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Benchmarking vec ser 1024 rows, int64 Key64, Pr[null]=0.01
Benchmarking vec ser 1024 rows, int64 Key64, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, int64 Key64, Pr[null]=0.01: Collecting 100 samples in estimated 5.0318 s (490k iterations)
Benchmarking vec ser 1024 rows, int64 Key64, Pr[null]=0.01: Analyzing
vec ser 1024 rows, int64 Key64, Pr[null]=0.01
                        time:   [10.260 µs 10.287 µs 10.316 µs]
                        change: [-74.072% -73.993% -73.913%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Benchmarking vec ser 1024 rows, int64 Key64, Pr[null]=0.1
Benchmarking vec ser 1024 rows, int64 Key64, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, int64 Key64, Pr[null]=0.1: Collecting 100 samples in estimated 5.0117 s (485k iterations)
Benchmarking vec ser 1024 rows, int64 Key64, Pr[null]=0.1: Analyzing
vec ser 1024 rows, int64 Key64, Pr[null]=0.1
                        time:   [10.274 µs 10.298 µs 10.324 µs]
                        change: [-74.242% -74.182% -74.117%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

Benchmarking vec ser 1024 rows, varchar KeySerialized, Pr[null]=0
Benchmarking vec ser 1024 rows, varchar KeySerialized, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, varchar KeySerialized, Pr[null]=0: Collecting 100 samples in estimated 5.4220 s (35k iterations)
Benchmarking vec ser 1024 rows, varchar KeySerialized, Pr[null]=0: Analyzing
vec ser 1024 rows, varchar KeySerialized, Pr[null]=0
                        time:   [152.58 µs 152.85 µs 153.13 µs]
                        change: [-14.529% -14.352% -14.183%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Benchmarking vec ser 1024 rows, varchar KeySerialized, Pr[null]=0.01
Benchmarking vec ser 1024 rows, varchar KeySerialized, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, varchar KeySerialized, Pr[null]=0.01: Collecting 100 samples in estimated 5.4057 s (40k iterations)
Benchmarking vec ser 1024 rows, varchar KeySerialized, Pr[null]=0.01: Analyzing
vec ser 1024 rows, varchar KeySerialized, Pr[null]=0.01
                        time:   [133.66 µs 134.03 µs 134.39 µs]
                        change: [-16.864% -16.659% -16.430%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  7 (7.00%) low mild

Benchmarking vec ser 1024 rows, varchar KeySerialized, Pr[null]=0.1
Benchmarking vec ser 1024 rows, varchar KeySerialized, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, varchar KeySerialized, Pr[null]=0.1: Collecting 100 samples in estimated 5.3180 s (45k iterations)
Benchmarking vec ser 1024 rows, varchar KeySerialized, Pr[null]=0.1: Analyzing
vec ser 1024 rows, varchar KeySerialized, Pr[null]=0.1
                        time:   [116.45 µs 116.81 µs 117.14 µs]
                        change: [-18.481% -18.189% -17.894%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) low mild
  4 (4.00%) high mild

Benchmarking vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0
Benchmarking vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0: Collecting 100 samples in estimated 5.3424 s (20k iterations)
Benchmarking vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0: Analyzing
vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0
                        time:   [264.28 µs 264.74 µs 265.23 µs]
                        change: [-8.2031% -7.9638% -7.7300%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  1 (1.00%) high severe

Benchmarking vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0.01
Benchmarking vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0.01: Collecting 100 samples in estimated 5.7194 s (25k iterations)
Benchmarking vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0.01: Analyzing
vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0.01
                        time:   [225.86 µs 226.90 µs 228.17 µs]
                        change: [-10.960% -10.589% -10.184%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

Benchmarking vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0.1
Benchmarking vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0.1: Collecting 100 samples in estimated 5.8698 s (30k iterations)
Benchmarking vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0.1: Analyzing
vec ser 1024 rows, composite varchar KeySerialized, Pr[null]=0.1
                        time:   [192.87 µs 193.57 µs 194.25 µs]
                        change: [-12.012% -11.734% -11.451%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0
Benchmarking vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0: Collecting 100 samples in estimated 5.0597 s (293k iterations)
Benchmarking vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0: Analyzing
vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0
                        time:   [17.230 µs 17.276 µs 17.324 µs]

Benchmarking vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0.01
Benchmarking vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0.01: Collecting 100 samples in estimated 5.0536 s (288k iterations)
Benchmarking vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0.01: Analyzing
vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0.01
                        time:   [17.490 µs 17.533 µs 17.577 µs]

Benchmarking vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0.1
Benchmarking vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0.1: Collecting 100 samples in estimated 5.0743 s (263k iterations)
Benchmarking vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0.1: Analyzing
vec ser 1024 rows, composite fixed, case 1 Key128, Pr[null]=0.1
                        time:   [19.260 µs 19.303 µs 19.346 µs]

Benchmarking vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0
Benchmarking vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0: Collecting 100 samples in estimated 5.1023 s (247k iterations)
Benchmarking vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0: Analyzing
vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0
                        time:   [20.579 µs 20.622 µs 20.667 µs]

Benchmarking vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0.01
Benchmarking vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0.01: Collecting 100 samples in estimated 5.0442 s (242k iterations)
Benchmarking vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0.01: Analyzing
vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0.01
                        time:   [20.750 µs 20.803 µs 20.857 µs]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Benchmarking vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0.1
Benchmarking vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0.1: Collecting 100 samples in estimated 5.0882 s (232k iterations)
Benchmarking vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0.1: Analyzing
vec ser 1024 rows, composite fixed, case 2 Key128, Pr[null]=0.1
                        time:   [21.849 µs 21.892 µs 21.939 µs]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0: Collecting 100 samples in estimated 5.7579 s (35k iterations)
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0: Analyzing
vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0
                        time:   [162.24 µs 162.67 µs 163.14 µs]
                        change: [-13.911% -13.622% -13.330%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0.01
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0.01: Collecting 100 samples in estimated 5.0910 s (35k iterations)
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0.01: Analyzing
vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0.01
                        time:   [145.74 µs 146.06 µs 146.38 µs]
                        change: [-15.497% -15.273% -15.060%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0.1
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0.1: Collecting 100 samples in estimated 5.2166 s (40k iterations)
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0.1: Analyzing
vec ser 1024 rows, mix fixed and not fixed, case 1 KeySerialized, Pr[null]=0.1
                        time:   [129.12 µs 129.43 µs 129.76 µs]
                        change: [-16.494% -16.316% -16.128%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild

Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0: Collecting 100 samples in estimated 5.0679 s (30k iterations)
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0: Analyzing
vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0
                        time:   [167.37 µs 167.72 µs 168.09 µs]
                        change: [-14.991% -13.822% -13.102%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0.01
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0.01: Collecting 100 samples in estimated 5.1593 s (35k iterations)
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0.01: Analyzing
vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0.01
                        time:   [146.33 µs 146.71 µs 147.12 µs]
                        change: [-15.601% -15.323% -15.053%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild

Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0.1
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0.1: Collecting 100 samples in estimated 5.2517 s (40k iterations)
Benchmarking vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0.1: Analyzing
vec ser 1024 rows, mix fixed and not fixed, case 2 KeySerialized, Pr[null]=0.1
                        time:   [129.20 µs 129.78 µs 130.35 µs]
                        change: [-17.176% -16.848% -16.527%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0
Benchmarking vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0: Collecting 100 samples in estimated 5.6422 s (40k iterations)
Benchmarking vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0: Analyzing
vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0
                        time:   [139.40 µs 139.63 µs 139.87 µs]
                        change: [-17.659% -17.142% -16.522%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) high mild
  5 (5.00%) high severe

Benchmarking vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0.01
Benchmarking vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0.01: Collecting 100 samples in estimated 5.6248 s (40k iterations)
Benchmarking vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0.01: Analyzing
vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0.01
                        time:   [139.09 µs 139.26 µs 139.43 µs]
                        change: [-18.261% -18.110% -17.953%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

Benchmarking vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0.1
Benchmarking vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0.1: Collecting 100 samples in estimated 5.0468 s (35k iterations)
Benchmarking vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0.1: Analyzing
vec ser 1024 rows, medium fixed KeySerialized, Pr[null]=0.1
                        time:   [142.35 µs 142.55 µs 142.77 µs]
                        change: [-18.120% -17.943% -17.751%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe

Benchmarking vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0
Benchmarking vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0: Collecting 100 samples in estimated 5.1193 s (20k iterations)
Benchmarking vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0: Analyzing
vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0
                        time:   [252.41 µs 252.82 µs 253.21 µs]
                        change: [-11.700% -11.408% -11.018%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) low mild
  1 (1.00%) high mild
  4 (4.00%) high severe

Benchmarking vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0.01
Benchmarking vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0.01: Collecting 100 samples in estimated 5.8915 s (25k iterations)
Benchmarking vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0.01: Analyzing
vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0.01
                        time:   [233.07 µs 233.38 µs 233.70 µs]
                        change: [-11.795% -11.595% -11.386%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

Benchmarking vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0.1
Benchmarking vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0.1: Collecting 100 samples in estimated 5.4841 s (25k iterations)
Benchmarking vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0.1: Analyzing
vec ser 1024 rows, medium mixed KeySerialized, Pr[null]=0.1
                        time:   [217.24 µs 217.59 µs 217.94 µs]
                        change: [-12.735% -12.554% -12.374%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Benchmarking vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0
Benchmarking vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0: Collecting 100 samples in estimated 5.2351 s (20k iterations)
Benchmarking vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0: Analyzing
vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0
                        time:   [258.90 µs 259.73 µs 260.63 µs]
                        change: [-12.731% -12.384% -12.021%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0.01
Benchmarking vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0.01: Collecting 100 samples in estimated 5.2136 s (20k iterations)
Benchmarking vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0.01: Analyzing
vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0.01
                        time:   [260.03 µs 261.03 µs 262.08 µs]
                        change: [-11.473% -11.166% -10.838%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  8 (8.00%) high mild
  1 (1.00%) high severe

Benchmarking vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0.1
Benchmarking vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0.1: Collecting 100 samples in estimated 5.4026 s (20k iterations)
Benchmarking vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0.1: Analyzing
vec ser 1024 rows, large fixed KeySerialized, Pr[null]=0.1
                        time:   [268.49 µs 269.90 µs 271.36 µs]
                        change: [-10.595% -10.265% -9.9564%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
  8 (8.00%) high mild
  6 (6.00%) high severe

Benchmarking vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0
Benchmarking vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0: Collecting 100 samples in estimated 5.8271 s (15k iterations)
Benchmarking vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0: Analyzing
vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0
                        time:   [383.31 µs 383.76 µs 384.25 µs]
                        change: [-8.1870% -8.0091% -7.8295%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild

Benchmarking vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0.01
Benchmarking vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0.01: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0.01: Collecting 100 samples in estimated 5.3821 s (15k iterations)
Benchmarking vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0.01: Analyzing
vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0.01
                        time:   [354.53 µs 355.00 µs 355.47 µs]
                        change: [-9.2836% -9.0957% -8.9173%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

Benchmarking vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0.1
Benchmarking vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0.1: Warming up for 3.0000 s
Benchmarking vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0.1: Collecting 100 samples in estimated 5.1647 s (15k iterations)
Benchmarking vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0.1: Analyzing
vec ser 1024 rows, large mixed KeySerialized, Pr[null]=0.1
                        time:   [340.23 µs 340.73 µs 341.29 µs]
                        change: [-9.9284% -9.7116% -9.5140%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

@kwannoel kwannoel marked this pull request as ready for review April 5, 2023 07:49
@kwannoel
Copy link
Contributor Author

kwannoel commented Apr 5, 2023 •
edited

To summarize here are the follow-ups required after this PR.

  • Use stack based data structure for serialized key / reserve memory for serialized key to avoid multiple allocation calls.
  • The Mixed case seems to be forced to use KeySerialized entirely. Perhaps we can come up with a better key encoding for it to segment mixed and fixed parts. (q17 falls under this case, since it has varchar group key.)
  • Additionally I have no idea why medium case, which has 8 bigint keys, uses KeySerialized rather than Key64. It has less than MAX_FIXED_SIZE_KEY_ELEMENTS (equals 8). Will figure that out separately too. They need to fit into max key size, which is Key256.

@kwannoel kwannoel requested a review from BugenZhao April 5, 2023 08:03
@codecov
Copy link

codecov bot commented Apr 6, 2023 •
edited

Codecov Report

Merging #8941 (127cfb6) into main (3091f16) will increase coverage by 0.00%.
The diff coverage is 85.45%.

❗ Current head 127cfb6 differs from pull request most recent head ce3fcc7. Consider uploading reports for the commit ce3fcc7 to get more accurate results

@@           Coverage Diff           @@
##             main    #8941   +/-   ##
=======================================
  Coverage   70.79%   70.79%           
=======================================
  Files        1185     1185           
  Lines      196254   196293   +39     
=======================================
+ Hits       138941   138974   +33     
- Misses      57313    57319    +6
Flag Coverage Δ
rust 70.79% <85.45%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/batch/src/executor/join/hash_join.rs 94.55% <ø> (ø)
src/batch/src/executor/join/lookup_join_base.rs 100.00% <ø> (ø)
src/common/src/hash/key.rs 84.77% <81.81%> (-0.13%) ⬇️
src/frontend/src/binder/select.rs 91.23% <100.00%> (+0.19%) ⬆️
src/stream/src/executor/hash_join.rs 96.60% <100.00%> (-0.01%) ⬇️
src/stream/src/executor/managed_state/join/mod.rs 90.49% <100.00%> (ø)

... and 6 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@github-actions github-actions bot removed the user-facing-changes Contains changes that are visible to users label Apr 6, 2023
BugenZhao
BugenZhao reviewed Apr 6, 2023
View reviewed changes
Copy link
Member

@BugenZhao BugenZhao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM. Thanks for your investigation!

My only concern is whether a maximum number of 64 is acceptable for group keys. 🤔

src/common/src/hash/key.rs Outdated Show resolved Hide resolved
src/common/src/hash/key.rs Outdated Show resolved Hide resolved
src/frontend/src/binder/select.rs
@@ -178,6 +180,14 @@ impl Binder {

// Bind GROUP BY clause.
self.context.clause = Some(Clause::GroupBy);
let number_of_group_keys = select.group_by.len();
if number_of_group_keys > MAX_GROUP_KEYS {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure whether this can cover all cases. For example, the columns in OVER (PARTITION BY <cols>) may also be used as a group key.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With workaround listed below, I think we can remove this, since no more max group keys.

This comment was marked as resolved.

Sign in to view

src/common/src/hash/key.rs Outdated Show resolved Hide resolved
@lmatz
Copy link
Contributor

lmatz commented Apr 6, 2023

My only concern is whether a maximum number of 64 is acceptable for group keys.

Is it possible to use a workaround for those extreme cases, i.e. putting several columns/keys into a struct first and then group by this struct?

@BugenZhao
Copy link
Member

  • Use stack based data structure for serialized key / reserve memory for serialized key to avoid multiple allocation calls.

Have we tried smallvec for serialized key? I guess it will be a seamless replacement and improvement.

@kwannoel
Copy link
Contributor Author

kwannoel commented Apr 6, 2023

  • Use stack based data structure for serialized key / reserve memory for serialized key to avoid multiple allocation calls.

Have we tried smallvec for serialized key? I guess it will be a seamless replacement and improvement.

Will try it, thanks for the suggestion.

@kwannoel
Copy link
Contributor Author

kwannoel commented Apr 6, 2023

My only concern is whether a maximum number of 64 is acceptable for group keys.

Is it possible to use a workaround for those extreme cases, i.e. putting several columns/keys into a struct first and then group by this struct?

I think there's another solution:
If exceeds 64, we revert to heap based data structure, e.g. FixedBitSet. Then we just store pointer to it, which is usize, usually same size as u64.

@BugenZhao
Copy link
Member

BugenZhao commented Apr 6, 2023 •
edited

My only concern is whether a maximum number of 64 is acceptable for group keys.

Is it possible to use a workaround for those extreme cases, i.e. putting several columns/keys into a struct first and then group by this struct?

Do you mean doing this by the optimizer or the compute node itself? As for the former, I guess we need a unified interface for hooking all group key derivations first, which might be hard. 🥵

If on the compute node, then what @kwannoel suggested above sounds good to me. 😄

@kwannoel
Copy link
Contributor Author

kwannoel commented Apr 6, 2023

My only concern is whether a maximum number of 64 is acceptable for group keys.

Is it possible to use a workaround for those extreme cases, i.e. putting several columns/keys into a struct first and then group by this struct?

I think there's another solution: If exceeds 64, we revert to heap based data structure, e.g. FixedBitSet. Then we just store pointer to it, which is usize, usually same size as u64.

Will add into this PR, since it will also resolve #8941 (comment) above.

@kwannoel @BugenZhao
Update src/common/src/hash/key.rs
340afe5 
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao
Copy link
Member

Then we just store pointer to it, which is usize, usually same size as u64.

Makes sense. Although there's an extra dereference with a pointer, I think it's tolerable since it's a rare case.

BTW, using an enum with pointer here may still increase the struct's stack size as the compiler may fail to find a position for the tag of the enum.

@kwannoel
Copy link
Contributor Author

kwannoel commented Apr 6, 2023 •
edited

Then we just store pointer to it, which is usize, usually same size as u64.

Makes sense. Although there's an extra dereference with a pointer, I think it's tolerable since it's a rare case.

BTW, using an enum with pointer here may still increase the struct's stack size as the compiler may fail to find a position for the tag of the enum.

Here's the approach I used to avoid overhead from enum: https://github.com/risingwavelabs/risingwave/pull/9026/files.
Just parameterize HashKeys by the Bitmap type, we save a usize, i.e. half the memory cost compared to enum.

However it's blocked due to clippy panic. Still investigating if there's a way around it.

I guess it is fine to merge this first, since group key > 64 seems unlikely in most cases.

Thoughts? @lmatz @BugenZhao @st1page

Can review it separately: #9026

@kwannoel kwannoel mentioned this pull request Apr 6, 2023
Merged
9 tasks
@kwannoel kwannoel mentioned this pull request Apr 6, 2023
Closed
kwannoel
kwannoel commented Apr 6, 2023
View reviewed changes
src/common/src/hash/key.rs Outdated Show resolved Hide resolved
@kwannoel kwannoel enabled auto-merge April 6, 2023 10:26
@kwannoel kwannoel changed the title perf(common): Optimize null_bitmap perf(common): Optimize null_bitmap with Set64 Apr 6, 2023
@kwannoel kwannoel added this pull request to the merge queue Apr 6, 2023
Merged via the queue into main with commit eb20393 Apr 6, 2023
8 of 9 checks passed
@kwannoel kwannoel deleted the kwannoel/optimize-bitmap-u8 branch April 6, 2023 11:00
Little-Wallace added a commit to Little-Wallace/risingwave that referenced this pull request Apr 11, 2023
@Little-Wallace
Squashed commit of the following:
347e4ad 
commit ee6d44f
Author: Clearlove <52417396+Eurekaaw@users.noreply.github.com>
Date:   Mon Apr 10 13:11:02 2023 -0400

    feat(optimizer): push `LogicalLimit` down `LogicalProject`, enabled in batch (risingwavelabs#8971)

    Signed-off-by: Clearlove <yifei.c.wei@gmail.com>

commit 5bb0911
Author: Bohan Zhang <tabvision@bupt.icu>
Date:   Mon Apr 10 23:01:53 2023 +0800

    fix: timestamptz type mismatch (risingwavelabs#9090)

    Signed-off-by: tabVersion <tabvision@bupt.icu>

commit ab09d29
Author: stonepage <40830455+st1page@users.noreply.github.com>
Date:   Mon Apr 10 21:54:15 2023 +0800

    chore: improve error msg (risingwavelabs#9089)

commit 7088fd3
Author: Richard Chien <stdrc@outlook.com>
Date:   Mon Apr 10 20:20:53 2023 +0800

    refactor(common): remove `unwrap` from `memcmp_encoding` and improve `encode_row` (risingwavelabs#9087)

    Signed-off-by: Richard Chien <stdrc@outlook.com>

commit 490161d
Author: StrikeW <wangsiyuanse@gmail.com>
Date:   Mon Apr 10 18:56:27 2023 +0800

    perf(storage): Merge multiple imms in the staging version to a large one (risingwavelabs#7368)

commit fcc069a
Author: Liang <44948473+soundOfDestiny@users.noreply.github.com>
Date:   Mon Apr 10 18:43:13 2023 +0800

    feat(meta cache): stats meta cache unhit in prometheus (risingwavelabs#9078)

commit 4929b30
Author: Dylan <chenzl25@mail2.sysu.edu.cn>
Date:   Mon Apr 10 17:31:52 2023 +0800

    feat(stream): stream window join naming (risingwavelabs#9076)

commit fd38f3d
Author: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>
Date:   Mon Apr 10 02:24:30 2023 -0700

    fix(planner): `stddev`/`var` rewriter panic due to mix use of pre/post project index (risingwavelabs#9081)

commit a3fc14a
Author: Bugen Zhao <i@bugenzhao.com>
Date:   Mon Apr 10 16:08:45 2023 +0800

    fix(expr): correctly handle unicode for substr (risingwavelabs#9079)

    Signed-off-by: Bugen Zhao <i@bugenzhao.com>

commit d1868a9
Author: August <pin@singularity-data.com>
Date:   Mon Apr 10 15:56:52 2023 +0800

    fix: reject ddl rather than panic when no available parallel units to schedule (risingwavelabs#9080)

commit 046f1d2
Author: Dylan <chenzl25@mail2.sysu.edu.cn>
Date:   Mon Apr 10 15:08:50 2023 +0800

    feat(optimizer): support functional index selection (risingwavelabs#9067)

commit a63807d
Author: Shanicky Chen <peng@singularity-data.com>
Date:   Mon Apr 10 14:45:43 2023 +0800

    chore: remove Uint256 functionality and related code (risingwavelabs#9073)

commit fc5836f
Author: xxchan <xxchan22f@gmail.com>
Date:   Mon Apr 10 08:33:22 2023 +0200

    ci: reuse some code across scripts (risingwavelabs#8926)

commit 10190e8
Author: Dylan <chenzl25@mail2.sysu.edu.cn>
Date:   Mon Apr 10 13:48:31 2023 +0800

    fix(stream): fix inner interval join (risingwavelabs#9071)

commit ed5af28
Author: congyi wang <58715567+wcy-fdu@users.noreply.github.com>
Date:   Mon Apr 10 12:14:57 2023 +0800

    feat(streaming): monitor materialize cache miss rate when handling pk conflict (risingwavelabs#8946)

commit 1e3221b
Author: Lanqing Yang <lanqingy@usc.edu>
Date:   Sun Apr 9 20:59:11 2023 -0700

    feat(expr): support for sqrt function (risingwavelabs#9017)

    Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>

commit 95ab15c
Author: Renjie Liu <liurenjie2008@gmail.com>
Date:   Sun Apr 9 17:30:45 2023 +0800

    feat: Estimate size for data chunk (risingwavelabs#9050)

commit ced72de
Author: Dylan <chenzl25@mail2.sysu.edu.cn>
Date:   Sun Apr 9 16:34:23 2023 +0800

    feat(frontend): support functional indexes creation (risingwavelabs#8976)

commit d8926b8
Author: William Wen <44139337+wenym1@users.noreply.github.com>
Date:   Sun Apr 9 01:47:25 2023 +0200

    feat(sink): introduce log trait and adopt log reader and write in sink executor (risingwavelabs#8558)

commit cc93b46
Author: Yuanxin Cao <60498509+xx01cyx@users.noreply.github.com>
Date:   Sat Apr 8 23:50:16 2023 +0800

    fix(sink): allow parsing `bool` and `u32` for kafka sink config (risingwavelabs#9057)

commit de6c5a3
Author: Bugen Zhao <i@bugenzhao.com>
Date:   Fri Apr 7 23:19:15 2023 +0800

    refactor(common): do not leak types on row (risingwavelabs#9053)

    Signed-off-by: Bugen Zhao <i@bugenzhao.com>

commit 1aa1278
Author: Tesla Zhang‮ <ice1000kotlin@foxmail.com>
Date:   Fri Apr 7 11:00:32 2023 -0400

    refactor(plan_node): create `BatchPlanRef`, simplify `HopWindow` nodes (risingwavelabs#9044)

commit c343f00
Author: congyi wang <58715567+wcy-fdu@users.noreply.github.com>
Date:   Fri Apr 7 22:19:21 2023 +0800

    refactor(metrics): fix cache metrics to be table id + actor id (risingwavelabs#9039)

commit 9a15a88
Author: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>
Date:   Fri Apr 7 03:19:50 2023 -0700

    chore: `rust_decimal` feature -db-tokio-postgres +maths (risingwavelabs#9043)

commit 85980fd
Author: Renjie Liu <liurenjie2008@gmail.com>
Date:   Fri Apr 7 16:55:24 2023 +0800

    feat: Implement GetSize for array (risingwavelabs#8995)

commit 36dd239
Author: Bugen Zhao <i@bugenzhao.com>
Date:   Fri Apr 7 15:20:21 2023 +0800

    refactor: do not leak the nested runtime (risingwavelabs#8968)

    Signed-off-by: Bugen Zhao <i@bugenzhao.com>
    Signed-off-by: Runji Wang <wangrunji0408@163.com>
    Co-authored-by: Runji Wang <wangrunji0408@163.com>

commit e685bdb
Author: Bohan Zhang <tabvision@bupt.icu>
Date:   Fri Apr 7 14:18:06 2023 +0800

    chore: deprecate console sink (risingwavelabs#8950)

    Signed-off-by: tabVersion <tabvision@bupt.icu>

commit 5169ce9
Author: Richard Chien <stdrc@outlook.com>
Date:   Fri Apr 7 14:14:17 2023 +0800

    feat(stream common): add internal cache for `SortBuffer` (risingwavelabs#8963)

    Signed-off-by: Richard Chien <stdrc@outlook.com>

commit 09e6a63
Author: ZENOTME <43447882+ZENOTME@users.noreply.github.com>
Date:   Fri Apr 7 14:08:15 2023 +0800

    refactor(frontend): refactor query to reduce redundent code in extended mode (risingwavelabs#9023)

commit 8e0bf7a
Author: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>
Date:   Thu Apr 6 10:11:45 2023 -0700

    fix(sqlparser): support more PostgreSQL `trim` syntax (risingwavelabs#8985)

commit 8038315
Author: Xinjing Hu <honeta@qq.com>
Date:   Thu Apr 6 20:59:17 2023 +0800

    perf(encoding): add benchmark for data chunk encoding (risingwavelabs#9035)

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

commit 5af13c1
Author: Zhanxiang (Patrick) Huang <hzxa21@hotmail.com>
Date:   Thu Apr 6 20:34:58 2023 +0800

    feat(metric): introduce an alert panel in grafana user dashboard (risingwavelabs#9031)

commit 3af7d21
Author: Noel Kwan <47273164+kwannoel@users.noreply.github.com>
Date:   Thu Apr 6 20:14:34 2023 +0800

    feat(sqlsmith): Generate more join expressions (risingwavelabs#8395)

commit 1293fda
Author: xxchan <xxchan22f@gmail.com>
Date:   Thu Apr 6 13:40:34 2023 +0200

    ci: increase timeout for e2e java-binding test (risingwavelabs#9032)

commit eb20393
Author: Noel Kwan <47273164+kwannoel@users.noreply.github.com>
Date:   Thu Apr 6 18:36:28 2023 +0800

    perf(common): Optimize null_bitmap with `Set64` (risingwavelabs#8941)

    Co-authored-by: Bugen Zhao <i@bugenzhao.com>

commit 29389e1
Author: Shanicky Chen <peng@singularity-data.com>
Date:   Thu Apr 6 18:07:42 2023 +0800

    feat(common): introduce ArrayBuilder and Array for Int256/Uint256  (risingwavelabs#8989)

    Co-authored-by: lmatz <lmatz823@gmail.com>

commit 83b7c79
Author: Bohan Zhang <tabvision@bupt.icu>
Date:   Thu Apr 6 17:52:55 2023 +0800

    feat(metrics): add metric for source upstream (risingwavelabs#9027)

    Signed-off-by: tabVersion <tabvision@bupt.icu>

commit 78505b1
Author: Runji Wang <wangrunji0408@163.com>
Date:   Thu Apr 6 17:40:40 2023 +0800

    fix(udf): fix missing chunks (risingwavelabs#9025)

    Signed-off-by: Runji Wang <wangrunji0408@163.com>
    Co-authored-by: lmatz <lmatz823@gmail.com>

commit ba2f6e1
Author: Zhanxiang (Patrick) Huang <hzxa21@hotmail.com>
Date:   Thu Apr 6 16:17:57 2023 +0800

    feat(metric): add recovery metrics (risingwavelabs#9022)

commit ea53cf3
Author: Runji Wang <wangrunji0408@163.com>
Date:   Thu Apr 6 15:23:51 2023 +0800

    test: simulate for worker node heartbeat timeout (risingwavelabs#7640)

    Signed-off-by: Runji Wang <wangrunji0408@163.com>
    Co-authored-by: August <pin@singularity-data.com>
    Co-authored-by: Shanicky Chen <peng@singularity-data.com>
    Co-authored-by: Shanicky Chen <peng@risingwave-labs.com>

commit b19e651
Author: Bohan Zhang <tabvision@bupt.icu>
Date:   Thu Apr 6 14:58:51 2023 +0800

    fix: ban types when creating sink (risingwavelabs#9008)

    Signed-off-by: tabVersion <tabvision@bupt.icu>

commit 4c0a0b4
Author: August <pin@singularity-data.com>
Date:   Thu Apr 6 14:42:52 2023 +0800

    fix: fix some check and clean for drop database and schema (risingwavelabs#8981)

commit 8f8c191
Author: Richard Chien <stdrc@outlook.com>
Date:   Thu Apr 6 14:28:01 2023 +0800

    refactor(stream sort): use new `SortBuffer` in `SortExecutor` (risingwavelabs#8891)

    Signed-off-by: Richard Chien <stdrc@outlook.com>

commit efe56ec
Author: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>
Date:   Thu Apr 6 14:17:47 2023 +0800

    fix(expr): align `position` syntax and `strpos` semantic with SQL standard (risingwavelabs#9000)

commit f1b7942
Author: WillyKidd <57129289+WillyKidd@users.noreply.github.com>
Date:   Thu Apr 6 13:39:31 2023 +0800

    feat(source): introduce create/show connection statement (risingwavelabs#8907)

commit 7c9da96
Author: Tesla Zhang‮ <ice1000kotlin@foxmail.com>
Date:   Thu Apr 6 01:24:48 2023 -0400

    refactor(plan_node): `TopN` simplification (risingwavelabs#9011)

commit f80ef5d
Author: Noel Kwan <47273164+kwannoel@users.noreply.github.com>
Date:   Thu Apr 6 12:26:26 2023 +0800

    bench(common): add more benchmark cases for unaligned number of keys (risingwavelabs#9018)

commit 3091f16
Author: Liang <44948473+soundOfDestiny@users.noreply.github.com>
Date:   Thu Apr 6 08:07:44 2023 +0800

    perf(hash join): do not store null rows (close risingwavelabs#9009) (risingwavelabs#9010)

commit 110972a
Author: lmatz <lmatz823@gmail.com>
Date:   Wed Apr 5 15:56:11 2023 +0800

    fix(binder): multiple unions should retain the same cte_to_relation context (risingwavelabs#9007)

commit be4f748
Author: Eric Fu <eric@singularity-data.com>
Date:   Wed Apr 5 15:31:30 2023 +0800

    fix(risedev): auto build connector node (risingwavelabs#9006)

commit b67e00f
Author: broccoliSpicy <93440049+broccoliSpicy@users.noreply.github.com>
Date:   Wed Apr 5 14:21:33 2023 +0800

    fix(binder): Incorrect cast when specifying columns (risingwavelabs#8770)

    Co-authored-by: xxchan <xxchan22f@gmail.com>

commit ce25cf5
Author: Runji Wang <wangrunji0408@163.com>
Date:   Wed Apr 5 14:20:55 2023 +0800

    fix(udf): fix wrong number of rows (risingwavelabs#9003)

    Signed-off-by: Runji Wang <wangrunji0408@163.com>

commit db072cb
Author: ZENOTME <43447882+ZENOTME@users.noreply.github.com>
Date:   Wed Apr 5 10:32:55 2023 +0800

    feat(frontend):support pull_up_hop rule (risingwavelabs#8954)

commit 731f211
Author: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>
Date:   Tue Apr 4 23:46:40 2023 +0800

    feat(binder): support some alternate names for existing exprs (risingwavelabs#8998)

commit 29c4185
Author: ZENOTME <43447882+ZENOTME@users.noreply.github.com>
Date:   Tue Apr 4 22:54:51 2023 +0800

    refactor(frontend): refine extended handle (risingwavelabs#8992)

commit 381e3b8
Author: Zhanxiang (Patrick) Huang <hzxa21@hotmail.com>
Date:   Tue Apr 4 22:35:27 2023 +0800

    feat(grafana): introduce a new dashboard to dsiplay user facing metrics (risingwavelabs#8994)

commit ef22c5b
Author: Bohan Zhang <tabvision@bupt.icu>
Date:   Tue Apr 4 20:52:54 2023 +0800

    chore: add metrics for native chunk (risingwavelabs#8997)

    Signed-off-by: tabVersion <tabvision@bupt.icu>

commit fadd752
Author: Noel Kwan <47273164+kwannoel@users.noreply.github.com>
Date:   Tue Apr 4 20:39:02 2023 +0800

    chore(common): add more benchmark cases for hash key (risingwavelabs#8993)

commit 5d8c0b4
Author: Tao Wu <wutao@singularity-data.com>
Date:   Tue Apr 4 19:59:09 2023 +0800

    test: add e2e tests for natural join and cross join (risingwavelabs#8948)

commit 35f7c0c
Author: Shanicky Chen <peng@singularity-data.com>
Date:   Tue Apr 4 19:03:55 2023 +0800

    feat(common): introduce Int256/Uint256 as Scalar and ScalarRef (risingwavelabs#8984)

commit 085dc31
Author: xxchan <xxchan22f@gmail.com>
Date:   Tue Apr 4 12:22:50 2023 +0200

    ci: reduce java-binding log (risingwavelabs#8967)

commit 506986f
Author: Yuhao Su <31772373+yuhao-su@users.noreply.github.com>
Date:   Tue Apr 4 18:09:28 2023 +0800

    fix: drop table with watermark panic (risingwavelabs#8990)

commit b9ff193
Author: Runji Wang <wangrunji0408@163.com>
Date:   Tue Apr 4 17:57:58 2023 +0800

    fix(udf): fix decimal and list types for UDF (risingwavelabs#8958)

    Signed-off-by: Runji Wang <wangrunji0408@163.com>

commit 80c477f
Author: Bugen Zhao <i@bugenzhao.com>
Date:   Tue Apr 4 17:20:20 2023 +0800

    feat(frontend): support update column to default value (risingwavelabs#8987)

    Signed-off-by: Bugen Zhao <i@bugenzhao.com>

commit 570a253
Author: August <pin@singularity-data.com>
Date:   Tue Apr 4 15:54:45 2023 +0800

    chore: notify function separately rather than inside the relation group (risingwavelabs#8983)

commit ffc634f
Author: xxchan <xxchan22f@gmail.com>
Date:   Tue Apr 4 09:49:10 2023 +0200

    fix: fix cargo audit issues and add audit to ci (risingwavelabs#8959)

commit bbe15be
Author: Runji Wang <wangrunji0408@163.com>
Date:   Tue Apr 4 15:36:43 2023 +0800

    feat: support `date_part` function and more fields for `extract` (risingwavelabs#8830)

    Signed-off-by: Runji Wang <wangrunji0408@163.com>
    Co-authored-by: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>

commit dca4ced
Author: zwang28 <70626450+zwang28@users.noreply.github.com>
Date:   Tue Apr 4 15:24:43 2023 +0800

    feat(metric): add metric for object size and count (risingwavelabs#8980)

commit c617632
Author: Huangjw <1223644280@qq.com>
Date:   Tue Apr 4 14:47:56 2023 +0800

    fix(ci): fix docker release scripts (risingwavelabs#8977)

commit da1822c
Author: Renjie Liu <liurenjie2008@gmail.com>
Date:   Tue Apr 4 14:01:32 2023 +0800

    refactor: Remove unused code (risingwavelabs#8974)

commit 9b53293
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Apr 4 13:31:14 2023 +0800

    chore(deps): bump spin from 0.9.6 to 0.9.8 (risingwavelabs#8969)

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 6687739
Author: Runji Wang <wangrunji0408@163.com>
Date:   Tue Apr 4 12:40:34 2023 +0800

    fix(expr): fix `lpad` and `rpad` on empty fill string (risingwavelabs#8970)

    Signed-off-by: Runji Wang <wangrunji0408@163.com>

commit 368ed93
Author: Bugen Zhao <i@bugenzhao.com>
Date:   Tue Apr 4 12:38:23 2023 +0800

    fix(test): clean-up temporary config file (risingwavelabs#8972)

    Signed-off-by: Bugen Zhao <i@bugenzhao.com>

commit 43bd8c9
Author: August <pin@singularity-data.com>
Date:   Tue Apr 4 12:14:10 2023 +0800

    fix: use original lru crate in meta client and simulation (risingwavelabs#8956)

commit 30dca71
Author: Tesla Zhang‮ <ice1000kotlin@foxmail.com>
Date:   Mon Apr 3 10:36:46 2023 -0400

    refactor(plan_node): simplify `Expand` and `Filter` (risingwavelabs#8932)

commit 626ff72
Author: xxchan <xxchan22f@gmail.com>
Date:   Mon Apr 3 15:28:41 2023 +0200

    refactor(risedev): replace std::fs with fs_err (risingwavelabs#8955)

commit 88d84a0
Author: Renjie Liu <liurenjie2008@gmail.com>
Date:   Mon Apr 3 18:51:10 2023 +0800

    refactor: Add backtrace for risedev (risingwavelabs#8960)

commit 1527b6c
Author: Huangjw <1223644280@qq.com>
Date:   Mon Apr 3 18:45:16 2023 +0800

    chore(release): update release version and fix ci release script (risingwavelabs#8913)

commit 8278061
Author: ZENOTME <43447882+ZENOTME@users.noreply.github.com>
Date:   Mon Apr 3 16:30:10 2023 +0800

    refactor(frontend): refactor extended query mode (risingwavelabs#8919)

commit 06025d5
Author: lmatz <lmatz823@gmail.com>
Date:   Mon Apr 3 16:06:38 2023 +0800

    chore(metrics): add and correct some description of metrics shown on Grafana (risingwavelabs#8890)

commit 806c623
Author: Yuanxin Cao <60498509+xx01cyx@users.noreply.github.com>
Date:   Mon Apr 3 15:49:08 2023 +0800

    feat(sink): reject invalid options when creating sink (risingwavelabs#8757)

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

commit 36c4605
Author: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>
Date:   Mon Apr 3 15:47:16 2023 +0800

    test(regress): enable `strings` (risingwavelabs#8952)

commit 3c884e1
Author: idx0-dev <124041366+idx0-dev@users.noreply.github.com>
Date:   Mon Apr 3 15:27:06 2023 +0800

    feat:  encode JSON in the same way as debezium (risingwavelabs#8865)

commit 312d418
Author: Bohan Zhang <tabvision@bupt.icu>
Date:   Mon Apr 3 15:04:38 2023 +0800

    fix: unstable test by checking cardinality instead of actual values (risingwavelabs#8947)

    Signed-off-by: tabVersion <tabvision@bupt.icu>

commit 6c3c80d
Author: Yuanxin Cao <60498509+xx01cyx@users.noreply.github.com>
Date:   Mon Apr 3 14:35:31 2023 +0800

    feat(streaming): introduce dedup cache and append-only dedup executor (risingwavelabs#8874)

    Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

commit 37ce407
Author: Liang <44948473+soundOfDestiny@users.noreply.github.com>
Date:   Mon Apr 3 14:16:59 2023 +0800

    fix(delete range): maintain unused iters in `seek` (close risingwavelabs#8928) (risingwavelabs#8929)

commit 6f46e4c
Author: Liang <44948473+soundOfDestiny@users.noreply.github.com>
Date:   Mon Apr 3 14:14:18 2023 +0800

    fix(shared buffer): shared buffer consistency issue (close risingwavelabs#8944) (risingwavelabs#8945)

commit 4254b42
Author: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>
Date:   Mon Apr 3 13:48:38 2023 +0800

    test(regress): enable `text` (risingwavelabs#8943)

commit 7c259eb
Author: WillyKidd <57129289+WillyKidd@users.noreply.github.com>
Date:   Mon Apr 3 13:42:10 2023 +0800

    fix(connector): dbz source handler avoid oom more elegantly (risingwavelabs#8799)

commit ea52212
Author: idx0-dev <124041366+idx0-dev@users.noreply.github.com>
Date:   Mon Apr 3 12:57:08 2023 +0800

    feat(e2esinktest): switch to stream_chunk (risingwavelabs#8914)

commit ac104a7
Author: xiangjinwu <17769960+xiangjinwu@users.noreply.github.com>
Date:   Mon Apr 3 10:13:17 2023 +0800

    test(regress): enable `case` (also covers `coalesce` `nullif`) (risingwavelabs#8921)

commit 9f56e80
Author: StrikeW <wangsiyuanse@gmail.com>
Date:   Sun Apr 2 23:54:44 2023 +0800

    fix(ci): quick fix for main cron ci (risingwavelabs#8931)

commit e217689
Author: Tesla Zhang‮ <ice1000kotlin@foxmail.com>
Date:   Sun Apr 2 09:20:10 2023 -0400

    refactor(plan_node): simplify agg-related nodes (risingwavelabs#8930)

commit 039d4a0
Author: Liang <44948473+soundOfDestiny@users.noreply.github.com>
Date:   Sat Apr 1 17:31:50 2023 +0800

    fix(move table): apply version correctly (close risingwavelabs#8764) (risingwavelabs#8765)

commit ad53077
Author: Shuxian Wang <wsx@berkeley.edu>
Date:   Fri Mar 31 15:21:13 2023 -0700

    fix(stream): Better cache handling for temporal join. (risingwavelabs#8871)

commit d687d71
Author: Bugen Zhao <i@bugenzhao.com>
Date:   Fri Mar 31 23:00:45 2023 +0800

    refactor(config): split streaming and batch developer config (risingwavelabs#8911)

    Signed-off-by: Bugen Zhao <i@bugenzhao.com>

commit ff1440f
Author: Renjie Liu <liurenjie2008@gmail.com>
Date:   Fri Mar 31 21:39:27 2023 +0800

    refactor: Remove unnecessary bash scripts (risingwavelabs#8925)

commit dbbd1cb
Author: Yuhao Su <31772373+yuhao-su@users.noreply.github.com>
Date:   Fri Mar 31 18:46:55 2023 +0800

    feat(generated_columns): support select generated columns from source (risingwavelabs#8841)

    Co-authored-by: Bugen Zhao <i@bugenzhao.com>
    Co-authored-by: stonepage <40830455+st1page@users.noreply.github.com>

commit e51b240
Author: Bohan Zhang <tabvision@bupt.icu>
Date:   Fri Mar 31 17:45:44 2023 +0800

    fix: propagate connector reader error to the upper layer (risingwavelabs#8920)

    Signed-off-by: tabVersion <tabvision@bupt.icu>

commit 74b935e
Author: Noel Kwan <47273164+kwannoel@users.noreply.github.com>
Date:   Fri Mar 31 17:25:54 2023 +0800

    docs: add metrics doc on barrier (risingwavelabs#8902)

commit 96d92ca
Author: Liang <44948473+soundOfDestiny@users.noreply.github.com>
Date:   Fri Mar 31 17:22:52 2023 +0800

    feat(sst meta): rollback fetch meta strategy (risingwavelabs#8917)

commit 6c99e93
Author: Li0k <yuli@singularity-data.com>
Date:   Fri Mar 31 16:45:39 2023 +0800

    chore(storage): monitor compacting task count per level (risingwavelabs#8681)

commit 188846e
Author: Renjie Liu <liurenjie2008@gmail.com>
Date:   Fri Mar 31 16:29:42 2023 +0800

    fix: Disable generated warning in docker compose (risingwavelabs#8916)

commit 85ecc73
Author: August <pin@singularity-data.com>
Date:   Fri Mar 31 15:45:56 2023 +0800

    fix: replay functions in snapshot on frontend (risingwavelabs#8912)

commit dc5d3de
Author: Kevin Axel <kevinaxel@163.com>
Date:   Fri Mar 31 15:42:03 2023 +0800

    feat(optimizer): bushy join equivalent condition derivation (risingwavelabs#8668)

    Signed-off-by: Kevin Axel <kevinaxel@163.com>

commit 010bec6
Author: Xinjing Hu <honeta@qq.com>
Date:   Fri Mar 31 15:32:24 2023 +0800

    perf(encoding): use estimate size to reserve memory in row encoding (risingwavelabs#8909)

    Co-authored-by: stonepage <40830455+st1page@users.noreply.github.com>

commit 1b05700
Author: Renjie Liu <liurenjie2008@gmail.com>
Date:   Fri Mar 31 15:10:23 2023 +0800

    refactor: Simplify some commands in kafka cdc example. (risingwavelabs#8910)

commit 9bcfb17
Author: Tesla Zhang‮ <ice1000kotlin@foxmail.com>
Date:   Fri Mar 31 02:20:44 2023 -0400

    refactor(plan_node): Further simplify join-related plan nodes (risingwavelabs#8905)

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BugenZhao BugenZhao BugenZhao left review comments

@lmatz lmatz lmatz approved these changes

@st1page st1page Awaiting requested review from st1page

Assignees
No one assigned
Labels
breaking-change type/perf
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@kwannoel @lmatz @BugenZhao @st1page