Skip to content

[improvement](parser) Localize identifier post-processing - #67457

Merged
morrySnow merged 2 commits into
apache:masterfrom
morrySnow:codex/antlr4-local-postprocessor
Sep 3, 2026
Merged

[improvement](parser) Localize identifier post-processing#67457
morrySnow merged 2 commits into
apache:masterfrom
morrySnow:codex/antlr4-local-postprocessor

Conversation

@morrySnow

@morrySnow morrySnow commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

PostProcessor was installed as a global ANTLR parse listener, so every rule exit paid listener-dispatch cost even though only three local actions were needed: normalizing non-reserved identifiers, normalizing quoted identifiers, and reporting malformed unquoted identifiers.

This PR moves those actions into the corresponding grammar rules and removes the listener from both DorisSqlParser and NereidsParser. It preserves identifier token text/type/positions, double-backtick unescaping, malformed-identifier errors, and parsing with buildParseTree=false.

Benchmark

Environment:

  • Baseline: 760b14ee62c (benchmark-only commit on f054492cbb9)
  • Candidate: 5b45223bc61
  • JDK 17.0.20.1, JMH 1.37, macOS arm64
  • 1 thread, 3 forks, -Xms1g -Xmx1g, 4 x 300 ms warmup, 7 x 400 ms measurement, -prof gc
  • Baseline is the average of two complete runs (B1/B2). Candidate error is JMH's 99.9% confidence interval.
  • A preliminary candidate run affected by host interference was discarded (typical parser-only was 88.8 us/op instead of the stable 7-9 us/op range); the final artifact was rebuilt and measured in a clean run.

Build and run:

LC_ALL=en_US.UTF-8 mvn -Pbenchmark -pl fe-sql-parser,fe-sql-parser-benchmark -am \
    -Dmaven.build.cache.enabled=false package
java -jar fe/fe-sql-parser-benchmark/target/fe-sql-parser-benchmark-*-benchmark.jar \
    'IdentifierPostProcessorBenchmark.*' -prof gc -rf json -rff result.json

Time is us/op; positive change means faster. The workloads cover a control query, a typical aggregate query, a 64-column query, a non-reserved-keyword-heavy query, and a quoted-identifier-heavy query.

Path / Workload Baseline (B1 / B2) Candidate Latency Improvement Allocation (B/op): Baseline → Candidate
end-to-end / control 2.817 / 2.694 2.191 ± 0.041 25.8% faster 4,933.4 → 4,848.0 (-1.73%)
end-to-end / typical 17.143 / 18.288 17.173 ± 1.742 3.2% faster 18,754.0 → 18,658.9 (-0.51%)
end-to-end / wide 66.555 / 72.165 60.816 ± 0.731 14.0% faster 141,501.2 → 141,432.9 (-0.05%)
end-to-end / nonReserved 12.278 / 14.034 11.044 ± 0.276 19.1% faster 22,844.4 → 22,530.9 (-1.37%)
end-to-end / quoted 13.154 / 11.128 8.089 ± 1.121 50.1% faster 16,081.5 → 15,984.1 (-0.61%)
parser-only / control 1.703 / 1.841 1.550 ± 0.016 14.3% faster 3,824.0 → 3,744.0 (-2.09%)
parser-only / typical 8.655 / 8.565 7.868 ± 0.096 9.4% faster 13,872.1 → 13,792.1 (-0.58%)
parser-only / wide 60.700 / 59.489 55.784 ± 0.624 7.7% faster 115,632.9 → 115,552.8 (-0.07%)
parser-only / nonReserved 10.127 / 10.148 9.863 ± 0.756 2.8% faster 19,261.5 → 19,189.5 (-0.37%)
parser-only / quoted 6.985 / 7.116 6.658 ± 0.186 5.9% faster 13,440.1 → 13,346.8 (-0.69%)

The direct parser path improves by 2.8%-14.3% and allocates less in every workload. End-to-end means also improve in every workload; the larger quoted/control figures have more host-level variance and are not used as the primary conclusion. The gain comes from removing global rule-exit listener dispatch; the remaining allocation reduction comes from avoiding listener bookkeeping.

Artifact/result hashes:

  • Baseline benchmark jar: b9941cb95ea3cc8797d0ad896259049613e822c0f72319403a062bfc9bcb51a4
  • Candidate benchmark jar: 2d33163832f90e0d1c1929e522e012becbab56e3bc463dcc19b42f0e9d27526a
  • B1 JSON: 3629629849730548e3e8ff5ab54bb9d02b14e3dde3aff3b8aa51c31cc74c8c5e
  • B2 JSON: ca94f64515f2b904244f78139c7c43f636bff1f72f5daa00a748f7b9800167e6
  • Candidate JSON: 98227ae4ce630bf306084573a2a315a5f3d3d0081c787a08c6eca9cea6c73000
  • Raw JSON retained locally under /private/tmp/doris-p7-benchmark.YSnF7t/.

Semantic verification

  • Parsed all 4,610 tracked .sql files in default and ANSI modes. Acceptance/error class/error position/statement count matched the frozen baseline in all 9,220 cases.
  • Tracked SQL path-list SHA-256: c1fdf48a22d311f7164516b512e850bef373126be1d3079148c109e5f80f80c1.
  • Detailed CST/token/error snapshots cover ordinary, non-reserved, quoted, doubled-backtick and multipart identifiers, expressions, DDL, both parser entry paths, and test-table / test-tbl errors. Every rule node and token field matched the baseline.
  • Verified the advanced parser path with buildParseTree=false.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
      • mvn -Pbenchmark -pl fe-sql-parser,fe-sql-parser-benchmark -am -Dmaven.build.cache.enabled=false package (193 parser tests passed)
      • ./run-fe-ut.sh --run org.apache.doris.nereids.parser.NereidsParserTest
    • Manual test
      • Baseline/candidate corpus, CST/token/error differential checks, and JMH benchmark described above
      • ./build.sh --fe
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@morrySnow
[test](fe) Add identifier post-processing benchmark
760b14e 
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Add fixed control, ordinary identifier, non-reserved keyword, quoted identifier, and wide projection workloads for measuring global parser listener removal through both the standalone facade and a pre-tokenized parser.

### Release note

None

### Check List (For Author)

- Test: Manual test
    - Benchmark module package with Checkstyle
    - JMH smoke run for all workloads and both benchmark methods

- Behavior changed: No

- Does this need documentation: No
@morrySnow
[improvement](parser) Localize identifier post-processing
5b45223 
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: The global ANTLR parse listener receives every rule-exit event even though it only normalizes non-reserved and quoted identifiers and reports malformed identifiers. Move those actions into their grammar rules and remove listener dispatch from both parser facades. Preserve token text, type, positions, malformed-identifier errors, and parsing without a CST. JMH parser-only workloads improve by 2.8%-14.3% (typical 9.4%, wide 7.7%) while allocation falls by 0.07%-2.09%; end-to-end typical and wide workloads improve by 3.2% and 14.0%.

### Release note

None

### Check List (For Author)

- Test: Unit Test and manual differential/performance tests
    - `mvn -Pbenchmark -pl fe-sql-parser,fe-sql-parser-benchmark -am -Dmaven.build.cache.enabled=false package`
    - `./run-fe-ut.sh --run org.apache.doris.nereids.parser.NereidsParserTest`
    - `./build.sh --fe`
    - 4,610 tracked SQL files in default and ANSI modes match the baseline
    - Identifier CST, token, and error snapshots match the baseline
    - JMH 1.37, 3 forks, 4 warmup and 7 measurement iterations
- Behavior changed: No
- Does this need documentation: No
@hello-stephen

hello-stephen commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow

morrySnow commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16802 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5b45223bc6134d2cb4128c81d566aacd08b02836, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17606	3094	3086	3086
q2	2113	255	214	214
q3	10234	871	517	517
q4	4673	246	201	201
q5	7681	553	367	367
q6	140	115	95	95
q7	533	503	389	389
q8	9240	858	942	858
q9	3430	2363	2376	2363
q10	6541	863	705	705
q11	392	195	182	182
q12	622	269	194	194
q13	18110	1523	1170	1170
q14	159	146	144	144
q15	q16	438	390	358	358
q17	1354	881	825	825
q18	3112	2251	2253	2251
q19	1300	909	807	807
q20	372	280	205	205
q21	5646	1649	1829	1649
q22	335	267	222	222
Total cold run time: 94031 ms
Total hot run time: 16802 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3456	3394	3382	3382
q2	526	404	374	374
q3	2223	2416	2139	2139
q4	1191	1151	899	899
q5	2148	2103	2090	2090
q6	167	118	86	86
q7	1054	905	835	835
q8	1604	1408	1429	1408
q9	3117	3110	3082	3082
q10	1857	1786	1586	1586
q11	355	268	258	258
q12	457	428	345	345
q13	1486	1520	1195	1195
q14	175	165	166	165
q15	q16	393	398	354	354
q17	3624	3288	3160	3160
q18	4786	4391	4699	4391
q19	895	861	856	856
q20	979	943	825	825
q21	3787	3074	3247	3074
q22	390	344	342	342
Total cold run time: 34670 ms
Total hot run time: 30846 ms

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82312 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5b45223bc6134d2cb4128c81d566aacd08b02836, data reload: false

query5	4281	409	322	322
query6	396	138	122	122
query7	4962	425	238	238
query8	297	122	116	116
query9	8676	2881	2888	2881
query10	390	217	188	188
query11	5363	1047	910	910
query12	118	72	67	67
query13	1203	449	335	335
query14	6079	2184	2058	2058
query14_1	1967	1964	1976	1964
query15	175	119	109	109
query16	944	366	355	355
query17	812	463	371	371
query18	2335	330	236	236
query19	161	144	109	109
query20	69	69	68	68
query21	205	102	86	86
query22	5584	5349	5380	5349
query23	6636	6261	6002	6002
query23_1	5973	6130	5952	5952
query24	7297	1111	779	779
query24_1	773	770	784	770
query25	437	314	257	257
query26	1242	241	133	133
query27	2778	422	258	258
query28	4659	1503	1485	1485
query29	952	451	361	361
query30	258	154	130	130
query31	835	409	336	336
query32	130	73	74	73
query33	465	219	193	193
query34	988	837	471	471
query35	419	411	342	342
query36	570	591	509	509
query37	142	82	75	75
query38	1018	838	815	815
query39	479	464	460	460
query39_1	448	479	464	464
query40	204	91	77	77
query41	60	57	57	57
query42	76	72	77	72
query43	255	246	225	225
query44	1058	546	559	546
query45	113	104	102	102
query46	758	816	532	532
query47	774	770	720	720
query48	319	321	245	245
query49	564	248	223	223
query50	772	263	200	200
query51	8291	8223	8154	8154
query52	65	68	63	63
query53	187	198	141	141
query54	223	229	151	151
query55	75	63	59	59
query56	182	164	173	164
query57	682	673	623	623
query58	216	161	147	147
query59	1217	1227	1094	1094
query60	232	185	188	185
query61	120	116	119	116
query62	364	195	194	194
query63	170	142	139	139
query64	2746	678	597	597
query65	1625	1609	1622	1609
query66	1839	258	208	208
query67	9953	9735	9679	9679
query68	2760	1202	766	766
query69	334	208	197	197
query70	687	622	616	616
query71	243	177	172	172
query72	2292	1744	1566	1566
query73	643	573	335	335
query74	1576	1209	1134	1134
query75	1162	1124	966	966
query76	2301	730	586	586
query77	245	263	215	215
query78	4106	3857	3284	3284
query79	2417	820	600	600
query80	1586	330	288	288
query81	490	155	134	134
query82	627	130	94	94
query83	285	204	190	190
query84	289	112	88	88
query85	787	348	299	299
query86	396	177	168	168
query87	1023	965	890	890
query88	2755	2114	2102	2102
query89	282	195	176	176
query90	1970	126	122	122
query91	132	117	104	104
query92	82	63	70	63
query93	1424	1061	724	724
query94	651	269	237	237
query95	525	254	231	231
query96	785	564	269	269
query97	1058	1041	996	996
query98	150	137	141	137
query99	432	344	303	303
Total cold run time: 177572 ms
Total hot run time: 82312 ms

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5b45223bc6134d2cb4128c81d566aacd08b02836, data reload: false

query1	0.01	0.00	0.01
query2	0.07	0.03	0.03
query3	0.24	0.10	0.09
query4	1.60	0.10	0.09
query5	0.17	0.16	0.16
query6	1.26	0.70	0.71
query7	0.03	0.01	0.00
query8	0.05	0.03	0.03
query9	0.29	0.21	0.22
query10	0.36	0.35	0.33
query11	0.16	0.13	0.12
query12	0.14	0.12	0.12
query13	0.31	0.32	0.31
query14	0.45	0.45	0.44
query15	0.35	0.34	0.35
query16	0.22	0.23	0.21
query17	0.72	0.75	0.69
query18	0.18	0.17	0.18
query19	1.12	1.22	1.06
query20	0.01	0.01	0.01
query21	15.43	0.16	0.11
query22	5.11	0.04	0.04
query23	16.17	0.26	0.10
query24	3.07	0.32	0.28
query25	0.10	0.04	0.03
query26	0.80	0.16	0.13
query27	0.03	0.03	0.03
query28	3.69	0.56	0.27
query29	12.47	3.18	2.59
query30	0.26	0.10	0.11
query31	2.75	0.35	0.17
query32	3.54	0.32	0.22
query33	1.35	1.53	1.50
query34	15.36	2.16	1.74
query35	1.74	1.73	1.72
query36	0.47	0.29	0.29
query37	0.06	0.04	0.03
query38	0.05	0.03	0.03
query39	0.03	0.03	0.03
query40	0.11	0.08	0.07
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 90.48 s
Total hot run time: 14.66 s

@morrySnow
morrySnow merged commit cdb0578 into apache:master Sep 3, 2026
33 of 35 checks passed
@morrySnow
morrySnow deleted the codex/antlr4-local-postprocessor branch September 3, 2026 07:56
@morrySnow morrySnow mentioned this pull request Sep 3, 2026
Merged
11 tasks
morrySnow added a commit that referenced this pull request Sep 3, 2026
@morrySnow
[fix](parser) Remove stale post-processor usage from limit test (#67482)
b42ebc8 
### What problem does this PR solve?

Related PR: #67455, #67457

Problem Summary:

`LimitClausePrefixTest` registered `PostProcessor` directly. After
#67457 moved identifier post-processing into grammar actions and deleted
`PostProcessor`, combining it with the LIMIT-prefix test from #67455
caused `fe-sql-parser` test compilation to fail.

Remove the obsolete import and listener registration. Identifier
post-processing is now performed by the grammar, and the LIMIT test does
not require a global parse listener.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@924060929 924060929 924060929 approved these changes
@starocean999 starocean999 Awaiting requested review from starocean999 starocean999 is a code owner
@englefly englefly Awaiting requested review from englefly englefly is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@morrySnow @hello-stephen @924060929