Skip to content

Comments

[INLONG-12019][SDK] Transformation supports a caching mechanism for processing identical function parameters#12020

Merged
dockerzhang merged 3 commits intoapache:masterfrom
luchunliang:INLONG-12019
Oct 17, 2025
Merged

[INLONG-12019][SDK] Transformation supports a caching mechanism for processing identical function parameters#12020
dockerzhang merged 3 commits intoapache:masterfrom
luchunliang:INLONG-12019

Conversation

@luchunliang
Copy link
Contributor

@luchunliang luchunliang commented Oct 16, 2025

Fixes #12019

Motivation

Transformation supports a caching mechanism for processing identical function parameters within the same SQL statement to reduce redundant computations.

  • Save the field value selected in the selectelement to the context cache so that it can be referenced for subsequent field assignments.
  • Save the result of the function(url_decode, parse_url) call to the context cache so that repeated calls can retrieve the result directly from the context cache, reducing CPU consumption.

Modifications

Add an SQL function parameter caching mechanism to reduce repeated calculations of the same parameters and improve processing performance.

  • Added a getRuntimeParamsmethod to return the runtime parameter mapping.
  • After field assignment, values are stored in the context object for subsequent use.
  • Added a test method to compare the performance of the processor with and without context.
  • Introduced runtime parameter caching logic in function parsing, unifying return value handling and caching results.
  • Added an exprKeyfield to store function expressions, and cached nullvalues in case of parsing failures or exceptions.

Repeat the test SQL 100,000 times; the time consumption is from 57000 ms to 24729 ms. Performance has been optimized by approximately 56%.

Verifying this change

(Please pick either of the following options)

  • This change is a trivial rework/code cleanup without any test coverage.

  • This change is already covered by existing tests, such as:
    (please describe tests)

  • This change added tests and can be verified as follows:

    (example:)

    • Added integration tests for end-to-end deployment with large payloads (10MB)
    • Extended integration test for recovery after broker failure

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation

…rocessing identical function parameters within the same SQL statement to reduce redundant computations
@luchunliang luchunliang added this to the 2.3.0 milestone Oct 16, 2025
@luchunliang luchunliang self-assigned this Oct 16, 2025
@dockerzhang dockerzhang changed the title [INLONG-12019][SDK] Transformation supports a caching mechanism for processing identical function parameters within the same SQL statement to reduce redundant computations [INLONG-12019][SDK] Transformation supports a caching mechanism for processing identical function parameters Oct 17, 2025
@dockerzhang dockerzhang merged commit 9a79697 into apache:master Oct 17, 2025
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

4 participants