Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix][fn] Fix JavaInstanceStarter inferring type class name error #19896

Merged
merged 2 commits into from May 24, 2023

Conversation

jiangpengcheng
Copy link
Contributor

@jiangpengcheng jiangpengcheng commented Mar 22, 2023

Fixes #19885

Master Issue: #xyz

PIP: #xyz

Motivation

JavaInstanceStarter still cannot infer type class name

Modifications

  1. create a new FunctionCacheManager in JavaInstanceStarter to create the function class loader and pass this fnCache to ThreadRuntimeFactory's constructor method.
  2. create a new constructor method for ThreadRuntimeFactory, which add a fnCache parameter so that it can reuse the fnCache created in JavaInstanceStarter
  3. make ThreadRuntime#loadJars public so it can be called in the JavaInstanceStarter
  4. use the function class loader to infer type class instead of the instance class loader(which cannot load the user's function class)
  5. check whether userConfigs is null before calling userConfigs.containsKey
  6. use functionDetailsBuilder.getSink().getClassName() to load sink class instead of functionDetailsBuilder.getClassName() since this one is always org.apache.pulsar.functions.api.utils.IdentityFunction
  7. use functionDetailsBuilder.getSource().getClassName(), classLoader to load source class instead of functionDetailsBuilder.getClassName() since this one is always org.apache.pulsar.functions.api.utils.IdentityFunction

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

  • This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: jiangpengcheng#7

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Mar 22, 2023
@jiangpengcheng jiangpengcheng changed the title Fix inferring type [fix][fn] Fix inferring type Mar 22, 2023
@jiangpengcheng jiangpengcheng changed the title [fix][fn] Fix inferring type [fix][fn] Fix JavaInstanceStarter inferring type class name error Mar 24, 2023
@nlu90
Copy link
Member

nlu90 commented Mar 27, 2023

@jiangpengcheng Could you add more details for the actual modifications you made in this PR?

@jiangpengcheng
Copy link
Contributor Author

@jiangpengcheng Could you add more details for the actual modifications you made in this PR?

Sure, Updated

@jiangpengcheng
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

@codecov-commenter
Copy link

Codecov Report

Merging #19896 (5ff88fb) into master (1e44ba1) will increase coverage by 37.52%.
The diff coverage is 25.00%.

Impacted file tree graph

@@              Coverage Diff              @@
##             master   #19896       +/-   ##
=============================================
+ Coverage     24.42%   61.94%   +37.52%     
- Complexity      291     3507     +3216     
=============================================
  Files          1603     1859      +256     
  Lines        124343   136750    +12407     
  Branches      13571    15034     +1463     
=============================================
+ Hits          30369    84709    +54340     
+ Misses        89490    44264    -45226     
- Partials       4484     7777     +3293     
Flag Coverage Δ
inttests 24.32% <0.00%> (-0.11%) ⬇️
systests 25.11% <25.00%> (?)
unittests 59.24% <12.50%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../pulsar/functions/runtime/JavaInstanceStarter.java 0.00% <0.00%> (ø)
...functions/runtime/thread/ThreadRuntimeFactory.java 68.23% <30.00%> (+68.23%) ⬆️
...pulsar/functions/runtime/thread/ThreadRuntime.java 50.40% <100.00%> (+50.39%) ⬆️

... and 1384 files with indirect coverage changes

@github-actions
Copy link

github-actions bot commented May 7, 2023

The pr had no activity for 30 days, mark with Stale label.

@Technoboy- Technoboy- merged commit 05e57dd into apache:master May 24, 2023
55 checks passed
jiangpengcheng added a commit to jiangpengcheng/pulsar that referenced this pull request May 29, 2023
Technoboy- pushed a commit to jiangpengcheng/pulsar that referenced this pull request May 29, 2023
poorbarcode pushed a commit that referenced this pull request May 30, 2023
liangyepianzhou pushed a commit that referenced this pull request Jul 7, 2023
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Jul 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment