provide config stream processor

[mrproliu]

Please answer these questions before submitting pull request

• Why submit this pull request?

• Bug fix

• New feature provided

• Improve performance

• Related issues
  [FEATURE] Performance profile at method level in production environment #4104

New feature or improvement

• Provide a new stream processor, because when I create the thread monitor task, need storage similar to inventory and support TTL deletion mode similar to the record.

[wu-sheng]

Please add comments to all classes. For all new core classes, this should be done.

Most codes are good, just one question about IConfigDAO.

@kezhenxu94 @dmsolr @wayilau @arugal Any interest to follow this PR?

[codecov-io]

Codecov Report

Merging #4133 into master will decrease coverage by 0.07%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #4133      +/-   ##
==========================================
- Coverage   27.26%   27.18%   -0.08%     
==========================================
  Files        1141     1145       +4     
  Lines       25057    25100      +43     
  Branches     3625     3629       +4     
==========================================
- Hits         6831     6824       -7     
- Misses      17620    17670      +50     
  Partials      606      606

Impacted Files	Coverage Δ
...torage/plugin/elasticsearch/base/StorageEsDAO.java	0% <0%> (ø)	⬆️
...re/analysis/worker/NoneStreamPersistentWorker.java	0% <0%> (ø)
...r/core/analysis/worker/NoneStreamingProcessor.java	0% <0%> (ø)
...server/core/analysis/StreamAnnotationListener.java	0% <0%> (ø)	⬆️
...orage/plugin/elasticsearch7/dao/StorageEs7DAO.java	0% <0%> (ø)	⬆️
...erver/storage/plugin/jdbc/h2/dao/H2StorageDAO.java	0% <0%> (ø)	⬆️
...er/storage/plugin/jdbc/h2/dao/H2NoneStreamDAO.java	0% <0%> (ø)
...age/plugin/elasticsearch/base/NoneStreamEsDAO.java	0% <0%> (ø)
...apm/agent/core/remote/GRPCStreamServiceStatus.java	29.16% <0%> (-25.01%)	⬇️
.../agent/core/context/trace/AbstractTracingSpan.java	60.36% <0%> (-0.91%)	⬇️
... and 4 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9415f68...cab0750. Read the comment docs.

[kezhenxu94]

• Provide a new stream processor, because when I create the thread monitor task, need storage similar to inventory and support TTL deletion mode similar to the record.

@mrproliu Does the configs need TTL and why?

As you wrote in #4144 , monitoring tasks are stored in the inventory here, I suppose the tasks are created by users in the web UI right? So why they should be deleted after a so-called TTL? Does the users expect this? Is it designed for so?

[wu-sheng]

monitoring tasks are stored in the inventory here, I suppose the tasks are created by users in the web UI right? So why they should be deleted after a so-called TTL? Does the users expect this? Is it designed for so?

Monitoring should be set by UI or GraphQL. But it isn't inventory, that is why we create a new type of streaming process. I think the point of deleting is the thread dump will be saved as a kind of record, similar to the segment. So even we don't set up TTL, it will miss all related thread dump data.
I agree to support TTL.

@mrproliu A reminder to you, as TTL setup based on time bucket, this field should be set based on task expected finish time, rather than config time. Because you need to avoid the task has been deleted before the task starts.

Also, I notice, this task entity is not time series, why? Deleting is a big payload for the ES, I would recommend setup the time series for this too.

---

Request rename

Config is a very widely used concept. In the streaming process, a mode called config is very strange and not clear. I want to propose the processor name, noneStreamingProcessor.

[arugal]

So, is Thread Monitor Task Segment snapshot a Config?

[mrproliu]

monitoring tasks are stored in the inventory here, I suppose the tasks are created by users in the web UI right? So why they should be deleted after a so-called TTL? Does the users expect this? Is it designed for so?

Monitoring should be set by UI or GraphQL. But it isn't inventory, that is why we create a new type of streaming process. I think the point of deleting is the thread dump will be saved as a kind of record, similar to the segment. So even we don't set up TTL, it will miss all related thread dump data.
I agree to support TTL.

@mrproliu A reminder to you, as TTL setup based on time bucket, this field should be set based on task expected finish time, rather than config time. Because you need to avoid the task has been deleted before the task starts.

Also, I notice, this task entity is not time series, why? Deleting is a big payload for the ES, I would recommend setup the time series for this too.

Request rename

Config is a very widely used concept. In the streaming process, a mode called config is very strange and not clear. I want to propose the processor name, noneStreamingProcessor.

I agree with all your suggestions. I will change that today.

[mrproliu]

So, is Thread Monitor Task Segment snapshot a Config?

No, It will create base on Record.

[mrproliu]

@wu-sheng All issue has been resolved.
please have a look.

[wu-sheng]

LGTM. @kezhenxu94 @arugal Recheck?

[kezhenxu94]

LGTM