Script application utilizing 100% CPU. #334
Comments
|
Hi @dkumarkartik , I am not aware of any script processor specific performance suggestions/tips. A few questions:
Thanks |
Hello @onobc, |
|
Hi @dkumarkartik ,
Thanks |
|
Hi @dkumarkartik, Are you increase both the request and limit on CPU? |
Hello, |
|
Please provide detail of what the specific script is doing and which OOTB apps your comparing to. |
Hello, 1- We created a stream which read data from S3 buckets using S3 source application and after that script application process those records and sends to kafka topic. function test(p) { try { p = JSON.parse(p); p.custom_field = 'NA'; } catch(e) {throw e} return p}; test(payload)
|
|
Isn't that Let's consider to replace your script processor with a plain So, I'd try something like |
@artembilan actually the json is not of size 1GB. Actually the input file size to pipeline is 1 GB file which has 3 million records, which is read by s3 source app and passed onto script application and sample record can be like |
|
No, you didn't mention that you split those "3 million record" into individual messages to process them one by one in the subsequent script processor. Also it is not clear if you run your whole data flow and the mentioned stream on the same machine or it is really distributed. Does it make sense to you? |
@artembilan thanks for the details, yes script process are running in its own containers in k8s but are not currently parallel, all records are in one topic and handled by the same script container. We will keep you posted on once we try other options and see some difference. |
|
@onobc / @corneil / @artembilan , request your help to get the below groovy script working for our app with groovy processor we see syntax is correct, but not working from groovy processor But no luck it's throwing some error as: |
|
I doubt that your Groovy script is really declared as a single line. Also, please, past a stack trace in a multi-line manner as well. Otherwise it is not readable and it is going to be hard to help. |
|
@artembilan & @Noushath , {
"date": 1669183955476,
"streams": [
{
"name": "groovy-script-1",
"dslText": ":test1 > test-groovy-script: groovy --management.metrics.tags.application.type=${spring.cloud.dataflow.stream.app.type:unknown} --management.metrics.export.prometheus.enabled=true --management.metrics.tags.stream.name=${spring.cloud.dataflow.stream.name:unknown} --management.metrics.tags.application=${spring.cloud.dataflow.stream.name:unknown}-${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown} --spring.cloud.stream.kafka.streams.binder.zknodes=zookeeper:2181 --management.metrics.tags.instance.index=${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}} --spring.cloud.stream.kafka.streams.binder.brokers=PLAINTEXT://broker:9092 --wavefront.application.service=${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}-${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}} --management.metrics.tags.application.guid=${spring.cloud.application.guid:unknown} --groovy-processor.script=\"import groovy.json.JsonSlurper; import groovy.json.JsonOutput; try { println payload.getClass(); def jsonSlurper = new JsonSlurper(); def obj = jsonSlurper.parseText(payload); obj.custom_field='NA'; return JsonOutput.toJson(obj); } catch(Exception e){ throw e }\" --management.metrics.export.prometheus.rsocket.port=7001 --management.metrics.tags.application.name=${vcap.application.application_name:${spring.cloud.dataflow.stream.app.label:unknown}} --management.metrics.export.prometheus.rsocket.enabled=true --management.metrics.export.prometheus.rsocket.host=prometheus-rsocket-proxy --spring.cloud.stream.kafka.binder.zknodes=zookeeper:2181 --wavefront.application.name=${spring.cloud.dataflow.stream.name:unknown} --spring.kafka.streams.properties.metrics.recording.level=DEBUG --spring.cloud.stream.kafka.binder.brokers=PLAINTEXT://broker:9092 > :test2",
"originalDslText": ":test1 > test-groovy-script: groovy --script=\"import groovy.json.JsonSlurper; import groovy.json.JsonOutput; try { println payload.getClass(); def jsonSlurper = new JsonSlurper(); def obj = jsonSlurper.parseText(payload); obj.custom_field='NA'; return JsonOutput.toJson(obj); } catch(Exception e){ throw e }\" > :test2",
"description": "",
"status": "DEPLOYED"
}
]
}Stream gets deployed successfully and below stack trace we get when a message is sent to source topic (e.g. test1 in this stream) |
That's not correct.
Reference to some file, but definitely not an in-line definition. Please, revise your configuration respectively. |
We have modified our streams to use groovy and looks better performing compared to javascript. |
|
Happy New Year, @Noushath ! Thank you for confirming that JavaScript engine is indeed a bottleneck in the whole solution. Re. message headers: yes, it is possible to add/remove/replace headers from script as long as you return the whole new I think we can close this issue from now on since we all agreed that JavaScript engine was not a good choice for target solution. Thank you everyone for support and discussions! |
Hello Team,
We are using script processor application in our stream definition. We have seen that script application is processing records at very slow rate. We observed that CPU is utilized 100% on that application. We had deployed stream with default CPU (500m) and MEMORY (1Gi) settings. MEMORY utilization was under limit.
However we tried to increase the CPU limit (2000m) for script application and we saw that script application was performing better than previous deployment. This time CPU utilization was around 60% only.
Just wanted to know are there any recommendation for performance improvement. I am attaching screen shots just for reference.
The text was updated successfully, but these errors were encountered: