org.apache.avro.SchemaParseException: Illegal character in: query-result

[nicoloboschi]

When using JSON (not AVRO!) schema, some agents will apply the result using a field name configured in the pipeline.
In many examples there is query-result, however AVRO doesn't support field names with dashes in it. To change the JSON schema, we use avro internally so then this happen:

[convhisthuman-pipeline-query-1-0] 08:01:12.765 [main] ERROR a.l.runtime.agent.AgentRunner -- Unrecoverable error while processing some the records, failing
[convhisthuman-pipeline-query-1-0] java.util.concurrent.CompletionException: org.apache.avro.SchemaParseException: Illegal character in: query-result
[convhisthuman-pipeline-query-1-0]      at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
[convhisthuman-pipeline-query-1-0]      at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:674)
[convhisthuman-pipeline-query-1-0]      at java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:662)
[convhisthuman-pipeline-query-1-0]      at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2168)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.ai.agents.GenAIToolKitAgent.processRecord(GenAIToolKitAgent.java:118)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.ai.agents.GenAIToolKitAgent.process(GenAIToolKitAgent.java:93)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.api.runner.code.AgentCodeAndLoader$3.lambda$process$0(AgentCodeAndLoader.java:208)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.api.runner.code.AgentCodeAndLoader.executeNoExceptionWithContextClassloader(AgentCodeAndLoader.java:56)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.api.runner.code.AgentCodeAndLoader$3.process(AgentCodeAndLoader.java:207)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.runtime.agent.CompositeAgentProcessor.invokeProcessor(CompositeAgentProcessor.java:165)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.runtime.agent.CompositeAgentProcessor.process(CompositeAgentProcessor.java:225)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.api.runner.code.AgentCodeAndLoader$3.lambda$process$0(AgentCodeAndLoader.java:208)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.api.runner.code.AgentCodeAndLoader.executeNoExceptionWithContextClassloader(AgentCodeAndLoader.java:56)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.api.runner.code.AgentCodeAndLoader$3.process(AgentCodeAndLoader.java:207)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.runtime.agent.AgentRunner.runProcessorAgent(AgentRunner.java:790)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.runtime.agent.AgentRunner.runMainLoop(AgentRunner.java:602)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.runtime.agent.AgentRunner.runJavaAgent(AgentRunner.java:370)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.runtime.agent.AgentRunner.run(AgentRunner.java:179)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.runtime.agent.AgentRunnerStarter.start(AgentRunnerStarter.java:111)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.runtime.agent.AgentRunnerStarter.main(AgentRunnerStarter.java:53)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.runtime.Main.main(Main.java:42)
[convhisthuman-pipeline-query-1-0] Caused by: org.apache.avro.SchemaParseException: Illegal character in: query-result
[convhisthuman-pipeline-query-1-0]      at org.apache.avro.Schema.validateName(Schema.java:1566)
[convhisthuman-pipeline-query-1-0]      at org.apache.avro.Schema.access$400(Schema.java:91)
[convhisthuman-pipeline-query-1-0]      at org.apache.avro.Schema$Field.<init>(Schema.java:546)
[convhisthuman-pipeline-query-1-0]      at org.apache.avro.Schema$Field.<init>(Schema.java:585)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.ai.agents.commons.TransformContext.setResultField(TransformContext.java:312)
[convhisthuman-pipeline-query-1-0]      at com.datastax.oss.streaming.ai.QueryStep.process(QueryStep.java:101)
[convhisthuman-pipeline-query-1-0]      at com.datastax.oss.streaming.ai.TransformStep.processAsync(TransformStep.java:41)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.ai.agents.GenAIToolKitAgent.processStep(GenAIToolKitAgent.java:139)
[convhisthuman-pipeline-query-1-0]      at ai.langstream.ai.agents.GenAIToolKitAgent.processRecord(GenAIToolKitAgent.java:117)
[convhisthuman-pipeline-query-1-0]      ... 16 common frames omitted

[eolivelli]

We should add a validation at planner/deploy time.
Avro doesn't allow all the possible characters (for instance the '-' character.

Currently we are alwaysing trying to also maintain the Avro schema, maybe an alternative solution would be to not do it if the input object is not a AVRO object (we will also some CPU cyclesl

@cbornet

[cbornet]

I'm surprised that we use a Schema with Kafka's JSON 🤔
This should only happen with Pulsar.

[eolivelli]

@nicoloboschi I cannot reproduce your problem
#616