CamelFileSink Connector appendChars = \n not working

[ChhatralaNirmal]

Hello, We are using CamelFileSink connector to read the messages from Kafka and dump into file but we want each messages to be written in next line in file. We have been trying to use appendChars option with double slash \n but it doesn't work.

Here is our file sink url :
file:/user/app/folder?appendChars=\n&autoCreate=true&fileExist=Append&fileName=messages-${date:now:yyyyMMdd}.txt&forceWrites=true&charset=utf-8

Here is what the error we get :
Caused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: file:/user/app/folder?appendChars=\n&autoCreate=true&fileExist=Append&fileName=messages-${date:now:yyyyMMdd}.txt&forceWrites=true&charset=utf-8 due to: Illegal character in query at index 34: file:/user/app/folder?appendChars=\n&autoCreate=true&fileExist=Append&fileName=messages-$%7Bdate:now:yyyyMMdd%7D.txt&forceWrites=true&charset=utf-8\n\tat org.apache.camel.support.EndpointHelper.normalizeEndpointUri(EndpointHelper.java:156)\n\tat org.apache.camel.support.NormalizedUri.newNormalizedUri(NormalizedUri.java:46)\n\tat

[orpiske]

Hello,

Please, can you try defining the URLs parameters as properties and let us know the result?

Such as:

camel.sink.path.directoryName=/user/app/folder/
camel.sink.endpoint.appendChars=\n
camel.sink.endpoint.autoCreate=true
...

The full list of properties is here: https://camel.apache.org/camel-kafka-connector/latest/reference/connectors/camel-file-kafka-sink-connector.html

[ChhatralaNirmal]

We tried using the properties instead of file sink url and the result is same. It gives us the same error.

Here is the error messages after defining the properties :

\nCaused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: file:/user/app/folder?appendChars=\n&autoCreate=true&charset=UTF-8&fileExist=Append&fileName=messages-${date:now:yyyyMMdd}.txt&forceWrites=true due to: Illegal character in query at index 34: file:/user/app/folder?appendChars=\n&autoCreate=true&charset=UTF-8&fileExist=Append&fileName=messages-$%7Bdate:now:yyyyMMdd%7D.txt&forceWrites=true\n\tat

Here is our full connect config :

{
"name": "camel-file-sink-connector",
"connector.class": "org.apache.camel.kafkaconnector.file.CamelFileSinkConnector",
"tasks.max": "1",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"topics": "camel-file-sink-kafka-topic",
"camel.sink.contentLogLevel": "DEBUG",
"camel.idempotency.enabled": "false",
"camel.sink.path.directoryName": "/user/app/folder",
"camel.sink.endpoint.charset": "UTF-8",
"camel.sink.endpoint.fileName": "messages-${date:now:yyyyMMdd}.txt",
"camel.sink.endpoint.appendChars": "\n",
"camel.sink.endpoint.fileExist": "Append",
"camel.sink.endpoint.forceWrites": "true",
"camel.sink.endpoint.autoCreate": "true"
}

[orpiske]

Can you try escaping the new line char, please? Such as \\n.

[ChhatralaNirmal]

We did that too but no change in result. same error.
"camel.sink.endpoint.appendChars": "\n",

\nCaused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: file:/user/app/folder?appendChars=\n&autoCreate=true&charset=UTF-8&fileExist=Append&fileName=messages-${date:now:yyyyMMdd}.txt&forceWrites=true due to: Illegal character in query at index 34: file:/user/app/folder?appendChars=\n&autoCreate=true&charset=UTF-8&fileExist=Append&fileName=messages-$%7Bdate:now:yyyyMMdd%7D.txt&forceWrites=true\n\

[orpiske]

Thanks for double checking. I'll investigate this today (or, at worst, in a few days).

[orpiske]

@ChhatralaNirmal I've run a test with the append char as %0A instead of \\n as it seems it is not encoded correctly when that is read from the settings. Are you able to validate that on your end, please?

[ChhatralaNirmal]

@orpiske - thank you so much for this advise. It worked perfectly fine and we were able to get the messages on next line.
Should we expect the documentation be updated with this so it could help many others , or will it get fixed to use "\n" as you pointed out it is not encoded properly ?
Once again, thanks a lot for digging out and pointing us to use the right value instead.

[orpiske]

@orpiske - thank you so much for this advise. It worked perfectly fine and we were able to get the messages on next line.
Should we expect the documentation be updated with this so it could help many others , or will it get fixed to use "\n" as you pointed out it is not encoded properly ?
Once again, thanks a lot for digging out and pointing us to use the right value instead.

I am glad it worked.

@oscerd / @valdar I am a bit conflicted about this one: I'm wondering if we should fix the docs or the code. Any ideas?

[valdar]

@orpiske

Is the "correct" value "\n" ?

What would be the fix in the code?

[orpiske]

@orpiske

Is the "correct" value "\n" ?

What would be the fix in the code?

Long story short is this: when using special chars for appendChars on camel-file, it requires us to escape the parameters (i.e.: \\n for appending new lines). However, it seems to me that because the way the configuration works on Kafka connect, the escaping doesn't work. Instead it requires this to be sent encoded as %0A from our config so it can be properly interpreted as a newline.

Therefore, I'm wondering if we should somehow fix this in CKC code or just update our documentation to instruct users to use %0A and others for special chars.

[orpiske]

I documented this in multiple places so that users have a greater chance to find out the information. Therefore, I am closing this one. Please refer to PR #1220 for details about the changes.