Is there any "length limitation" in parse_json function?

[iamdobi]

Hello, I have a question regarding the message size and parse_json function.

First, I am trying to send log message as a json format, with exception field which contains whole stacktraces emitted by spring boot project.

I found strange things when I worked with transform with rvl.

This my configurations:

[sources.generate_syslog]
  type = "stdin"

[transforms.remap_syslog]
  inputs = [ "generate_syslog"]
  type = "remap"
  source = '''
  ¦ structured = parse_json!(.message)
  ¦ ., err = merge(., structured)
  '''

[sinks.emit_syslog]
  inputs = ["remap_syslog"]
  type = "console"
  encoding.codec = "json"

I think there's no special things to check.

The weird part is this.

If the message size is larger than "4095" bytes, then vector says like this:

Jul 15 03:22:20.806  WARN transform{component_kind="transform" component_name=remap_syslog component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for \"parse_json\" at (17:38): unable to parse json: EOF while parsing an object at line 1 column 4095" internal_log_rate_secs=30

I tried with this message:

{"exception":{"exception_class":"org.springframework.data.redis.RedisConnectionFailureException","exception_message":"Unable to connect to Redis; nested exception is io.lettuce.core.RedisConnectionException: Unable to connect to engine2-dev-01001-rd-jp2v-dev.lineinfra-dev.com/<unresolved>:7000","stacktrace":"1--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------"},"@version":1,"source_host":"AL01922436","message":{"formattedMessage":"Redis health check failed"},"thread_name":"boundedElastic-2","@timestamp":"2021-07-14T21:01:39.865+09:00","level":"WARN","logger_name":"org.springframework.boot.actuate.redis.RedisReactiveHealthIndicator"}

If I delete 1 character in any string, then I is accepted with vector.

It seems that there may be some limit with size of input message, right?

My question is:

1. I want to know whether this is a intended spec or not.
2. Is there any way to workaround this? (because the size of stacktrace string is easily larger than 4095 bytes..)

I tried to search and find regarding this issue but no result. 😢

[StephenWakely]

So the limit here is actually in the terminal. When you paste a line with a length > 4096 the tty will truncate the string, so Vector never actually gets the full string.

There is no limit, beyond available RAM, to the length of string that parse_json can parse.

You should find the transform parses correctly if you save the json to a file and pipe that to Vector:

./bin/vector -c conf.toml  < input.txt

[iamdobi]

Ah!!! got it!
I'm sorry to ask you such a stupid question. T.T
Then I'm gonna go on happy "vectoring".

Thank you for your rapid answer. 😄

[StephenWakely]

Not a stupid question at all! I also learned something whilst looking into it.