Is your feature request related to a problem? Please describe.
According to input chunk documentation and the source code, in->storage_type is mutually exclusive with memory and filesystem.
People may not want to use the filesystem due to securityContext.readOnlyRootFilesystem policy enforcement. So they prefer to use memory. We may want to use filesystem to get more resiliency during incidents or network problems.
Memory is a limited space, which makes it easy to fill. But it's much faster and no I/O required.
The use case here is we don't want any log lost/drop issues. It's about to create eventually consistent architecture. We want that log to be redirected to the output, no matter what.
Describe the solution you'd like
Add a new option called both (or something) to in->storage_type. It should use memory to read/create the chunks until memory buffer limit gets exceeded, like as-is. When ring-buffer reached the limit, then it should start to use filesystem as a fallback mechanism. But should not overwrite/clear the memory buffer.
Open Question:
- What if filesystem also reached the limit? By default, if it has less than 1% of it capacity as available space, fd will be released. (overwrite?)
- In case container/pod restarts, both memory and file systems will be reset. (See the below: remote storage idea)
Describe alternatives you've considered
Remote write/read for chunks. i.e. remote storage solutions for better resiliency.
Additional context
-
Is your feature request related to a problem? Please describe.
According to input chunk documentation and the source code,
in->storage_typeis mutually exclusive withmemoryandfilesystem.People may not want to use the
filesystemdue tosecurityContext.readOnlyRootFilesystempolicy enforcement. So they prefer to usememory. We may want to usefilesystemto get more resiliency during incidents or network problems.Memory is a limited space, which makes it easy to fill. But it's much faster and no I/O required.
The use case here is we don't want any log lost/drop issues. It's about to create eventually consistent architecture. We want that log to be redirected to the output, no matter what.
Describe the solution you'd like
Add a new option called
both(or something) toin->storage_type. It should use memory to read/create the chunks until memory buffer limit gets exceeded, like as-is. When ring-buffer reached the limit, then it should start to use filesystem as a fallback mechanism. But should not overwrite/clear the memory buffer.Open Question:
Describe alternatives you've considered
Remote write/read for chunks. i.e. remote storage solutions for better resiliency.
Additional context
-