Broken json filters

[Homulvas]

Query

{namespace="X"} | json | resources_some_resource!=`something`

Results in a complete crash of the application with the following error:

DEFAULT 2024-02-13T12:35:16.215204948Z {"err":"premature close Error: premature close at onclosenexttick (/app/node_modules/end-of-stream/index.js:54:86) at process.processTicksAndRejections (node:internal/process/task_queues:77:11)", "hostname":"name", "level":50, "msg":"premature close", "name":"qryn", "pid":25, "reqId":"req-112i", "time":1.707827716215E12}
DEFAULT 2024-02-13T12:35:16.439535971Z node:events:496
DEFAULT 2024-02-13T12:35:16.439601792Z throw er; // Unhandled 'error' event
DEFAULT 2024-02-13T12:35:16.439610332Z ^
DEFAULT 2024-02-13T12:35:16.439615708Z
DEFAULT 2024-02-13T12:35:16.439622202Z <ref *1> StreamError: Cannot read properties of undefined (reading 'resources_some_resource')
DEFAULT 2024-02-13T12:35:16.439628783Z at /app/node_modules/scramjet-core/lib/util/mk-transform.js:59:44
DEFAULT 2024-02-13T12:35:16.439635736Z at runNextTicks (node:internal/process/task_queues:60:5)
DEFAULT 2024-02-13T12:35:16.439642277Z at process.processImmediate (node:internal/timers:449:9)
DEFAULT 2024-02-13T12:35:16.439647975Z caused by:
DEFAULT 2024-02-13T12:35:16.439654741Z at DataStream.<anonymous> (/app/parser/registry/stream_selector_operator_registry/stream_selector_operator_registry.js:121:43)
DEFAULT 2024-02-13T12:35:16.439662120Z ... 3 lines matching cause stack trace ...
DEFAULT 2024-02-13T12:35:16.439668725Z at async Promise.all (index 0)
DEFAULT 2024-02-13T12:35:16.439674368Z --- raised in DataStream(412) constructed ---
DEFAULT 2024-02-13T12:35:16.439680714Z at new PromiseTransformStream (/app/node_modules/scramjet-core/lib/util/promise-transform-stream.js:65:27)
DEFAULT 2024-02-13T12:35:16.439686395Z at new DataStream (/app/node_modules/scramjet-core/lib/data-stream.js:43:9)
DEFAULT 2024-02-13T12:35:16.439693127Z at StringStream.map (/app/node_modules/scramjet-core/lib/data-stream.js:197:26)
DEFAULT 2024-02-13T12:35:16.439699046Z at preprocessStream (/app/lib/db/clickhouse.js:832:6)
DEFAULT 2024-02-13T12:35:16.439704963Z at queryFingerprintsScan (/app/lib/db/clickhouse.js:384:22)
DEFAULT 2024-02-13T12:35:16.439713280Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
DEFAULT 2024-02-13T12:35:16.439718661Z at async Object.handler (/app/lib/handlers/query_range.js:41:22) {
DEFAULT 2024-02-13T12:35:16.439724408Z chunk: '',
DEFAULT 2024-02-13T12:35:16.439730542Z stream: DataStream {
DEFAULT 2024-02-13T12:35:16.439736219Z _events: {
DEFAULT 2024-02-13T12:35:16.439741931Z close: undefined,
DEFAULT 2024-02-13T12:35:16.439746988Z error: undefined,
DEFAULT 2024-02-13T12:35:16.439752162Z prefinish: [Function: prefinish],
DEFAULT 2024-02-13T12:35:16.439757641Z finish: undefined,
DEFAULT 2024-02-13T12:35:16.439763112Z drain: undefined,
DEFAULT 2024-02-13T12:35:16.439769297Z data: undefined,
DEFAULT 2024-02-13T12:35:16.439774806Z end: undefined,
DEFAULT 2024-02-13T12:35:16.439781143Z readable: undefined,
DEFAULT 2024-02-13T12:35:16.439786819Z unpipe: undefined
DEFAULT 2024-02-13T12:35:16.439791952Z },
DEFAULT 2024-02-13T12:35:16.439797777Z _readableState: ReadableState {
DEFAULT 2024-02-13T12:35:16.439802943Z highWaterMark: 16,
DEFAULT 2024-02-13T12:35:16.439808193Z buffer: [],
DEFAULT 2024-02-13T12:35:16.439813630Z bufferIndex: 0,
DEFAULT 2024-02-13T12:35:16.439818929Z length: 0,
DEFAULT 2024-02-13T12:35:16.439824099Z pipes: [],
DEFAULT 2024-02-13T12:35:16.439829868Z awaitDrainWriters: null,
DEFAULT 2024-02-13T12:35:16.439836186Z [Symbol(kState)]: 1049023,
DEFAULT 2024-02-13T12:35:16.439847252Z [Symbol(kErroredValue)]: [Circular *1]
DEFAULT 2024-02-13T12:35:16.439853177Z },
DEFAULT 2024-02-13T12:35:16.439858569Z _writableState: WritableState {
DEFAULT 2024-02-13T12:35:16.439864508Z highWaterMark: 16,
DEFAULT 2024-02-13T12:35:16.439869902Z length: 0,
DEFAULT 2024-02-13T12:35:16.439875511Z corked: 0,
DEFAULT 2024-02-13T12:35:16.439881207Z onwrite: [Function: bound onwrite],
DEFAULT 2024-02-13T12:35:16.439886617Z writelen: 0,
DEFAULT 2024-02-13T12:35:16.439891997Z bufferedIndex: 0,
DEFAULT 2024-02-13T12:35:16.439920262Z pendingcb: 1,
DEFAULT 2024-02-13T12:35:16.439926316Z [Symbol(kState)]: 1091327423,
DEFAULT 2024-02-13T12:35:16.439972931Z [Symbol(kBufferedValue)]: null,
DEFAULT 2024-02-13T12:35:16.439980223Z [Symbol(kErroredValue)]: [Circular *1]
DEFAULT 2024-02-13T12:35:16.439985983Z },
DEFAULT 2024-02-13T12:35:16.439992027Z allowHalfOpen: true,
DEFAULT 2024-02-13T12:35:16.439997946Z _maxListeners: 26,
DEFAULT 2024-02-13T12:35:16.440003498Z _eventsCount: 1,
DEFAULT 2024-02-13T12:35:16.440009006Z _tapped: false,
DEFAULT 2024-02-13T12:35:16.440014448Z _error_handlers: [],
DEFAULT 2024-02-13T12:35:16.440022227Z _scramjet_options: {
DEFAULT 2024-02-13T12:35:16.440027706Z referrer: StringStream {
DEFAULT 2024-02-13T12:35:16.440033355Z _events: {
DEFAULT 2024-02-13T12:35:16.440039086Z close: undefined,
DEFAULT 2024-02-13T12:35:16.440044501Z error: [Function (anonymous)],
DEFAULT 2024-02-13T12:35:16.440050263Z prefinish: [Function: prefinish],
DEFAULT 2024-02-13T12:35:16.440055551Z finish: undefined,
DEFAULT 2024-02-13T12:35:16.440060946Z drain: undefined,
DEFAULT 2024-02-13T12:35:16.440066206Z data: undefined,
DEFAULT 2024-02-13T12:35:16.440071309Z end: undefined,
DEFAULT 2024-02-13T12:35:16.440076312Z readable: undefined,
DEFAULT 2024-02-13T12:35:16.440081809Z unpipe: undefined
DEFAULT 2024-02-13T12:35:16.440086652Z },
DEFAULT 2024-02-13T12:35:16.440091737Z _readableState: ReadableState {
DEFAULT 2024-02-13T12:35:16.440097269Z highWaterMark: 16,
DEFAULT 2024-02-13T12:35:16.440103405Z buffer: [],
DEFAULT 2024-02-13T12:35:16.440108943Z bufferIndex: 0,
DEFAULT 2024-02-13T12:35:16.440115150Z length: 0,
DEFAULT 2024-02-13T12:35:16.440121108Z pipes: [],
DEFAULT 2024-02-13T12:35:16.440126718Z awaitDrainWriters: null,
DEFAULT 2024-02-13T12:35:16.440132387Z [Symbol(kState)]: 110626685
DEFAULT 2024-02-13T12:35:16.440138176Z },
DEFAULT 2024-02-13T12:35:16.440144533Z _writableState: WritableState {
DEFAULT 2024-02-13T12:35:16.440150175Z highWaterMark: 16,
DEFAULT 2024-02-13T12:35:16.440155778Z length: 0,
DEFAULT 2024-02-13T12:35:16.440163358Z corked: 0,
DEFAULT 2024-02-13T12:35:16.440172760Z onwrite: [Function: bound onwrite],
DEFAULT 2024-02-13T12:35:16.440178975Z writelen: 0,
DEFAULT 2024-02-13T12:35:16.440185006Z bufferedIndex: 0,
DEFAULT 2024-02-13T12:35:16.440190443Z pendingcb: 0,
DEFAULT 2024-02-13T12:35:16.440195986Z [Symbol(kState)]: 1091466621,
DEFAULT 2024-02-13T12:35:16.440201651Z [Symbol(kBufferedValue)]: null
DEFAULT 2024-02-13T12:35:16.440207481Z },
DEFAULT 2024-02-13T12:35:16.440213954Z allowHalfOpen: true,
DEFAULT 2024-02-13T12:35:16.440219558Z _maxListeners: 26,
DEFAULT 2024-02-13T12:35:16.440225670Z _eventsCount: 2,
DEFAULT 2024-02-13T12:35:16.440264851Z _tapped: true,
DEFAULT 2024-02-13T12:35:16.440271822Z _error_handlers: [ [AsyncFunction (anonymous)] ],
DEFAULT 2024-02-13T12:35:16.440278245Z _scramjet_options: {
DEFAULT 2024-02-13T12:35:16.440284692Z referrer: StringStream {
DEFAULT 2024-02-13T12:35:16.440290695Z _events: [Object],
DEFAULT 2024-02-13T12:35:16.440296738Z _readableState: [ReadableState],
DEFAULT 2024-02-13T12:35:16.440302761Z _writableState: [WritableState],
DEFAULT 2024-02-13T12:35:16.440308611Z allowHalfOpen: true,
DEFAULT 2024-02-13T12:35:16.440314425Z _maxListeners: 26,
DEFAULT 2024-02-13T12:35:16.440320242Z _transform: [Function: transform],
DEFAULT 2024-02-13T12:35:16.440326657Z _flush: [Function: flush],
DEFAULT 2024-02-13T12:35:16.440336131Z _eventsCount: 2,
DEFAULT 2024-02-13T12:35:16.440344227Z _tapped: true,
DEFAULT 2024-02-13T12:35:16.440365479Z _error_handlers: [Array],
DEFAULT 2024-02-13T12:35:16.440372175Z _scramjet_options: [Object],
DEFAULT 2024-02-13T12:35:16.440378892Z seq: 410,
DEFAULT 2024-02-13T12:35:16.440384718Z type: 'Transform-',
DEFAULT 2024-02-13T12:35:16.440390763Z TimeSource: [Function: Date],
DEFAULT 2024-02-13T12:35:16.440397488Z setTimeout: [Function],
DEFAULT 2024-02-13T12:35:16.440404129Z clearTimeout: [Function: clearTimeout],
DEFAULT 2024-02-13T12:35:16.440410154Z buffer: '',
DEFAULT 2024-02-13T12:35:16.440415569Z encoding: 'utf8',
DEFAULT 2024-02-13T12:35:16.440421618Z [Symbol(shapeMode)]: true,
DEFAULT 2024-02-13T12:35:16.440427861Z [Symbol(kCapture)]: false,
DEFAULT 2024-02-13T12:35:16.440454587Z [Symbol(kCallback)]: null
DEFAULT 2024-02-13T12:35:16.440462294Z },
DEFAULT 2024-02-13T12:35:16.440468222Z constructed: 'Error\n' +
DEFAULT 2024-02-13T12:35:16.440474654Z ' at new PromiseTransformStream (/app/node_modules/scramjet-core/lib/util/promise-transform-stream.js:65:27)\n' +
DEFAULT 2024-02-13T12:35:16.440481736Z ' at new DataStream (/app/node_modules/scramjet-core/lib/data-stream.js:43:9)\n' +
DEFAULT 2024-02-13T12:35:16.440487764Z ' at new StringStream (/app/node_modules/scramjet-core/lib/string-stream.js:34:9)\n' +
DEFAULT 2024-02-13T12:35:16.440494283Z ' at StringStream._selfInstance (/app/node_modules/scramjet-core/lib/util/promise-transform-stream.js:344:16)\n' +
DEFAULT 2024-02-13T12:35:16.440500787Z ' at StringStream.endWith (/app/node_modules/scramjet/lib/data-stream.js:245:31)\n' +
DEFAULT 2024-02-13T12:35:16.440507461Z ' at preprocessStream (/app/lib/db/clickhouse.js:831:59)\n' +
DEFAULT 2024-02-13T12:35:16.440514875Z ' at queryFingerprintsScan (/app/lib/db/clickhouse.js:384:22)\n' +
DEFAULT 2024-02-13T12:35:16.440520357Z ' at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n' +
DEFAULT 2024-02-13T12:35:16.440526310Z ' at async Object.handler (/app/lib/handlers/query_range.js:41:22)',
DEFAULT 2024-02-13T12:35:16.440533015Z objectMode: true,
DEFAULT 2024-02-13T12:35:16.440539949Z promiseRead: null,
DEFAULT 2024-02-13T12:35:16.440545658Z promiseWrite: null,
DEFAULT 2024-02-13T12:35:16.440551483Z promiseTransform: [Function: promiseTransform],
DEFAULT 2024-02-13T12:35:16.440557137Z promiseFlush: [Function: flushPromise],
DEFAULT 2024-02-13T12:35:16.440562795Z beforeTransform: null,
DEFAULT 2024-02-13T12:35:16.440568527Z afterTransform: null,
DEFAULT 2024-02-13T12:35:16.440573978Z writableObjectMode: true,
DEFAULT 2024-02-13T12:35:16.440578988Z readableObjectMode: true,
DEFAULT 2024-02-13T12:35:16.440584567Z transforms: [ [Function] ],
DEFAULT 2024-02-13T12:35:16.440590488Z runFlush: [Function: flushPromise]
DEFAULT 2024-02-13T12:35:16.440596578Z },
DEFAULT 2024-02-13T12:35:16.440603355Z seq: 411,
DEFAULT 2024-02-13T12:35:16.440609401Z type: 'Transform',
DEFAULT 2024-02-13T12:35:16.440616110Z _transform: [Function (anonymous)],
DEFAULT 2024-02-13T12:35:16.440621287Z _flush: [Function (anonymous)],
DEFAULT 2024-02-13T12:35:16.440626908Z TimeSource: [Function: Date],
DEFAULT 2024-02-13T12:35:16.440632512Z setTimeout: [Function: setTimeout] {
DEFAULT 2024-02-13T12:35:16.440638515Z [Symbol(nodejs.util.promisify.custom)]: [Getter]
DEFAULT 2024-02-13T12:35:16.440643895Z },
DEFAULT 2024-02-13T12:35:16.440654543Z clearTimeout: [Function: clearTimeout],
DEFAULT 2024-02-13T12:35:16.440661418Z buffer: '',
DEFAULT 2024-02-13T12:35:16.440676446Z encoding: 'utf8',
DEFAULT 2024-02-13T12:35:16.440683096Z [Symbol(shapeMode)]: true,
DEFAULT 2024-02-13T12:35:16.440704166Z [Symbol(kCapture)]: false,
DEFAULT 2024-02-13T12:35:16.440710623Z [Symbol(kCallback)]: null
DEFAULT 2024-02-13T12:35:16.440716874Z },
DEFAULT 2024-02-13T12:35:16.440722488Z constructed: 'Error\n' +
DEFAULT 2024-02-13T12:35:16.440728582Z ' at new PromiseTransformStream (/app/node_modules/scramjet-core/lib/util/promise-transform-stream.js:65:27)\n' +
DEFAULT 2024-02-13T12:35:16.440734033Z ' at new DataStream (/app/node_modules/scramjet-core/lib/data-stream.js:43:9)\n' +
DEFAULT 2024-02-13T12:35:16.440740442Z ' at StringStream.map (/app/node_modules/scramjet-core/lib/data-stream.js:197:26)\n' +
DEFAULT 2024-02-13T12:35:16.440746483Z ' at preprocessStream (/app/lib/db/clickhouse.js:832:6)\n' +
DEFAULT 2024-02-13T12:35:16.440752875Z ' at queryFingerprintsScan (/app/lib/db/clickhouse.js:384:22)\n' +
DEFAULT 2024-02-13T12:35:16.440758941Z ' at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n' +
DEFAULT 2024-02-13T12:35:16.440765018Z ' at async Object.handler (/app/lib/handlers/query_range.js:41:22)',
DEFAULT 2024-02-13T12:35:16.440770930Z objectMode: true,
DEFAULT 2024-02-13T12:35:16.440777189Z promiseRead: null,
DEFAULT 2024-02-13T12:35:16.440783002Z promiseWrite: null,
DEFAULT 2024-02-13T12:35:16.440788795Z promiseTransform: [Function (anonymous)],
DEFAULT 2024-02-13T12:35:16.440794723Z promiseFlush: null,
DEFAULT 2024-02-13T12:35:16.440800314Z beforeTransform: null,
DEFAULT 2024-02-13T12:35:16.440807763Z afterTransform: null,
DEFAULT 2024-02-13T12:35:16.440813166Z writableObjectMode: true,
DEFAULT 2024-02-13T12:35:16.440817355Z readableObjectMode: true,
DEFAULT 2024-02-13T12:35:16.440821281Z transforms: [
DEFAULT 2024-02-13T12:35:16.440828258Z [Function: bound ] { ref: [Function (anonymous)] },
DEFAULT 2024-02-13T12:35:16.440832398Z [Function: bound ] { ref: [Function (anonymous)] },
DEFAULT 2024-02-13T12:35:16.440836422Z [Function: bound ] { ref: [Function (anonymous)] },
DEFAULT 2024-02-13T12:35:16.440840343Z [AsyncFunction (anonymous)] { ref: [Function (anonymous)] }
DEFAULT 2024-02-13T12:35:16.440844408Z ]
DEFAULT 2024-02-13T12:35:16.440848432Z },
DEFAULT 2024-02-13T12:35:16.440852232Z seq: 412,
DEFAULT 2024-02-13T12:35:16.440856226Z type: 'Transform',
DEFAULT 2024-02-13T12:35:16.440860378Z _transform: [Function (anonymous)],
DEFAULT 2024-02-13T12:35:16.440864166Z _flush: [Function (anonymous)],
DEFAULT 2024-02-13T12:35:16.440867875Z TimeSource: [Function: Date],
DEFAULT 2024-02-13T12:35:16.440871664Z setTimeout: [Function: setTimeout] {
DEFAULT 2024-02-13T12:35:16.440875573Z [Symbol(nodejs.util.promisify.custom)]: [Getter]
DEFAULT 2024-02-13T12:35:16.440879339Z },
DEFAULT 2024-02-13T12:35:16.440883213Z clearTimeout: [Function: clearTimeout],
DEFAULT 2024-02-13T12:35:16.440887075Z buffer: null,
DEFAULT 2024-02-13T12:35:16.440890991Z [Symbol(shapeMode)]: true,
DEFAULT 2024-02-13T12:35:16.440894849Z [Symbol(kCapture)]: false,
DEFAULT 2024-02-13T12:35:16.440898586Z [Symbol(kCallback)]: null
DEFAULT 2024-02-13T12:35:16.440902403Z },
DEFAULT 2024-02-13T12:35:16.440906242Z code: 'ERR_SCRAMJET_EXTERNAL',
DEFAULT 2024-02-13T12:35:16.440910416Z cause: TypeError: Cannot read properties of undefined (reading 'resources_some_resource')
DEFAULT 2024-02-13T12:35:16.440914383Z at DataStream.<anonymous> (/app/parser/registry/stream_selector_operator_registry/stream_selector_operator_registry.js:121:43)
DEFAULT 2024-02-13T12:35:16.440918347Z at /app/node_modules/scramjet-core/lib/util/promise-transform-stream.js:322:56
DEFAULT 2024-02-13T12:35:16.440922178Z at runNextTicks (node:internal/process/task_queues:60:5)
DEFAULT 2024-02-13T12:35:16.440926335Z at process.processImmediate (node:internal/timers:449:9)
DEFAULT 2024-02-13T12:35:16.440959826Z at async Promise.all (index 0),
DEFAULT 2024-02-13T12:35:16.440989803Z constructor: [class StreamError extends Error]
DEFAULT 2024-02-13T12:35:16.440996658Z }
DEFAULT 2024-02-13T12:35:16.441002687Z
DEFAULT 2024-02-13T12:35:16.441009219Z Node.js v20.11.0

I'm running v3.2.4

The resource label might not exist on all the log entries

Changing the query to {namespace="X"} | json resources_some_resource!="something" makes it work but the first query is generated by the Grafana UI "Filter out value" selectors so there is absolutely no reason why it shouldn't work.

[lmangani]

Thanks for the report @Homulvas 👍
We'll work out more uniform support of quoting and extend the CI to make sure its not missed

[akvlad]

Hello @Homulvas . are you using the bun version?

[Homulvas]

Hello @Homulvas . are you using the bun version?

I originally installed it from the helm chart. I assume that uses the NodeJS runtime. Should I try bun instead?

[akvlad]

@Homulvas I see the bug. Thanks for the issue.
Just for the information: using paremeterized json parser like | json resources_some_resource="resources.some_resource" | resources_some_resource!="something" is highly encouraged as it works much faster than the parameterless json parser

[Homulvas]

This becomes a bit of a usability issue then. I'm using Grafana with Loki source to query the logs and using the Grafana UI to filter based on the values from some log entry will not build a query using the parameterized json.

[akvlad]

@Homulvas Anyway, the reported issue should be fixed in 3.2.6.
Please recheck.

[asvasyanin]

Got same error in 3.2.6
Stacktrace: https://gist.github.com/asvasyanin/13d305881fb033b39e4a231f41d46d23

[akvlad]

@asvasyanin a bit different error. Will fix. Thanks for the report.

[akvlad]

@asvasyanin fixed in 3.2.7

[lmangani]

Closing as resolved. Feel free to reopen is needed 👍