how to receive ndjson?

[Calemy]

What are you trying to achieve, or the steps to reproduce?

I'm trying to receive ndjson from meilisearch's new webhook feature which states the following in documentation
"he request body will be one or more task objects in ndjson format"
everything i do is currently not working and resulting in me getting a 415. i've been searching for parsers but had no luck.

Context

• node version: 18.18
• fastify version: >= 4.26.1
• os: Linux Ubuntu 22.04

[mcollina]

What content-type is Mellisearch sending ndjson as?

[Calemy]

sending currently as undefined for which i already opened an issue of meilisearch's repository, which i change to "application/x-ndjson" on the onRequest hook

[mcollina]

Something like this would work:

const split = require('split2')
fastify.addContentTypeParser('application/x-ndjson', function (request, payload, done) {
  const logs = []

  // handle errors in split
  request.pipe(split(JSON.parse))
    .on('data', (log) => logs.push(log))
    .on('end', done)
    .on('error', done)
})

[Calemy]

i get request.pipe is not a function, did i forget to add anything?

import "dotenv/config"
import { fastify as f } from "fastify"
import split from "split2"
const fastify = f({ logger: true, trustProxy: true })

fastify.addContentTypeParser('application/x-ndjson', function (request, payload, done) {
  const logs = []

  // handle errors in split
  request.pipe(split(JSON.parse))
    .on('data', (log) => logs.push(log))
    .on('end', done)
    .on('error', done)
})
fastify.addHook("onRequest", (req, reply, done) => {
    //* MeiliSearch please add a content-type :(
    if(req.headers["user-agent"].includes("ureq/")){
        req.headers["content-type"] = "application/x-ndjson"
    }
    done()
})


fastify.get("/", (req, reply) => {
    return "hello :3"
})

fastify.post("/api/update", (req, reply) => {
    return "ok"
})

fastify.listen({ host: "0.0.0.0", port: process.env.port })

[Eomm]

Try payload.pipe(split(JSON.parse))... or request.raw

[Calemy]

well now it's just that the json parser can't parse it properly:
"Unexpected token \u001f in JSON at position 0"

[mcollina]

Well, that does not look like encoded in ndjson.

[Calemy]

seems that the encoding is in gzip that contains ndjson.
I've written another example in the Meilisearch repository (meilisearch/meilisearch#4449).
Maybe i've made an error somewhere but i really do not know what could be the issue.

[mcollina]

Then you need to add https://www.npmjs.com/package/@fastify/compress if they set the correct content-type headers.

[Calemy]

i already did that exact thing like i did in the example written in the meilisearch repository.
it's still throwes me an undefined body. am i using it wrong?

[mcollina]

paste your code

[Calemy]

import "dotenv/config"
import { fastify as f } from "fastify"
import compress from "@fastify/compress"
const fastify = f({ logger: true, trustProxy: true })

fastify.register(compress) //adds support for gzip

fastify.addContentTypeParser('application/x-ndjson', function (request, payload, done) {
    done()
})
fastify.addHook("onRequest", (req, reply, done) => {
    //* MeiliSearch please add a content-type :(
    if(req.headers["user-agent"].includes("ureq/")){
        req.headers["content-type"] = "application/x-ndjson"
    }
    done()
})

fastify.post("/api/update", (req, reply) => {
    console.log(req.body) // returns undefined
    return "ok"
})

fastify.listen({ host: "0.0.0.0", port: process.env.port }

[mcollina]

You still need the ndjson parsing logic. You've removed that.

[Calemy]

what exactly am i supposed to parse and how? the body is undefined and everything else mentioned here tells me that this is not the right format