NDJSON special support

[phretor]

Hello!

I feel that with FSCrawler my search for the perfect personal search engine is about to come to a successful end: it's exactly what I've been looking for.

I use NDJSON a lot to lump together different JSON files and reduce the open-close file requests at query time, and I was wondering if FSCrawler is able to handle it as a special case of JSON file, for which I see there is support already.

Speaking in ElasticSearch terminology, it would probably be mapped as a bulk request.

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk-api-desc (see here for NDJSON and bulk)

If that doesn't work out of the box, I'm happy to contribute some code to implement it, if someone could pinpoint the best place to plug this functionality in.

Thanks!

[dadoonet]

Hey @phretor

Thanks for the proposal. That sounds a good idea to me.
I believe it should be a similar option as https://fscrawler.readthedocs.io/en/latest/admin/fs/local-fs.html#indexing-json-docs

name: "test"
fs:
  ndjson_support: true

About the implementation, you can start from here I think.

fscrawler/core/src/main/java/fr/pilato/elasticsearch/crawler/fs/FsParserAbstract.java

Lines 454 to 468 in a6be178

	if (fsSettings.getFs().isJsonSupport()) {
	FSCrawlerLogger.documentDebug(generateIdFromFilename(filename, dirname),
	computeVirtualPathName(stats.getRootPath(), fullFilename),
	"Indexing json content");
	// We index the json content directly
	if (!closed) {
	documentService.indexRawJson(
	fsSettings.getElasticsearch().getIndex(),
	id,
	read(inputStream),
	fsSettings.getElasticsearch().getPipeline());
	} else {
	logger.warn("trying to add new file while closing crawler. Document [{}]/[{}] has been ignored",
	fsSettings.getElasticsearch().getIndex(), id);
	}

The XML support is more helpful to get inspiration from I believe:

fscrawler/core/src/main/java/fr/pilato/elasticsearch/crawler/fs/FsParserAbstract.java

Lines 469 to 484 in a6be178

	} else if (fsSettings.getFs().isXmlSupport()) {
	FSCrawlerLogger.documentDebug(generateIdFromFilename(filename, dirname),
	computeVirtualPathName(stats.getRootPath(), fullFilename),
	"Indexing xml content");
	// We index the xml content directly (after transformation to json)
	if (!closed) {
	documentService.indexRawJson(
	fsSettings.getElasticsearch().getIndex(),
	id,
	XmlDocParser.generate(inputStream),
	fsSettings.getElasticsearch().getPipeline());
	} else {
	logger.warn("trying to add new file while closing crawler. Document [{}]/[{}] has been ignored",
	fsSettings.getElasticsearch().getIndex(), id);
	}
	}

May be copying the "skeleton" from this class would help?

https://github.com/dadoonet/fscrawler/blob/master/tika/src/main/java/fr/pilato/elasticsearch/crawler/fs/tika/XmlDocParser.java

LMK if you need more information. And thanks a lot for proposing your help!