processing pipeline example

[eklem]

See if I can get the Chinese tokenizer working in the add pipeline.

[eklem]

@fergiemcdowall, could you explain a little how a module would fit into the processing pipeline? It could take a URL key/value and duplicate it to also become an ID key with the same value.

[eklem]

Then I'll make some examples and create some documentation.

[fergiemcdowall]

Yes, you could do something like:

fs.createReadStream('myData')
  .pipe(myAmazingProcessingStage)
  .pipe(index.feed())

[eklem]

I'll test. myAmazingProcessingStage doesn't need to be written streamy, or do it? I guess I have no idea what I'm heading into =)

Espen

[fergiemcdowall]

Its not toooo ridiculous- they are just transform streams. Here are some examples: https://github.com/fergiemcdowall/docproc/tree/master/pipeline

[eklem]

Thanks, I'll test based on the IngestDoc.js. It seems doable.

[eklem]

Or, maybe Spy.js is the easiest to base it on, @fergiemcdowall ?

[fergiemcdowall]

Yes, and you would of course do something with doc before pushing it back into the pipeline

[eklem]

Thanks. I can do this =)
I'll leave it open since its a documentation task and not just a question. I'll test and then document.

[fergiemcdowall]

I've added this to the list: #458