Export in database - what would be the way to do it?

[panthony]

Thanks for this nice project,

I'm playing with it and now I'm trying to store the result of a crawl within a database and I noticed that the BaseExporter expect the export to be in a file.

And I noticed that you had DB export in mind (#15).

So what would be the proper way?

Use the onSuccess callback? Or have a BaseExporter agnostic from the support (and have a FileExporter on top).

[yujiosaka]

@panthony
Good point.
I suggest you using onSuccess callback now because It's way simpler.

I'd like to start thinking about making exporters to DB.
What kind of DBs you have in mind? I believe it should be schema-less, so the main candidate would be MongoDB for example.

[panthony]

@yujiosaka For my use case I had Bigtable in mind.

Ok, for now I'll just try with the onSuccess

[BubuAnabelas]

@panthony I'm not familiar with Bigtable, but since it's NoSQL perhaps JSONLineExporter could help. It writes to a file but since it's done through writable streams maybe it can be changed to pipe it to Bigtable.

[yujiosaka]

Closing this because the problem seems to be solved.
Feel free to open a new issue if you'd like support features to store crawled results to DBs by exporters