Project reorganization

[jakekdodd]

Given our discussion in #24, and previous discussions on Kafka spouts, HBase indexing, etc, we should think about reorganizing the project so that we have a core SDK and external SDK(s).

I was thinking something like

root
     pom.xml
    |-> crawler-core
        pom.xml
    |-> crawler-external
        pom.xml

The external sub-project would include things that depend on external technologies and libraries.

[jnioche]

or do as discussed before have a completely separate project for this [https://github.com/DigitalPebble/spiderling] and provide a crawler that would be fully functional and ready to use.

I was thinking of a 100% ElasticSearch pipeline where we'd store the url status, content to index, metrics and maybe even the logs. We'd provide schemas for ES (if necessary) as well as Kibana.

We could maybe do both? We already have a few things in the project that are related to external resources (ES for instance).

Storm do something similar [https://github.com/apache/storm/tree/master/external] but have one pom per external resource

[jakekdodd]

We could maybe do both? We already have a few things in the project that are related to external resources (ES for instance).

but have one pom per external resource

This seems like the best option to me. The Elasticsearch bolt would come out of crawler-core, and we'd have a crawler-external subproject with several subprojects itself (one would be crawler-elasticsearch, which would include the indexing bolt, a metrics consumer, maybe other things; another would be crawler-redis, which would include a Redis spout, cache, metrics writer, etc).

[jnioche]

The only concern I have is to spread ourselves too thin and have to maintain loads of resources related to external projects. My experience with Apache GORA and Nutch 2 has been quite a traumatic one from that perspective. What I like about Storm is that people can maintain their own spouts and bolts while leveraging our core components : it's all in the pom.xml as dependencies.
It will take some effort to make sure that the external part of the project does not become a dumping ground. I also means that we'll publish a lot more artefacts, but since this should be automatic it should not mean any additional work.

[jakekdodd]

Yeah I hear you on that.

How about I use the spiderling project as a platform to demonstrate some of the externally-dependent spouts and bolts? I'll create several example projects that leverage Redis, Kafka, and some other stuff. Then, if we decide that any of those elements would be generally useful, we can pull them into the core project.

[jnioche]

I still think it would make sense to follow your suggestion and have an external directory as some of our stuff could already go there. +1 to use spiderling as a testing ground but I would then pull the most useful things from there into external and maybe not core. What do you think?

[jakekdodd]

Sounds good to me. What should we name them, sc-core and sc-external?

[jnioche]

The directories can simply be core and external. As for the artefacts in pom we could leave core as is and use a suffix for external?

[jakekdodd]

Yep that works!

How do we want to organize packages in the external directory? We can make it component-oriented (like com.digitalpebble.storm.crawler.indexing.elasticsearch.*) or dependency-oriented (like com.digitalpebble.storm.crawler.elasticsearch.*)

[jnioche]

Would do the latter.

[jakekdodd]

Ok great, let's do this after we resolve the pending PRs, so we don't have to resolve merge conflicts for those

[jnioche]

@DigitalPebble/committers-crawler shall we proceed with this and then release 0.4? Anything else we'd want to put in the release?

[GuiForget]

I'm ok with it. The other option is to make 0.5 be just this change + the merge of metadata.