Implement initial basic crawler for extracting structred relations from PO.DAAC

[lewismc]

As discussed at our first meeting, we should use the Any23 basic-crawler to implement a lightweight crawler which will run Any23 extractions over Webpages.
I will link a WIP implementation which we can build on.

[Yongyao]

@lewismc , it just came to my mind that the results that Any23 will get is like "DOI:XXX, shortName:xxx,......", which is pretty much the same as metadata, if I understand correctly. People would argue that why don't we just use PO.DAAC web service to do it.

[lewismc]

The reason being is that initial, first order extractions are just the beginning for ESKG :)
There is plenty more work to do. This is only s start @Yongyao there is loads more to do.

[lewismc]

Lets walk before we can run!

[lewismc]

Another commit made and the new crawler command line looks as follows

usage: ESKGCrawler
    --maxDepth <mDepth>          Max allowed crawler depth.
    --maxPages <mPages>          Max number of pages before interrupting
                                 crawl.
    --numCrawlers <nCrawlers>    Sets the number of crawlers.
    --pageFilter <filter>        Regex used to filter out page URLs during
                                 crawling.
    --politenessDelay <pDelay>   Politeness delay in milliseconds.
    --seedUrl <seed>             An individual seed URL used to bootstrap
                                 the crawl.
    --storageFolder <storage>    Folder used to store crawler temporary
                                 data.

The only parameter which is mandatory is --seedUrl

[Yongyao]

Great. Just realized the basic crawler is based on crawler4j.

[lewismc]

ack