Integrate URLFrontier as a backend for URL storage

[jnioche]

Hi,

This is a question, not a bug report.

url-frontier is an API to define a crawl frontier. It uses gRPC and has a service implementation. It is crawler-neutral and is suitable to being used in a distributed environment.

How could URLFrontier fit within Webmagic? Is there a similar abstraction over the URL storage? Is there something missing in URLFrontier's API?

Thanks