Skip to content

Distributed and easy to use

Choose a tag to compare

View all tags
@sibiryakov sibiryakov released this 30 Dec 20:23
· 656 commits to master since this release
v0.4.0
be3a952

A tremendous work was done:

  • distributed-frontera and frontera were merged together into the single project: to make it easier to use and understand,
  • Backend was completely redesigned. Now it's consisting of Queue, Metadata and States objects for low-level code and higher-level Backend implementations for crawling policies,
  • Added definition of run modes: single process, distributed spiders, distributed spider and backend.
  • Overall distributed concept is now integrated into Frontera, making difference between usage of components in single process and distributed spiders/backend run modes clearer.
  • Significantly restructured and augmented documentation, addressing user needs in a more accessible way.
  • Much less configuration footprint.

Enjoy this new year release and let us know what you think!

Assets 2
Loading