Collection of generic docs and relevant resources: standards, examples, data sources etc.
(since nearly everything in here deals with data, the word data is intentionally omitted in definitions)
Any sort of raw data suppliers: platforms, services, publicly available APIs/DBs, aggregators over IoT sensors etc.
Essentially, a worker (COULD be daemonized script, crawler, tiny ETL pipeline etc.) aimed to access a single source, or even multiple sources (merging related data segments), retrieve (load, parse) raw data, process (filter, clean, normalize) and store it accordingly.
For now, storing SHOULD happen explicitly via INPUT queries, accessing back-end potential POST API surface is left
for future considerations.
Providers COULD be derived from a generic provider (implementation details), exposing a set of templates to descendants.
Also, providers are about to be monitored and scheduled by an orchestrator - primarily a set of config declarations.
Ideally, "almost" a logic-less proxying service acting in the middle of DB and clients.
If some comprehensive business logic appears, it COULD potentially be associated with this service OR derived into a dedicated module.
Main responsibilities:
- internally: storage accessor and data mapper, leveraged by models (representing multiple categories)
- externally: RESTful API wrapper (serving controllers and middlewares), together with a live update service (sockets) for clients
API itself acts "as a client", meaning serving JSON instead of explicit UI capabilities.
Existing and upcoming models/collections:
- news
- news types
- incidents
- incident types
- locations
- toponyms
- population
- reports
Aimed for data representation and UX/UI (map/clustering + filtering/pagination).
Any app/service/crawler authorized to access aggregated data streams.
REST and RSS services are considered as first options.