Move sensor data ingestion to job queue

[Copilot]

Web nodes block on large data POSTs because save_and_enqueue runs save_to_db synchronously in the request cycle. This moves the actual DB write to a dedicated ingestion RQ queue, keeping POST endpoints lightweight.

Changes

• New ingestion queue — added to app.queues alongside forecasting and scheduling
• flexmeasures/data/services/data_ingestion.py (new) — add_beliefs_to_database is the job function: saves the BeliefsDataFrame, commits, and enqueues any downstream forecasting jobs
• save_and_enqueue — checks for live workers on the ingestion queue; enqueues if present, otherwise falls back to synchronous processing with a warning logged:

workers = Worker.all(queue=ingestion_queue)
if workers:
    ingestion_queue.enqueue(add_beliefs_to_database, data, ...)
    return request_processed()
else:
    current_app.logger.warning(
        "No workers connected to the ingestion queue. Processing sensor data directly."
    )
    # ... original synchronous path

• CLI — run-worker --queue help text updated to include ingestion
• Docs — queues.rst updated with ingestion worker examples and a note on the fallback behaviour

Behaviour notes

• No ingestion worker = silent degradation to the old synchronous path. Existing deployments continue to work without any config change.
• Data is validated by the schema layer before enqueuing, so the 400/422 feedback path is unaffected.
• Run a dedicated ingestion worker in production: flexmeasures jobs run-worker --name ingester --queue ingestion

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #32236358 | 📁 Comparing ea634db against latest (8934f22)

  🔍 Preview build  

Show files changed (6 files in total): 📝 5 modified | ➕ 1 added | ➖ 0 deleted

File	Status
genindex.html	📝 modified
py-modindex.html	📝 modified
_autosummary/flexmeasures.data.services.data_ingestion.html	➕ added
_autosummary/flexmeasures.data.services.html	📝 modified
api/v3_0.html	📝 modified
host/queues.html	📝 modified