Migrate from minikube to k3d as backend implementation

[renatolfc]

The main change proposed in this PR is migrating our kubernetes cluster implementation from minikube to k3d. After running large workflows for a while now, we noticed that minikube with the Docker driver was too unstable, causing cascading crashes of both kubernetes services and our services.

We've been experimenting with k3d internally for more than a month now and it seems to be much more stable as a docker-based kubernetes platform, and we believe FarmVibes.AI users will benefit from that change.

When running the farmvibes-ai.sh script, it will automatically backup workflow state and suggest migrating a cluster from minikube to k3d.

Apart from that, we've made the following changes, listed below:

Client

• [📈 IMPROVEMENT] We've improved the monitor client library to display correct durations for workflows that run for more than 24h
• [📈 IMPROVEMENT] The client now differentiates between tasks that have been queued, and tasks that have actually started running
• [📈 IMPROVEMENT] We now add the status of the workflow run to the header of the monitor method

Notebooks

• [🎉 NEW] Added a notebook for computing a timelapse of NDVI values using SpaceEye
• [🎉 NEW] Added a notebook for calculating various spectral indices (all indices listed in the Awesome Spectral Indices are supported)
• [🎉 NEW] Added a notebook that illustrates how to use the chunk_onnx workflow to run time-series analysis over raster images (in parallel)
• [📈 IMPROVEMENT] Updated the DeepMC notebooks to download recent and archived numerical weather prediction (NWP) model outputs from different cloud archive sources

Workflows

• [🎉 NEW] Added a workflow to download forecast data using the herbie python package
• [🎉 NEW] Added workflows for downloading LIDAR/GEDI data
• [🎉 NEW] Added a workflows for downloading ERA5 monthly data
• [🎉 NEW] Added a workflow for applying ONNX models to chunks of raster images that span multiple time steps (to enable, for example, time-series analyses over raster images)
• [📈 IMPROVEMENT] We now list Sentinel-1 products from the Planetary Computer, as opposed to SciHub
• [📈 IMPROVEMENT] Updated the index workflow to support more spectral indices

Backend

• [⚒️ FIX] We fixed the workflow spec validator to check for destination ports also being a source
• [⚒️ FIX] The kubernetes backend has become much more stable due to the port to k3d
• [⚒️ FIX] We changed how we distribute work to farmvibes.ai workers, reducing the overhead of computation
• [⚒️ FIX] We now use dapr distributed locks to prevent duplication of work, making workflows run faster
• [📈 IMPROVEMENT] The history of workflow execution is now maintained between clusters
  • As a consequence, migrating from minikube to k3d will keep all the workflow data users already had
• [📈 IMPROVEMENT] The workflow validator now checks whether parameters have multiple default values
• [📈 IMPROVEMENT] We now resolve workflow parameters to infer default values and descriptions from tasks (ops)
• [📈 IMPROVEMENT] We now persist all pod logs to ~/.cache/farmvibes-ai/logs
• [📈 IMPROVEMENT] Logs are persisted in a JSON-like format
• [📈 IMPROVEMENT] We now propagate the failed and cancelled states to all ops that did not run when we cancel/fail a workflow
• [📈 IMPROVEMENT] We now support fan-out of workflow sources, enabling parallel execution of source nodes
• [📈 IMPROVEMENT] We added an ACKnowledgement message type for differentiating between started and queued tasks
• [📈 IMPROVEMENT] Allow STAC query parameters in Planetary Computer collections
• [📈 IMPROVEMENT] We've changed how tasks are assigned to workers, reducing the amount of messages interchanged between the worker and orchestrator components
• [📈 IMPROVEMENT] Added the option to skip chips that have no data when running models over large rasters

---

Co-authored-by: Bruno Silva brunosilva@microsoft.com
Co-authored-by: Eduardo Rodrigues edrodrigues@microsoft.com
Co-authored-by: Naga Bilwanth Gangarapu Naga@zensa.co
Co-authored-by: Rafael Padilha rpadilha@microsoft.com
Co-authored-by: Renato Luiz de Freitas Cunha renato.cunha@microsoft.com
Co-authored-by: Roberto de Moura Estevão Filho robertode@microsoft.com
Co-authored-by: Sara Malvar saramalvar@microsoft.com