Skip to content

Conversation

@iambriccardo
Copy link
Contributor

@iambriccardo iambriccardo commented Dec 31, 2025

This PR introduces validation endpoints for destinations, pipelines, and their combined configuration, with the goal of making it easier for users to verify that all required preconditions are satisfied before a pipeline is created or executed.

The validation mechanism is implemented through three new endpoints, each responsible for validating a pipeline, a destination, or both together. These endpoints are intended to be called prior to resource creation, allowing clients to determine upfront whether a configuration is viable and whether all prerequisites are met. The system is structured to support incremental validation steps, making it straightforward to extend with additional checks over time.

The design follows the principle that the replicator itself should not perform validation, and should instead fail fast when encountering invalid configurations. This keeps the replicator code simpler and avoids negatively impacting startup performance, which would otherwise be affected by executing potentially expensive validation logic during initialization

@iambriccardo iambriccardo changed the title riccardobusetti/etl 468 implement validation endpoint feat(api): Implement validation mechanism Jan 1, 2026
const MAX_POOL_CONNECTIONS: u32 = 1;

/// Connects to the source database with the specified configuration and default connection pool size.
pub async fn connect_to_source_database_with_defaults(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved here.

@coveralls
Copy link

coveralls commented Jan 5, 2026

Pull Request Test Coverage Report for Build 20738127653

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 305 of 455 (67.03%) changed or added relevant lines in 17 files are covered.
  • 434 unchanged lines in 24 files lost coverage.
  • Overall coverage decreased (-0.2%) to 81.393%

Changes Missing Coverage Covered Lines Changed/Added Lines %
etl-api/src/routes/destinations_pipelines.rs 10 11 90.91%
etl-api/src/validation/validators.rs 68 69 98.55%
etl-destinations/src/bigquery/client.rs 10 11 90.91%
etl-replicator/src/core.rs 0 2 0.0%
etl-config/src/environment.rs 0 5 0.0%
etl-examples/src/main.rs 0 7 0.0%
etl-api/src/main.rs 0 8 0.0%
etl-replicator/src/main.rs 0 8 0.0%
etl-api/src/validation/mod.rs 55 65 84.62%
etl-destinations/src/bigquery/core.rs 0 20 0.0%
Files with Coverage Reduction New Missed Lines %
etl-replicator/src/core.rs 2 0.0%
etl-api/src/config.rs 3 26.09%
etl/src/workers/pool.rs 3 92.96%
etl-postgres/src/replication/worker.rs 5 61.54%
etl-config/src/load.rs 6 82.97%
etl-postgres/src/replication/state.rs 7 96.62%
etl/src/test_utils/notify.rs 8 90.78%
etl/src/test_utils/test_destination_wrapper.rs 8 84.71%
etl-api/src/startup.rs 9 58.58%
etl-destinations/src/iceberg/core.rs 9 95.56%
Totals Coverage Status
Change from base Build 20601971115: -0.2%
Covered Lines: 17126
Relevant Lines: 21041

💛 - Coveralls

@iambriccardo iambriccardo changed the title feat(api): Implement validation mechanism feat(api): Implement validation endpoints Jan 5, 2026
@iambriccardo iambriccardo marked this pull request as ready for review January 5, 2026 04:55
@iambriccardo iambriccardo requested a review from a team as a code owner January 5, 2026 04:55
@iambriccardo iambriccardo merged commit 78585c0 into main Jan 6, 2026
11 checks passed
@iambriccardo iambriccardo deleted the riccardobusetti/etl-468-implement-validation-endpoint branch January 6, 2026 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants