Fixed AI-assisted sql_query generation and made has_valid_schema compatible with older Spark versions

[ghanse]

Changes

• Improved Agent guidelines
• Make has_valid_schema check compatible with older spark versions (< 4)
• Fix issue with subqueries in sql expression check in Serverless v5 when check name is not provided (auto-derived)
• Added guidelines on how to configure DQX with LDP/DLT (Lakeflow Declarative Pipelines) to enable incrementalization for Materialized Views (MVs)
• Improved validation of required check function arguments
• Fix CI issues

Linked issues

Resolves: #1053

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

[github-actions]

✅ 681/681 passed, 39 skipped, 5h33m17s total

_{Running from acceptance #4371}

[codecov]

Codecov Report

❌ Patch coverage is 96.55172% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.96%. Comparing base (040adc7) to head (d7a7f75).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
src/databricks/labs/dqx/checks_validator.py	92.30%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #995      +/-   ##
==========================================
- Coverage   92.06%   91.96%   -0.10%     
==========================================
  Files         101      101              
  Lines        9387     9408      +21     
==========================================
+ Hits         8642     8652      +10     
- Misses        745      756      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.