Implement duplicate identifiers check#229
Merged
javihern98 merged 6 commits intomainfrom Aug 27, 2025
Merged
Conversation
javihern98
approved these changes
Aug 27, 2025
Contributor
javihern98
left a comment
javihern98
left a comment
There was a problem hiding this comment.
Looks good, thanks! 😊 Note the big amount of changes is because of adding a new id in these big csv files
javihern98
added a commit
that referenced
this pull request
Oct 22, 2025
* Bump pytest-cov from 6.1.1 to 6.2.1 (#160) * Bump mypy from 1.16.0 to 1.16.1 (#161) * Bump urllib3 from 1.26.20 to 2.5.0 in the pip group (#163) * Bump ruff from 0.11.13 to 0.12.0 (#167) * Bump pytest from 8.4.0 to 8.4.1 (#168) * Allow sub and filter driven by scalars (#165) * Updated vtl grammar to allow sub and filter to work with scalars. * Added related tests. * Updated tests. * Updated tests. * Updated tests. * Updated ANTLR Runtime and other dependencies (#166) * Updated ANTLR Runtime version and many dependencies. Fixed typos in grammar. * Ruff format and removed pandas-stubs update. * Regenerated antlr lexer and parser. Updated dependencies. Small changes on tests. * Rollback to original error message on cast. * Fixed error messages on cast. * Scalar Values were imported to run function. (#169) * First approach to task * Implemented scalar argument * Amend error in condition * Small error * Some changes to manage scalars inside DAG. There is still work to do * Refactored internal functions and removed previous changes on scalar handling * Main changes to support scalar handling. Now it should work * Handling scalars and creation of output files with it. Added testing. * More testing * More testing was added * Full implementation of scalar values in run. * Old params were removed * documentation on scalar handling. * Check ambiguity with component that works also as scalar * Implementation of scalars fully done. There are still 3 tests that not work properly. * Refactored Helper on Semantic tests due to an error where scalars were treated as datasets. Solved an issue were scalars weren´t be loading when they are components. * Ruff and mypy fixes * Bump ruff from 0.12.0 to 0.12.1 (#171) Bumps [ruff](https://github.com/astral-sh/ruff) from 0.12.0 to 0.12.1. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.12.0...0.12.1) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.12.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump pytest-xdist from 3.7.0 to 3.8.0 (#191) * 175 update the error messages (#187) * Bump aiohttp from 3.12.13 to 3.12.14 in the pip group (#197) * Bump types-jsonschema from 4.24.0.20250528 to 4.24.0.20250708 (#190) * Fixed reference to pysdmx.vtl. Updated pysdmx version. (#198) * Updated version to 1.2.0. Updated pysdmx to latest rc. Fixed reference to generate_vtl_script. * Updated version to 1.2.0 in package. * Updated poetry lock * Fix if then else with explicit nulls (#205) * Fixed if the else null handling. * Fixed tests. * Fixed walkthrough. Improved error messages. Improved typing on run function. * Bump ruff from 0.12.3 to 0.12.7 (#213) * Bump types-jsonschema from 4.24.0.20250708 to 4.25.0.20250720 (#214) * Bump mypy from 1.16.1 to 1.17.1 (#215) * Bump mypy from 1.16.1 to 1.17.1 Bumps [mypy](https://github.com/python/mypy) from 1.16.1 to 1.17.1. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@v1.16.1...v1.17.1) --- updated-dependencies: - dependency-name: mypy dependency-version: 1.17.1 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Fixed mypy errors * Updated lock --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Javier <javier.hernandez@meaningfuldata.eu> * Fixed bug on Virtual names in Analytic with Calc (#217) * Fix nullability on conditional operators (#219) * Fixed nullability on conditional operators when we have scalars as then-else clauses. Fixed related tests. * Added main.py as exclusion for ruff. * Added specific test for case with calc identifier * Ruff changes * Fixed check on nullability in if to not use value, instead use data_type * Bumped version to 1.2.1rc1 * Added op to Join operators (#216) * Bumped version to 1.2.1 * Implement duplicate identifiers check (#229) * Updated identifiers check. Removed outdated checks. * fixed ruff errors. * fixed mypy errors. * Reactivated tests with duplicated identifiers. * Reactivated tests with duplicated identifiers. * Fixed ruff errors. * Bump ruff from 0.12.7 to 0.12.10 (#232) * Bump types-jsonschema from 4.25.0.20250720 to 4.25.1.20250822 (#230) Bumps [types-jsonschema](https://github.com/typeshed-internal/stub_uploader) from 4.25.0.20250720 to 4.25.1.20250822. - [Commits](https://github.com/typeshed-internal/stub_uploader/commits) --- updated-dependencies: - dependency-name: types-jsonschema dependency-version: 4.25.1.20250822 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump pytest from 8.4.1 to 8.4.2 (#236) Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.4.1 to 8.4.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@8.4.1...8.4.2) --- updated-dependencies: - dependency-name: pytest dependency-version: 8.4.2 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump ruff from 0.12.10 to 0.12.12 (#237) Bumps [ruff](https://github.com/astral-sh/ruff) from 0.12.10 to 0.12.12. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.12.10...0.12.12) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.12.12 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump mypy from 1.17.1 to 1.18.1 (#245) * Bump pytest-cov from 6.2.1 to 7.0.0 (#246) * Bump ruff from 0.12.12 to 0.13.0 (#247) * Case does not handle correctly the dependant expressions using condition data (#243) * Fixed Case conditions. * Fixed ruff errors. * Fixed mypy errors. * Fixed test resolution. * Fixed ruff errors. * Added related test. * Fixed ruff errors. * Fixed ruff errors. * Fixed mypy errors. * Added related test. * Fixed ruff errors. * Fixed sdmx-like time period month values are now set as {yyyy}-M{mm} format. Dates prior 1900 are now accepted. (#251) * Updated pysdmx version to 1.5.2. (#258) * Bump mypy from 1.18.1 to 1.18.2 (#253) * Bump ruff from 0.13.0 to 0.13.1 (#252) * Bump ruff from 0.13.1 to 0.13.2 (#260) * Bump ruff from 0.13.2 to 0.13.3 (#264) Bumps [ruff](https://github.com/astral-sh/ruff) from 0.13.2 to 0.13.3. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.13.2...0.13.3) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.13.3 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Solved bug with UDO operation. (#265) * First approach on solving bug. The bug with this solution is solved, but it makes conflict with self.isfromregularaggr * Progress in finding a solution to the bug. I've added a condition to search for the component within the dataset. Now there are only 13 tests left to fix. * Added UDO boolean. * Added test of bug UDO * Added recursive UDO handling. * Resolved UDO test_12. fixed Ruff and mypy errors. --------- Co-authored-by: Mateo <mateo.delorenzo@meaningfuldata.eu> * Bump ruff from 0.13.3 to 0.14.0 (#266) Bumps [ruff](https://github.com/astral-sh/ruff) from 0.13.3 to 0.14.0. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.13.3...0.14.0) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.14.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Solved bug with errorlevel (#259) * Solved bug with errorlevel. It could not manage string types. Now it can * Updated Dpruleset and Hruleset AST erLevel methods to support string values * ErrorLevel typing was changed to support integer and string values. * Updated typo on check, check_datapoints and check_hierarchy methods. * Implemmented conditions on error_level taking into account the data type it has. Testing with string only values has been added. --------- Co-authored-by: Mateo de Lorenzo Argelés <160473799+mla2001@users.noreply.github.com> * Bump types-jsonschema from 4.25.1.20250822 to 4.25.1.20251009 (#267) * Time operator only usable with EQ and NEQ operators (#262) * Updated Time comparison methods. * Updated Time comparison methods. * Fixed ruff and mypy errors. * Updated Time comparison and Time Max/Min error messages. * Updated Grammar test to adapt to new errors. * Fixed ruff errors. * Added related tests. * Fixed ruff errors. * Added lint check and updated output format (#269) * Updated workflows to add lint check. Prettify responses on ruff check and mypy * Small change to trigger ruff check * Rolled back change * Bumped version to 1.2.2. Removed patch versions from stubs (#270) * Bumped version to 1.2.2. Removed patch versions from stubs * Bumped version to 1.2.2 on init * Downgraded ANTLR to 4.9.2 and updated duckdb to 1.4 (#277) * Updated grammar and ANTLR version. * Updated main.py to use persistent assignment. * Bump ruff from 0.14.0 to 0.14.1 (#275) Bumps [ruff](https://github.com/astral-sh/ruff) from 0.14.0 to 0.14.1. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.14.0...0.14.1) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.14.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump version to 1.3.0 (#278) * Bump version to 1.3.0 * Bump version to 1.3.0 * Changed dev dependencies to use ranges instead of caret * Updated dependencies to latest versions * Mypy errors fixes. Prevent unhandled exceptions when a NA reaches an If * Ruff format changes * Mypy change to prevent issues with python 3.9 * Mypy change to prevent issues with python 3.9 * Update time period comparison (#274) * Updated TimePeriod comparison. * Added related tests. * Fixed ruff errors. * deactivated new failing tests. * Implemented TimePeriod Max/Min op with different period indicator exception. * implemented related tests. * Fixed ruff errors. * Fixed TimePeriod additional tests. * Fixed TimePeriod Aggregate operators tests. * Fixed ruff errors. * Updated 2-1-19-20 error message. * Fixing imports after merge * Fixed API tests with scalars * Updated poetry.lock * Updated clean_connection to enhance testing performance and garbage collection * Updated testing workflow to check ruff format * Ruff changes fix * Bumped version to 2.0.0rc1 * Minor merge fix. * Fixed ruff errors. * Minor fix. * Fixed ruff errors. * Fixed some tests references. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mateo de Lorenzo Argelés <160473799+mla2001@users.noreply.github.com> Co-authored-by: Alberto <155883871+albertohernandez1995@users.noreply.github.com> Co-authored-by: Mateo <mateo.delorenzo@meaningfuldata.eu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The duplicate identifiers check was implemented, raising an exception whenever a group of duplicate identifiers is found in the same dataset.
Fixed tests with duplicate identifiers.