Bump up the max number of code points for JSON/YAML parser DAT-12657 #3552

wwillard7800 · 2022-12-08T19:41:55Z

Refactored the creation of the Yaml object in order to pass a LoaderOptions instance which sets the maximum number of code points higher so that larger JSON files can be parsed.

DAT-12657

github-actions · 2022-12-08T20:24:42Z

Unit Test Results

  4 812 files +12   4 812 suites +12 32m 2s ⏱️ - 2m 41s
  4 754 tests +19   4 520 ✔️ +22   234 💤 - 3 0 ❌ ±0
56 076 runs +24 50 736 ✔️ +29 5 340 💤 - 5 0 ❌ ±0

Results for commit 252018b. ± Comparison against base commit feb7f35.

♻️ This comment has been updated with latest results.

StevenMassaro · 2022-12-09T12:38:52Z

liquibase-core/src/main/java/liquibase/parser/core/yaml/YamlSnapshotParser.java


 import java.io.InputStream;
 import java.io.InputStreamReader;
 import java.util.Map;

 public class YamlSnapshotParser extends YamlParser implements SnapshotParser {

+    public static final int CODE_POINT_LIMIT = 9 * 1024 * 1024;


Why not just make this MAX_VALUE? I see no reason to limit the size on our end.

StevenMassaro · 2022-12-09T12:40:38Z

liquibase-core/src/main/java/liquibase/parser/core/yaml/YamlSnapshotParser.java

    @SuppressWarnings("java:S2095")
    @Override
    public DatabaseSnapshot parse(String path, ResourceAccessor resourceAccessor) throws LiquibaseParseException {
-        Yaml yaml = new Yaml(new SafeConstructor());
+        Yaml yaml = createYaml();


We create a Yaml object in the pro code too, I wonder if we should increase the code point limit there as well.

I'll take a look.

This may be best done with another story, up to you.

This MR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [org.liquibase.ext:liquibase-hibernate5](https://github.com/liquibase/liquibase-hibernate/wiki) ([source](https://github.com/liquibase/liquibase-hibernate)) | build | minor | `4.18.0` -> `4.19.0` | | [org.liquibase:liquibase-maven-plugin](http://www.liquibase.org/liquibase-maven-plugin) ([source](https://github.com/liquibase/liquibase)) | build | minor | `4.18.0` -> `4.19.0` | | [io.quarkus:quarkus-maven-plugin](https://github.com/quarkusio/quarkus) | build | minor | `2.15.3.Final` -> `2.16.0.Final` | --- ### Release Notes <details> <summary>liquibase/liquibase-hibernate</summary> ### [`v4.19.0`](https://github.com/liquibase/liquibase-hibernate/releases/tag/v4.19.0) [Compare Source](liquibase/liquibase-hibernate@v4.18.0...v4.19.0) Support for Liquibase 4.19.0. #### What's Changed - Support Hibernate ORM 6.1 with Jakarta EE 10 by [@papegaaij](https://github.com/papegaaij) in liquibase/liquibase-hibernate#434 - Bump hibernate.version from 6.1.5.Final to 6.1.6.Final by [@dependabot](https://github.com/dependabot) in liquibase/liquibase-hibernate#447 - Bump spring.version from 6.0.2 to 6.0.3 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase-hibernate#448 #### New Contributors - [@papegaaij](https://github.com/papegaaij) made their first contribution in liquibase/liquibase-hibernate#434 **Full Changelog**: liquibase/liquibase-hibernate@v4.18.0...v4.19.0 </details> <details> <summary>liquibase/liquibase</summary> ### [`v4.19.0`](https://github.com/liquibase/liquibase/releases/tag/v4.19.0) [Compare Source](liquibase/liquibase@v4.18.0...v4.19.0) ##### Liquibase v4.19.0 is a patch release with an XSD upgrade. #### Enhancements - Simplify assert calls and replaced with simpler and equivalent calls. by [@arturobernalg](https://github.com/arturobernalg) in liquibase/liquibase#3497 - Delete unused import statement. by [@arturobernalg](https://github.com/arturobernalg) in liquibase/liquibase#3522 - Added call to modifyChangeSet during execute method to allow the changeSet to be correctly set on the executor DAT-12388 by [@wwillard7800](https://github.com/wwillard7800) in liquibase/liquibase#3511 - Remove unnecessary local variables that add nothing to the comprehensibility of a method. by [@arturobernalg](https://github.com/arturobernalg) in liquibase/liquibase#3373 - Allow loading gzipped data files by [@mike-seger](https://github.com/mike-seger) in liquibase/liquibase#3379 - Use try-with-resources Statement when is possible. by [@arturobernalg](https://github.com/arturobernalg) in liquibase/liquibase#3374 - Define and reuse constants. Use an empty array styles to convert a collection to an array. by [@arturobernalg](https://github.com/arturobernalg) in liquibase/liquibase#3500 - Add support for block comment rollback commands on SQL changesets by [@krishnaenugandula](https://github.com/krishnaenugandula) in liquibase/liquibase#1399 - Allow primary key on addColumn for H2 by [@nick318](https://github.com/nick318) in liquibase/liquibase#3372 - Additional (optional) tableType attribute on the CreateTableChange by [@MartinRied](https://github.com/MartinRied) in liquibase/liquibase#3108 - Include "path" in databasechangelog's description column for all change types with "path" attributes by [@MichaelKern-IVV](https://github.com/MichaelKern-IVV) in liquibase/liquibase#3244 - [#1466](liquibase/liquibase#1466): Add ignore:true changeset attribute to Formatted SQL changeLogs by [@skrivenko](https://github.com/skrivenko) in liquibase/liquibase#3377 - [#1290](liquibase/liquibase#1290): Forbid empty changeSet id and author by [@skrivenko](https://github.com/skrivenko) in liquibase/liquibase#3397 - Allow to drop and create a view for a Postgres database if replacing the view would fail by [@rozenshteyn](https://github.com/rozenshteyn) in liquibase/liquibase#3399 - Rename DatabaseObjectComparator class to be DatabaseObjectCollectionComparator for clarity DAT-10112 by [@wwillard7800](https://github.com/wwillard7800) in liquibase/liquibase#3544 - Do not lower case the ProvidedValue description if the string is capitalized, i.e. it starts with 2 upper-case characters DAT-12614 by [@wwillard7800](https://github.com/wwillard7800) in liquibase/liquibase#3589 - Use 'Integer.compare' instead by [@arturobernalg](https://github.com/arturobernalg) in liquibase/liquibase#3528 - Simplify 'Map' operations. by [@arturobernalg](https://github.com/arturobernalg) in liquibase/liquibase#3527 - Remove unnecessary semicolon. by [@arturobernalg](https://github.com/arturobernalg) in liquibase/liquibase#3571 - Missing Override annotations. by [@arturobernalg](https://github.com/arturobernalg) in liquibase/liquibase#3558 - Prevents redundant loop iterations. Early loop exit in 'if' condition. by [@arturobernalg](https://github.com/arturobernalg) in liquibase/liquibase#3547 - DAT 6635 - Implement TagCommand by [@filipelautert](https://github.com/filipelautert) in liquibase/liquibase#3570 - DAT-12576 update release workflow to attach artifact by run_id by [@ap-liquibase](https://github.com/ap-liquibase) in liquibase/liquibase#3629 - DAT-12365 update install4j script to version 10.x by [@jnewton03](https://github.com/jnewton03) in liquibase/liquibase#3641 - Upgrades installer JDK version for next release. by [@filipelautert](https://github.com/filipelautert) in liquibase/liquibase#3440 - remove licenses that were moved to individual extensions (DAT-12784) by [@StevenMassaro](https://github.com/StevenMassaro) in liquibase/liquibase#3646 - DAT-12597 include commercial sources and javadoc in reversion by [@ap-liquibase](https://github.com/ap-liquibase) in liquibase/liquibase#3671 #### Security, Driver, and other updates - Bump mariadb-java-client from 3.0.8 to 3.1.0 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3471 - Bump testcontainers-bom from 1.17.5 to 1.17.6 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3477 - Bump snowflake-jdbc from 3.13.22 to 3.13.25 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3475 - Bump slf4j-jdk14 from 2.0.3 to 2.0.4 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3485 - Bump maven-install-plugin from 3.0.1 to 3.1.0 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3484 - Bump castlabs/get-package-version-id-action from 2.0 to 2.1 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3490 - Bump sqlite-jdbc from 3.39.4.0 to 3.40.0.0 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3510 - Bump jaybird from 4.0.6.java8 to 4.0.8.java8 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3509 - Bump mockito-inline from 4.8.1 to 4.10.0 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3580 - Bump targetMavenVersion from 3.8.5 to 3.8.6 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3593 - Bump junit-jupiter-params from 5.8.1 to 5.9.1 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3592 - Bump jaybird from 4.0.6.java8 to 4.0.8.java8 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3602 - Bump slf4j-jdk14 from 2.0.4 to 2.0.6 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3566 - Bump snowflake-jdbc from 3.13.25 to 3.13.26 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3579 - Bump robinraju/release-downloader from 1.6 to 1.7 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3603 - Bump ojdbc8 from 21.7.0.0 to 21.8.0.0 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3551 - Bump up the max number of code points for JSON/YAML parser DAT-12657 by [@wwillard7800](https://github.com/wwillard7800) in liquibase/liquibase#3552 - Bump actions/cache from 3.0.11 to 3.2.3 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3654 - update changelog version - 4.19 by [@suryaaki2](https://github.com/suryaaki2) in liquibase/liquibase#3676 - Update changelog xsd 4.19 by [@suryaaki2](https://github.com/suryaaki2) in liquibase/liquibase#3678 - Bump targetMavenVersion from 3.8.5 to 3.8.7 by [@dependabot](https://github.com/dependabot) in liquibase/liquibase#3634 - Update release-published.yml by [@jnewton03](https://github.com/jnewton03) in liquibase/liquibase#3540 - DAT-12783 Adding extension license information by [@wwillard7800](https://github.com/wwillard7800) in liquibase/liquibase#3614 #### Fixes - fix overwriteOutputFile parameter for GenerateChangelog (DAT-12036) by [@StevenMassaro](https://github.com/StevenMassaro) in liquibase/liquibase#3543 - Avoid ClassCastException when loading LogService from Scope by [@mattbertolini](https://github.com/mattbertolini) in liquibase/liquibase#3518 - Adds exclusions for mariadb newly added waffle dependency. by [@filipelautert](https://github.com/filipelautert) in liquibase/liquibase#3559 - Correctly handle indexes with descending columns in snapshot DAT-11447 by [@wwillard7800](https://github.com/wwillard7800) in liquibase/liquibase#3535 - Fix getting default schema issue for firebird by [@MalloD12](https://github.com/MalloD12) in liquibase/liquibase#3390 - Fix generatedSQL logic to allow setting up a function as a default value for MySQL version 8 by [@MalloD12](https://github.com/MalloD12) in liquibase/liquibase#3362 - DAT-11579: when generating changelogs for MySQL, ignore table column order for PKs by [@StevenMassaro](https://github.com/StevenMassaro) in liquibase/liquibase#3486 - Included SQL to return unique constraints for Sybase by [@crenan](https://github.com/crenan) in liquibase/liquibase#3517 - Add varbinary and binary support for DB2 - fixes [#3408](liquibase/liquibase#3408) by [@mihaelaDev](https://github.com/mihaelaDev) in liquibase/liquibase#3428 - Added support for COMPUTED values inside CSV files for loadData change by [@zbynekvavros](https://github.com/zbynekvavros) in liquibase/liquibase#944 - Prevent Executors collision due to hash used as a Map's key part by [@Dasiu](https://github.com/Dasiu) in liquibase/liquibase#3533 - Breaks out of LockService.init loop after validations are completed by [@filipelautert](https://github.com/filipelautert) in liquibase/liquibase#3576 - implement SingletonObject to solve [#2349](liquibase/liquibase#2349) by [@yairogen](https://github.com/yairogen) in liquibase/liquibase#3624 - Postgresql - Fallback to default schema name in SequenceSnapshotGenerator when this is null by [@djochim](https://github.com/djochim) in liquibase/liquibase#3637 ##### OWASP Dependency Check: Reported Vulnerabilities - snakeyaml.jar - This is a ["critical" vulnerability reported against the snakeyaml library](https://ossindex.sonatype.org/vulnerability/CVE-2022-1471). We are currently on the newest version of snakeyaml and there is no fix for the issue as of yet. #### New Contributors - [@mike-seger](https://github.com/mike-seger) made their first contribution in liquibase/liquibase#3379 - [@crenan](https://github.com/crenan) made their first contribution in liquibase/liquibase#3517 - [@mihaelaDev](https://github.com/mihaelaDev) made their first contribution in liquibase/liquibase#3428 - [@krishnaenugandula](https://github.com/krishnaenugandula) made their first contribution in liquibase/liquibase#1399 - [@skrivenko](https://github.com/skrivenko) made their first contribution in liquibase/liquibase#3397 - [@zbynekvavros](https://github.com/zbynekvavros) made their first contribution in liquibase/liquibase#944 - [@Dasiu](https://github.com/Dasiu) made their first contribution in liquibase/liquibase#3533 - [@yairogen](https://github.com/yairogen) made their first contribution in liquibase/liquibase#3624 - [@djochim](https://github.com/djochim) made their first contribution in liquibase/liquibase#3637 **Full Changelog**: liquibase/liquibase@v4.18.0...v4.19.0 ##### Get Certified Learn all the Liquibase fundamentals from free online courses by Liquibase experts and see how to apply them in the real world at https://learn.liquibase.com/. ##### Read the Documentation Please check out and contribute to the continually improving docs, now at https://docs.liquibase.com/. ##### Meet the Community Our community has built a lot. From extensions to integrations, you’ve helped make Liquibase the amazing open source project that it is today. Keep contributing to making it stronger: [Contribute code](https://www.liquibase.org/development/contribute.html) [Make doc updates](https://github.com/Datical/liquibase-docs) [Help by asking and answering questions](https://forum.liquibase.org/) [Set up a chat with the Product team](https://calendly.com/liquibase-outreach/product-feedback) Thanks to everyone who helps make the Liquibase community strong! #### File Descriptions - **Liquibase CLI** -- Includes open source + commercial functionality - **liquibase-x.y.z.tar.gz** -- Archive in tar.gz format - **liquibase-x.y.z.zip** -- Archive in zip format - **liquibase-windows-x64-installer-x.y.z.exe** -- Installer for Windows - **liquibase-macos-installer-x.y.z.dmg** -- Installer for MacOS - **Primary Libraries** - For embedding in other software - **liquibase-core-x.y.z.jar** – Base Liquibase library (open source) - **liquibase-commerical-x.y.z.jar** – Additional commercial functionality - **liquibase-additional-x.y.z.zip** – Contains additional, less commonly used files - Additional libraries such as liquibase-maven-plugin.jar and liquibase-cdi.jar - Javadocs for all the libraries - Source archives for all the open source libraries - ASC/MD5/SHA1 verification hashes for all files **NOTE: liquibase-core-<version>.jar** contains only the open-source license. If you use Liquibase Pro or other commercial add-ons, you must also **install liquibase-commercial-<version>.jar** </details> <details> <summary>quarkusio/quarkus</summary> ### [`v2.16.0.Final`](quarkusio/quarkus@2.15.3.Final...2.16.0.Final) [Compare Source](quarkusio/quarkus@2.15.3.Final...2.16.0.Final) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR is behind base branch, or you tick the rebase/retry checkbox. 👻 **Immortal**: This MR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

Bump up the max number of code points for JSON/YAML parser

cf90383

DAT-12657

wwillard7800 requested a review from filipelautert as a code owner December 8, 2022 19:41

Added test for large snapshot

451a131

DAT-12657

filipelautert approved these changes Dec 8, 2022

View reviewed changes

Merge branch 'master' into DAT-12657

5544f37

wwillard7800 requested a review from StevenMassaro December 9, 2022 03:54

StevenMassaro reviewed Dec 9, 2022

View reviewed changes

wwillard7800 added 3 commits December 9, 2022 09:57

Bump code point limit to MAX_VALUE DAT-12657

437c681

Merge remote-tracking branch 'origin/DAT-12657' into DAT-12657

3452c3a

Merge remote-tracking branch 'origin/master' into DAT-12657

252018b

suryaaki2 approved these changes Dec 13, 2022

View reviewed changes

suryaaki2 merged commit a3d7271 into master Dec 13, 2022

suryaaki2 deleted the DAT-12657 branch December 13, 2022 14:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump up the max number of code points for JSON/YAML parser DAT-12657 #3552

Bump up the max number of code points for JSON/YAML parser DAT-12657 #3552

wwillard7800 commented Dec 8, 2022

github-actions bot commented Dec 8, 2022 •

edited

StevenMassaro Dec 9, 2022

wwillard7800 Dec 9, 2022

StevenMassaro Dec 9, 2022

wwillard7800 Dec 9, 2022

StevenMassaro Dec 9, 2022

Bump up the max number of code points for JSON/YAML parser DAT-12657 #3552

Bump up the max number of code points for JSON/YAML parser DAT-12657 #3552

Conversation

wwillard7800 commented Dec 8, 2022

github-actions bot commented Dec 8, 2022 • edited

Unit Test Results

StevenMassaro Dec 9, 2022

Choose a reason for hiding this comment

wwillard7800 Dec 9, 2022

Choose a reason for hiding this comment

StevenMassaro Dec 9, 2022

Choose a reason for hiding this comment

wwillard7800 Dec 9, 2022

Choose a reason for hiding this comment

StevenMassaro Dec 9, 2022

Choose a reason for hiding this comment

github-actions bot commented Dec 8, 2022 •

edited