Skip to content

Add pagination for snowflake usage and lineage queries sql#23781

Merged
mohittilala merged 3 commits intomainfrom
snowflake-queries-pagination
Oct 8, 2025
Merged

Add pagination for snowflake usage and lineage queries sql#23781
mohittilala merged 3 commits intomainfrom
snowflake-queries-pagination

Conversation

@mohittilala
Copy link
Copy Markdown
Contributor

@mohittilala mohittilala commented Oct 8, 2025 •
edited
Loading

Describe your changes:

This PR includes changes of pagination with fixed batch size to fetch queries for snowflake lineage and usage since for larger number of queries if sql executing for more than 6 hours, it throws 403 forbidden error.

[2025-09-30 07:52:09] ERROR    {snowflake.connector.result_batch:result_batch:403} - Failed to fetch the large result set batch data_0_6_0_1 for the 1 th time, backing off for 1s for the reason: '290403: HTTP 403: Forbidden'"
Traceback (most recent call last):"
  File \"/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py\", line 390, in _download"
    raise RetryRequest(error)"
snowflake.connector.network.RetryRequest: 290403: HTTP 403: Forbidden"
[2025-09-30 07:52:10] ERROR    {snowflake.connector.result_batch:result_batch:403} - Failed to fetch the large result set batch data_0_6_0_1 for the 2 th time, backing off for 1s for the reason: '290403: HTTP 403: Forbidden'"
Traceback (most recent call last):"
  File \"/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py\", line 390, in _download"
    raise RetryRequest(error)"
snowflake.connector.network.RetryRequest: 290403: HTTP 403: Forbidden"
[2025-09-30 07:52:11] ERROR    {snowflake.connector.result_batch:result_batch:403} - Failed to fetch the large result set batch data_0_6_0_1 for the 3 th time, backing off for 2s for the reason: '290403: HTTP 403: Forbidden'"
Traceback (most recent call last):"
  File \"/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py\", line 390, in _download"
    raise RetryRequest(error)"
snowflake.connector.network.RetryRequest: 290403: HTTP 403: Forbidden"
[2025-09-30 07:52:13] ERROR    {snowflake.connector.result_batch:result_batch:403} - Failed to fetch the large result set batch data_0_6_0_1 for the 4 th time, backing off for 0s for the reason: '290403: HTTP 403: Forbidden'"
Traceback (most recent call last):"
  File \"/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py\", line 390, in _download"
    raise RetryRequest(error)"
snowflake.connector.network.RetryRequest: 290403: HTTP 403: Forbidden"
[2025-09-30 07:52:13] ERROR    {snowflake.connector.result_batch:result_batch:403} - Failed to fetch the large result set batch data_0_6_0_1 for the 5 th time, backing off for 0s for the reason: '290403: HTTP 403: Forbidden'"
Traceback (most recent call last):"
  File \"/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py\", line 390, in _download"
    raise RetryRequest(error)"
snowflake.connector.network.RetryRequest: 290403: HTTP 403: Forbidden"
[2025-09-30 07:52:13] ERROR    {snowflake.connector.result_batch:result_batch:403} - Failed to fetch the large result set batch data_0_6_0_1 for the 6 th time, backing off for 3s for the reason: '290403: HTTP 403: Forbidden'"
Traceback (most recent call last):"
  File \"/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py\", line 390, in _download"
    raise RetryRequest(error)"
snowflake.connector.network.RetryRequest: 290403: HTTP 403: Forbidden"
[2025-09-30 07:52:16] ERROR    {snowflake.connector.result_batch:result_batch:403} - Failed to fetch the large result set batch data_0_6_0_1 for the 7 th time, backing off for 4s for the reason: '290403: HTTP 403: Forbidden'"
Traceback (most recent call last):"
  File \"/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py\", line 390, in _download"
    raise RetryRequest(error)"
snowflake.connector.network.RetryRequest: 290403: HTTP 403: Forbidden"
[2025-09-30 07:52:20] ERROR    {snowflake.connector.result_batch:result_batch:403} - Failed to fetch the large result set batch data_0_6_0_1 for the 8 th time, backing off for 16s for the reason: '290403: HTTP 403: Forbidden'"
Traceback (most recent call last):"
  File \"/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py\", line 390, in _download"
    raise RetryRequest(error)"
snowflake.connector.network.RetryRequest: 290403: HTTP 403: Forbidden"
[2025-09-30 07:52:37] ERROR    {snowflake.connector.result_batch:result_batch:403} - Failed to fetch the large result set batch data_0_6_0_1 for the 9 th time, backing off for 11s for the reason: '290403: HTTP 403: Forbidden'"
Traceback (most recent call last):"
  File \"/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py\", line 390, in _download"
    raise RetryRequest(error)"
snowflake.connector.network.RetryRequest: 290403: HTTP 403: Forbidden"
[2025-09-26 08:40:56] WARNING  {metadata.Ingestion:step:242} - Encountered exception running step [<metadata.ingestion.source.database.snowflake.lineage.SnowflakeLineageSource object at 0x7f5adb682aa0>]: [(snowflake.connector.errors.ForbiddenError) 290403: HTTP 403: Forbidden
(Background on this error at: https://sqlalche.me/e/14/dbapi)]
[2025-09-26 08:40:56] WARNING  {metadata.Ingestion:status:95} - Encountered exception running step [<metadata.ingestion.source.database.snowflake.lineage.SnowflakeLineageSource object at 0x7f5adb682aa0>]: [(snowflake.connector.errors.ForbiddenError) 290403: HTTP 403: Forbidden
(Background on this error at: https://sqlalche.me/e/14/dbapi)]
[2025-09-26 08:40:56] DEBUG    {metadata.Ingestion:status:96} - Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py", line 390, in _download
    raise RetryRequest(error)
snowflake.connector.network.RetryRequest: 290403: HTTP 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 969, in fetchone
    row = dbapi_cursor.fetchone()
  File "/usr/local/lib/python3.10/site-packages/snowflake/connector/cursor.py", line 1526, in fetchone
    _next = next(self._result, None)
  File "/usr/local/lib/python3.10/site-packages/snowflake/connector/result_set.py", line 139, in result_set_iterator
    batch_iterator = get_fetch_result(future.result())
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py", line 828, in create_iter
    return self._create_iter(iter_unit=iter_unit, connection=connection)
  File "/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py", line 756, in _create_iter
    response = self._download(connection=connection)
  File "/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py", line 401, in _download
    raise e
snowflake.connector.errors.ForbiddenError: 290403: HTTP 403: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/metadata/ingestion/api/step.py", line 216, in run
    for result in self._iter():
  File "/usr/local/lib/python3.10/site-packages/metadata/ingestion/source/database/lineage_source.py", line 553, in _iter
    yield from self.yield_query_lineage() or []
  File "/usr/local/lib/python3.10/site-packages/metadata/ingestion/source/database/lineage_source.py", line 400, in yield_query_lineage
    yield from self.generate_lineage_with_processes(
  File "/usr/local/lib/python3.10/site-packages/metadata/ingestion/source/database/lineage_source.py", line 278, in generate_lineage_with_processes
    if start_next_process():
  File "/usr/local/lib/python3.10/site-packages/metadata/ingestion/source/database/lineage_source.py", line 171, in start_next_process
    chunk = next(chunk_iter)
  File "/usr/local/lib/python3.10/site-packages/metadata/ingestion/source/database/lineage_source.py", line 133, in chunk_generator
    for item in producer_fn():
  File "/usr/local/lib/python3.10/site-packages/metadata/ingestion/source/database/snowflake/query_parser.py", line 97, in get_table_query
    yield from super().get_table_query()
  File "/usr/local/lib/python3.10/site-packages/metadata/ingestion/source/database/lineage_source.py", line 371, in get_table_query
    yield from self.yield_table_query()
  File "/usr/local/lib/python3.10/site-packages/metadata/ingestion/source/database/snowflake/lineage.py", line 84, in yield_table_query
    for row in rows:
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 382, in iterrows
    for row in self._fetchiter_impl():
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 1810, in _fetchiter_impl
    row = fetchone(self, self.cursor)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 974, in fetchone
    self.handle_exception(result, dbapi_cursor, e)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 955, in handle_exception
    result.connection._handle_dbapi_exception(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 969, in fetchone
    row = dbapi_cursor.fetchone()
  File "/usr/local/lib/python3.10/site-packages/snowflake/connector/cursor.py", line 1526, in fetchone
    _next = next(self._result, None)
  File "/usr/local/lib/python3.10/site-packages/snowflake/connector/result_set.py", line 139, in result_set_iterator
    batch_iterator = get_fetch_result(future.result())
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py", line 828, in create_iter
    return self._create_iter(iter_unit=iter_unit, connection=connection)
  File "/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py", line 756, in _create_iter
    response = self._download(connection=connection)
  File "/usr/local/lib/python3.10/site-packages/snowflake/connector/result_batch.py", line 401, in _download
    raise e
sqlalchemy.exc.DBAPIError: (snowflake.connector.errors.ForbiddenError) 290403: HTTP 403: Forbidden
(Background on this error at: https://sqlalche.me/e/14/dbapi)

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@mohittilala mohittilala added this to the Sept 29 - Oct 10 milestone Oct 8, 2025
@mohittilala mohittilala self-assigned this Oct 8, 2025
@mohittilala mohittilala requested a review from a team as a code owner October 8, 2025 03:15
@mohittilala mohittilala added Ingestion safe to test Add this label to run secure Github workflows on PRs lineage To release Will cherry-pick this PR into the release branch labels Oct 8, 2025
@mohittilala mohittilala moved this to In Review / QA 👀 in Roadmap Oct 8, 2025
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Oct 8, 2025

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Oct 8, 2025

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Oct 8, 2025

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Oct 8, 2025

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion-base-slim:trivy (debian 12.12)

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (31)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.12.7 2.15.0
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.13.4 2.15.0
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
commons-beanutils:commons-beanutils CVE-2025-48734 🚨 HIGH 1.9.4 1.11.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.netty:netty-codec-http2 CVE-2025-55163 🚨 HIGH 4.1.96.Final 4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p 🚨 HIGH 4.1.96.Final 4.1.100.Final
io.netty:netty-handler CVE-2025-24970 🚨 HIGH 4.1.96.Final 4.1.118.Final
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.3 3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server CVE-2024-13009 🚨 HIGH 9.4.56.v20240826 9.4.57.v20241219

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (3)

Package Vulnerability ID Severity Installed Version Fixed Version
Werkzeug CVE-2024-34069 🚨 HIGH 2.2.3 3.0.3
deepdiff CVE-2025-58367 🔥 CRITICAL 7.0.1 8.6.1
setuptools CVE-2025-47273 🚨 HIGH 70.3.0 78.1.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/extended_sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/lineage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.yaml

No Vulnerabilities Found

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Oct 8, 2025

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion:trivy (debian 12.9)

Vulnerabilities (19)

Package Vulnerability ID Severity Installed Version Fixed Version
libexpat1 CVE-2023-52425 🚨 HIGH 2.5.0-1+deb12u1 2.5.0-1+deb12u2
libexpat1 CVE-2024-8176 🚨 HIGH 2.5.0-1+deb12u1 2.5.0-1+deb12u2
libgnutls30 CVE-2025-32988 🚨 HIGH 3.7.9-2+deb12u3 3.7.9-2+deb12u5
libgnutls30 CVE-2025-32990 🚨 HIGH 3.7.9-2+deb12u3 3.7.9-2+deb12u5
libicu72 CVE-2025-5222 🚨 HIGH 72.1-3 72.1-3+deb12u1
libperl5.36 CVE-2023-31484 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u3
libperl5.36 CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
libsqlite3-0 CVE-2025-6965 🔥 CRITICAL 3.40.1-2+deb12u1 3.40.1-2+deb12u2
libxslt1.1 CVE-2024-55549 🚨 HIGH 1.1.35-1 1.1.35-1+deb12u1
libxslt1.1 CVE-2025-24855 🚨 HIGH 1.1.35-1 1.1.35-1+deb12u1
libxslt1.1 CVE-2025-7424 🚨 HIGH 1.1.35-1 1.1.35-1+deb12u2
perl CVE-2023-31484 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u3
perl CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
perl-base CVE-2023-31484 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u3
perl-base CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
perl-modules-5.36 CVE-2023-31484 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u3
perl-modules-5.36 CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
sqlite3 CVE-2025-6965 🔥 CRITICAL 3.40.1-2+deb12u1 3.40.1-2+deb12u2
sudo CVE-2025-32462 🚨 HIGH 1.9.13p3-1+deb12u1 1.9.13p3-1+deb12u2

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (31)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.12.7 2.15.0
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.13.4 2.15.0
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
commons-beanutils:commons-beanutils CVE-2025-48734 🚨 HIGH 1.9.4 1.11.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.netty:netty-codec-http2 CVE-2025-55163 🚨 HIGH 4.1.96.Final 4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p 🚨 HIGH 4.1.96.Final 4.1.100.Final
io.netty:netty-handler CVE-2025-24970 🚨 HIGH 4.1.96.Final 4.1.118.Final
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.3 3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server CVE-2024-13009 🚨 HIGH 9.4.56.v20240826 9.4.57.v20241219

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (9)

Package Vulnerability ID Severity Installed Version Fixed Version
Authlib CVE-2025-59420 🚨 HIGH 1.3.1 1.6.4
Werkzeug CVE-2024-34069 🚨 HIGH 2.2.3 3.0.3
apache-airflow-providers-common-sql CVE-2025-30473 🚨 HIGH 1.21.0 1.24.1
deepdiff CVE-2025-58367 🔥 CRITICAL 7.0.1 8.6.1
redshift-connector CVE-2025-5279 🚨 HIGH 2.1.5 2.1.7
setuptools CVE-2024-6345 🚨 HIGH 65.5.1 70.0.0
setuptools CVE-2025-47273 🚨 HIGH 65.5.1 78.1.1
setuptools CVE-2025-47273 🚨 HIGH 70.3.0 78.1.1
tornado CVE-2025-47287 🚨 HIGH 6.4.2 6.5

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /home/airflow/openmetadata-airflow-apis/openmetadata_managed_apis.egg-info/PKG-INFO

No Vulnerabilities Found

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud Bot commented Oct 8, 2025

Quality Gate Failed Quality Gate failed for 'open-metadata-ingestion'

Failed conditions
12.8% Coverage on New Code (required ≥ 20%)
13.8% Duplication on New Code (required ≤ 10%)

See analysis details on SonarQube Cloud

@mohittilala mohittilala merged commit da8c50d into main Oct 8, 2025
21 of 25 checks passed
@mohittilala mohittilala deleted the snowflake-queries-pagination branch October 8, 2025 15:15
@github-project-automation github-project-automation Bot moved this from In Review / QA 👀 to Done ✅ in Roadmap Oct 8, 2025
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Oct 8, 2025

Changes have been cherry-picked to the 1.9.13 branch.

github-actions Bot pushed a commit that referenced this pull request Oct 8, 2025
@mohittilala
Add pagination for snowflake usage and lineage queries sql (#23781)
d3f3982 
* Add pagination for snowflake usage and lineage queries sql

* py_format

(cherry picked from commit da8c50d)
ayush-shah pushed a commit that referenced this pull request Oct 16, 2025
@mohittilala @ayush-shah
Add pagination for snowflake usage and lineage queries sql (#23781)
4b456f0 
* Add pagination for snowflake usage and lineage queries sql

* py_format
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@harshsoni2024 harshsoni2024 harshsoni2024 approved these changes

Assignees

@mohittilala mohittilala

Labels

Ingestion lineage safe to test Add this label to run secure Github workflows on PRs To release Will cherry-pick this PR into the release branch

Projects

Roadmap
Status: Done ✅

Milestone

Sept 29 - Oct 10

Development

Successfully merging this pull request may close these issues.

2 participants

@mohittilala @harshsoni2024