Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DVX-488: Implement infinite paging approach #351

Merged
merged 7 commits into from
Jun 18, 2024
Merged

DVX-488: Implement infinite paging approach #351

merged 7 commits into from
Jun 18, 2024

Conversation

Aryamanz29
Copy link
Member

@Aryamanz29 Aryamanz29 commented Jun 14, 2024

The feature implements a new pagination approach for search, known as bulk search, based on asset creation timestamps. This approach is used when the number of results exceeds the predefined threshold (i.e: 100,000 assets). Users can explicitly run this search by setting the optional keyword argument bulk=True in the FluentSearch.execute() or asset.search() methods.

results = (
    FluentSearch()
    .where(CompoundQuery.active_assets())
    .where(CompoundQuery.asset_type(AtlasGlossaryTerm))
    .page_size(2)
).execute(client, bulk=True)

request = (
    FluentSearch()
    .where(CompoundQuery.active_assets())
    .where(CompoundQuery.asset_type(AtlasGlossaryTerm))
    .page_size(2)
).to_request()

results = client.asset.search(criteria=request, bulk=True)

@Aryamanz29 Aryamanz29 self-assigned this Jun 14, 2024
@Aryamanz29 Aryamanz29 force-pushed the DVX-488 branch 2 times, most recently from 2dd4902 to f39715d Compare June 14, 2024 15:24
Copy link
Collaborator

@ErnestoLoma ErnestoLoma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one question

pyatlan/client/asset.py Outdated Show resolved Hide resolved
@Aryamanz29 Aryamanz29 merged commit 16593a2 into main Jun 18, 2024
32 of 33 checks passed
@Aryamanz29 Aryamanz29 deleted the DVX-488 branch June 18, 2024 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants