[Bug] Search of Pages and / or Database in SearchComponent Currently Doesn't Support Pagination

[Nabhag8848]

What happened?

• Current Implemenatation doesn't support pagination of Searching pages and/ or databases.
• Api endpoint of Search Page, Returns maximum of 100 pages
• https://developers.notion.com/reference/post-search
• methods searchPages, searchDatabase, SearchPagesandDatabases under NotionSDK, (maybe more) uses this API.

Steps to reproduce

• Improve methods such a way that it gets all the avaialable pages and / or databases.
• Check if there is performance issue while doing this.

Relevant ScreenShots

No response

Version

v6.5

What browsers are you seeing the problem on?

Chrome

[brf153]

@Nabhag8848 working on this issue.

[brf153]

How I Will Implement the Changes:

1. After reviewing Notion's API guide, I realized that the response allows a maximum of 100 items. If the number of items exceeds 100, the response will contain 'has_more: true' along with a cursor value. To retrieve additional items, we need to pass this cursor value back to Notion's API, either through a query in a GET request or using the body in a POST request.
   

2. Consequently, I will implement a mechanism to check 'has_more' in every response. If 'has_more' is true, we will extract the cursor value from the response and send another request, continuing this process until 'has_more' becomes false. Subsequently, we will merge all the responses into one array, which will be passed as the final response for subsequent implementations.

3. To verify the implementation and address potential performance issues, I will initially set the 'page_size' to 10 and create more than 10 pages in my workspace. Following this, I will send the query and assess its functionality to ensure its effectiveness.

[brf153]

How I Will Implement the Changes:

1. After reviewing Notion's API guide, I realized that the response allows a maximum of 100 items. If the number of items exceeds 100, the response will contain 'has_more: true' along with a cursor value. To retrieve additional items, we need to pass this cursor value back to Notion's API, either through a query in a GET request or using the body in a POST request.
   
2. Consequently, I will implement a mechanism to check 'has_more' in every response. If 'has_more' is true, we will extract the cursor value from the response and send another request, continuing this process until 'has_more' becomes false. Subsequently, we will merge all the responses into one array, which will be passed as the final response for subsequent implementations.
3. To verify the implementation and address potential performance issues, I will initially set the 'page_size' to 10 and create more than 10 pages in my workspace. Following this, I will send the query and assess its functionality to ensure its effectiveness.

@Nabhag8848 Could you please review this and advise if I should proceed with this approach?

[Nabhag8848]

How I Will Implement the Changes:

1. After reviewing Notion's API guide, I realized that the response allows a maximum of 100 items. If the number of items exceeds 100, the response will contain 'has_more: true' along with a cursor value. To retrieve additional items, we need to pass this cursor value back to Notion's API, either through a query in a GET request or using the body in a POST request.
   
2. Consequently, I will implement a mechanism to check 'has_more' in every response. If 'has_more' is true, we will extract the cursor value from the response and send another request, continuing this process until 'has_more' becomes false. Subsequently, we will merge all the responses into one array, which will be passed as the final response for subsequent implementations.
3. To verify the implementation and address potential performance issues, I will initially set the 'page_size' to 10 and create more than 10 pages in my workspace. Following this, I will send the query and assess its functionality to ensure its effectiveness.

• Overall you got the implementation but check by setting page_size to 100 itself, cause larger page_size (>10) is proved to work feasibly. Also note that lesser the number of page_size more the Network request. thus we will try to make minimum network request and get maximum page a each request, I believe will involve recursion concept.

• Create around 300 page with api with random names.

• Create around 150 pages with api which is part of record / db, Note: You might be aware every Notion Table Record / Row we have is also a page.

• Create around 150 DB with Random Properties with api

and Try to Fetch it using /search endpoint.

1. Search only Pages.
2. Search only Databases.
3. Search Pages and Databases both at a time.

See all this cases, you will have a chance to explore api.

[brf153]

@Nabhag8848 how will I check with page_size 100? Will I have to make more than 100 pages to check this?

[Nabhag8848]

@Nabhag8848 how will I check with page_size 100? Will I have to make more than 100 pages to check this?

Edited the Message. Yep Use Api to create it.

[brf153]

@Nabhag8848 oki I will do it