Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Task]: Introduce additional perf instrumentation for datasources section of consolidated APIs #33711

Closed
sneha122 opened this issue May 24, 2024 · 0 comments · Fixed by #33712
Closed

[Task]: Introduce additional perf instrumentation for datasources section of consolidated APIs #33711

sneha122 opened this issue May 24, 2024 · 0 comments · Fixed by #33712
Assignees
@sneha122
Labels
Integrations Pod General Issues related to the Integrations Pod that don't fit into other tags. Integrations Pod Issues related to a specific integration Task A simple Todo

Comments

@sneha122
Copy link
Contributor

SubTasks

Consolidated API is responsible for fetch all application resources at once, as part of this API we also fetch list of datasources. In this task we need to add additional perf instrumentation to see how much time it takes to fetch all datasources, their datasource storages and collect them.
@sneha122 sneha122 added Task A simple Todo Integrations Pod General Issues related to the Integrations Pod that don't fit into other tags. labels May 24, 2024
@sneha122 sneha122 self-assigned this May 24, 2024
@github-actions github-actions bot added the Integrations Pod Issues related to a specific integration label May 24, 2024
@sneha122 sneha122 mentioned this issue May 24, 2024
Merged
2 tasks
sneha122 added a commit that referenced this issue Jun 3, 2024
@sneha122
fix: [Perf Improvement] removed unnecessary individual DB calls to pl…
3c38ca9 
…ugin and fetched al… (#33712)

## Description

This PR adds:
- Additional instrumentation around `datasources` section of
consolidated API

Datasources fetch process looks as below:
1. It first fetches all datasources belonging to a workspace
2. For each datasources, fetches its datasourceStorages
a. Calls populateHintMessages to populate messages property in
datasourceStorage
     b. Fetches plugin from DB
     c. Gets pluginExecutor from plugin
     d. calls getHintMessages on pluginExecutorMono
3. Marks recently used datasources (sorts all datasources based on
createdAt in desc order and marks top 3 as recent)

This PR adds improvements around point 2.b so that we fetch all plugins
at once in one DB call and then use that to process hint messages for
all datasourceStorages, this avoiding multiple calls to DB


Fixes #33711 
Fixes #33678
_or_  
Fixes `Issue URL`
> [!WARNING]  
> _If no issue exists, please create an issue first, and check with the
maintainers if the issue is valid._

## Automation

/ok-to-test tags="@tag.Datasource"

### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results  -->
> [!TIP]
> 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/9347912473>
> Commit: 1564331
> Cypress dashboard url: <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=9347912473&attempt=1"
target="_blank">Click here!</a>

<!-- end of auto-generated comment: Cypress test results  -->








## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [ ] No

---------

Co-authored-by: “sneha122” <“sneha@appsmith.com”>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sneha122 sneha122

Labels
Integrations Pod General Issues related to the Integrations Pod that don't fit into other tags. Integrations Pod Issues related to a specific integration Task A simple Todo
Projects
None yet
Milestone
No milestone
1 participant
@sneha122