Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch insert suggestions #6080

Closed
4e6 opened this issue Mar 27, 2023 · 4 comments · Fixed by #6189
Closed

Batch insert suggestions #6080

4e6 opened this issue Mar 27, 2023 · 4 comments · Fixed by #6189
Assignees
@4e6
Labels
-language-server p-medium Should be completed in the next few sprints
Milestone
Beta Release

Comments

@4e6
Copy link
Contributor

4e6 commented Mar 27, 2023 •
edited
Loading

When inserting suggestions in the database, they are stored together with the arguments. This slows down the suggestions inserts.

Currently, suggestions are stored only for the search request, and the arguments are unused. We can remove them to make inserts more efficient.

Task
@4e6 4e6 added p-medium Should be completed in the next few sprints -language-server labels Mar 27, 2023
@4e6 4e6 self-assigned this Mar 27, 2023
@JaroslavTulach JaroslavTulach moved this from ❓New to 📤 Backlog in Issues Board Mar 27, 2023
@jdunkerley jdunkerley added this to the Beta Release milestone Mar 28, 2023
@enso-bot
Copy link

enso-bot bot commented Mar 28, 2023

Dmitry Bushev reports a new STANDUP for today (2023-03-28):

Progress: Started working on the task. Started removing the arguments table. Found out that it can be used during the project renaming. After some consideration decided that we can only store the arguments for project suggestions. Started implementing batch insert query. It should be finished by 2023-03-31.

Next Day: Next day I will be working on the #6080 task. Continue working on the task

This was referenced Mar 29, 2023
Closed
Merged
mergify bot pushed a commit that referenced this issue Mar 31, 2023
@4e6
When renaming the project clean old modules instead of updating (#6148)
2338e5d 
close #6139
close #6137

When the project is renamed, the engine cleans up affected modules and initiates modules re-indexing to fill the suggestions database with new records. This way it reduces the amount of information stored in the suggestions database and helps implement #6080 optimization.

Changelog:
- remove: rename features from the suggestions database
- update: rename command to initiate modules cleanup and project re-execution
- fix: #6137
@enso-bot
Copy link

enso-bot bot commented Apr 1, 2023

Dmitry Bushev reports a new STANDUP for yesterday (2023-03-31):

Progress: Retuning to the task. Implemented batch insert. Removed suggestion arguments from the database logic. Updated LS suggestions handling with the new database API. Fixed suggestion database tests. It should be finished by 2023-03-31.

Next Day: Next day I will be working on the #6080 task. Continue working on the task

@4e6 4e6 mentioned this issue Apr 3, 2023
Merged
3 tasks
@enso-bot
Copy link

enso-bot bot commented Apr 3, 2023

Dmitry Bushev reports a new 🔴 DELAY for today (2023-04-03):

Summary: There is 3 days delay in implementation of the Batch insert suggestions (#6080) task.
It will cause 0 days delay for the delivery of this weekly plan.

Delay Cause: Had to switch to #6139 while this task was in progress

@enso-bot
Copy link

enso-bot bot commented Apr 3, 2023

Dmitry Bushev reports a new STANDUP for today (2023-04-03):

Progress: Continue with the task. Cleaned up unused methods from SuggestionsRepo. Cleaned up the arguments table. Measured the performance. Fixed legal review report. Cleaned up and prepared the PR It should be finished by 2023-04-03.

Next Day: Next day I will be working on the #6080 task. Continue working on the task

@jdunkerley jdunkerley moved this from 📤 Backlog to 👁️ Code review in Issues Board Apr 4, 2023
@mergify mergify bot closed this as completed in #6189 Apr 6, 2023
mergify bot pushed a commit that referenced this issue Apr 6, 2023
@4e6
Batch insert suggestions (#6189)
df4491d 
close #6080

Changelog
- add: implement `SuggestionsRepo.insertAll` as a batch SQL insert
- update: `search/getSuggestionsDatabase` returns empty suggestions. Currently, the method is only used at startup and returns the empty response anyway because the libs are not loaded at that point.
- update: serialize only global (defined in the module scope) suggestions during the distribution building. There's no sense in storing the local library suggestions.
- update: sqlite dependency
- remove: unused methods from `SuggestionsRepo`
- remove: Arguments table

# Important Notes
Speeds up libraries loading by ~1 second.

![2023-04-03-173423_2086x324_scrot](https://user-images.githubusercontent.com/357683/229597470-19dcc010-2a34-43e1-87be-60af99afd275.png)
![2023-04-03-173514_2083x321_scrot](https://user-images.githubusercontent.com/357683/229597476-bf5b3c33-6321-4ac9-a0ca-2fb57d257857.png)
@github-project-automation github-project-automation bot moved this from 👁️ Code review to 🟢 Accepted in Issues Board Apr 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@4e6 4e6

Labels
-language-server p-medium Should be completed in the next few sprints
Projects
Issues Board
Archived in project
Milestone
Beta Release
Development

Successfully merging a pull request may close this issue.

Batch insert suggestions enso-org/enso
2 participants
@4e6 @jdunkerley