Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: deactivate bns table indices before subdomain import #1086

Merged
merged 1 commit into from
Feb 28, 2022

Conversation

rafaelcr
Copy link
Collaborator

@rafaelcr rafaelcr commented Feb 28, 2022

This patch temporarily disables indices on the subdomains, zonefiles, namespaces, and names tables before subdomain import, and also removes unused indices from these tables.

From postgres docs on populating a database:

If you are adding large amounts of data to an existing table, it might be a win to drop the indexes, load the table, and then recreate the indexes.

Instead of dropping and re-creating, this code just disables and reindexes them after import is complete via the use of the indisready and indisvalid index state flags.

On local tests, it now takes ~2 seconds to import 10,000 subdomains when before this change it took ~30 seconds per 10,000.

Fixes #1081

@codecov-commenter
Copy link

codecov-commenter commented Feb 28, 2022

Codecov Report

Merging #1086 (949a131) into beta (296c619) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             beta    #1086      +/-   ##
==========================================
- Coverage   72.82%   72.81%   -0.01%     
==========================================
  Files          90       90              
  Lines        9813     9811       -2     
  Branches     1907     1907              
==========================================
- Hits         7146     7144       -2     
  Misses       2553     2553              
  Partials      114      114              
Impacted Files Coverage Δ
src/migrations/1608030374841_namespaces.ts 100.00% <ø> (ø)
src/migrations/1608030374842_names.ts 100.00% <ø> (ø)
src/datastore/postgres-store.ts 86.80% <100.00%> (ø)
src/import-v1/index.ts 88.46% <100.00%> (+0.27%) ⬆️
src/migrations/1610030345948_subdomains.ts 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba2d4de...949a131. Read the comment docs.

@github-actions
Copy link

@github-actions github-actions bot temporarily deployed to pull request February 28, 2022 04:48 Inactive
Copy link
Member

@zone117x zone117x left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat postgres feature!

@rafaelcr rafaelcr merged commit d8d4d4c into beta Feb 28, 2022
@rafaelcr rafaelcr deleted the fix/subdomain-import-speed branch February 28, 2022 14:38
blockstack-devops pushed a commit that referenced this pull request Feb 28, 2022
# [3.0.0-beta.2](v3.0.0-beta.1...v3.0.0-beta.2) (2022-02-28)

### Bug Fixes

* deactivate indices before subdomain import ([#1086](#1086)) ([d8d4d4c](d8d4d4c))
* index principal_stx_txs tx_id to speed up reorg updates ([#1080](#1080)) ([f6d7d0c](f6d7d0c))
@blockstack-devops
Copy link

🎉 This PR is included in version 3.0.0-beta.2 🎉

The release is available on:

Your semantic-release bot 📦🚀

blockstack-devops pushed a commit that referenced this pull request Mar 3, 2022
# [3.0.0](v2.1.1...v3.0.0) (2022-03-03)

### Bug Fixes

* capture re-organized txs correctly in `/extended/v1/:address/transactions` ([#1074](#1074)) ([81d039d](81d039d))
* deactivate indices before subdomain import ([#1086](#1086)) ([d8d4d4c](d8d4d4c))
* index principal_stx_txs tx_id to speed up reorg updates ([#1080](#1080)) ([f6d7d0c](f6d7d0c))
* principal_stx_txs sorting ([#1056](#1056)) ([b0a0e94](b0a0e94))
* remove unused indices, add others for re-org queries ([#1087](#1087)) ([2a2fb8d](2a2fb8d))
* sort NFT events by event_index too ([#1063](#1063)) ([77b2587](77b2587))

* chore!: major version bump for breaking db schema changes ([296c619](296c619))

### Features

* add `chain_tip` materialized view to track chain tip stats ([#1028](#1028)) ([803ac18](803ac18))
* add chain tip info to `/extended/v1/status` ([#1070](#1070)) ([fb573b1](fb573b1))
* added feature for rendering docs ([#991](#991)) ([a521a39](a521a39))
* change string and hex column indices to Hash method ([#1042](#1042)) ([aae6cc0](aae6cc0))

### BREAKING CHANGES

* use event-replay to upgrade, this version includes breaking changes to the db sql schema
@blockstack-devops
Copy link

🎉 This PR is included in version 3.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants