Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: enable the preimage flag for paritydb #3007

Merged
merged 2 commits into from
Jun 19, 2023
Merged

Conversation

lemmih
Copy link
Contributor

@lemmih lemmih commented Jun 19, 2023

Summary of changes

Changes introduced in this pull request:

  • Recently, we've moved to only store IPLD data in ParityDB. This let's us turn on more aggressive optimizations.
  • This PR enables the preimage flag which improves performance in very specific situations.

Benchmarks done using storage-optimized droplets with 16GiB of RAM and 300 GiB of SSD.

Import time (seconds) Peak RSS
Importing a mainnet snapshot 1563 ± 132 8.76GiB ± 0.3GiB
Re-importing the same snapshot 2837 ± 30 7.24 GiB ± 0.01 GiB
Re-importing with the preimage flag 1837 ± 16 5.73 GiB ± 0.13 GiB

Reference issue to close (if applicable)

Closes

Other information and links

Change checklist

  • I have performed a self-review of my own code,
  • I have made corresponding changes to the documentation,
  • I have added tests that prove my fix is effective or that my feature works (if possible),
  • I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

@lemmih
Copy link
Contributor Author

lemmih commented Jun 19, 2023

Note: This invalidates existing databases. They have to be manually cleaned. We probably want a better way of handling this.

@lemmih
Copy link
Contributor Author

lemmih commented Jun 19, 2023

Data from Hubert's machine (which is much more powerful than the droplets):

main first
2023-06-19T10:32:55.032075Z  INFO forest_filecoin::daemon::daemon: Imported snapshot in: 313s

main second
2023-06-19T10:58:05.854629Z  INFO forest_filecoin::daemon::daemon: Imported snapshot in: 1462s

preimage first
2023-06-19T11:05:32.713542Z  INFO forest_filecoin::daemon::daemon: Imported snapshot in: 307s 

preimage second
2023-06-19T11:26:56.904707Z  INFO forest_filecoin::daemon::daemon: Imported snapshot in: 1233s

preimage helps a bit but the performance penalty is still extreme.

@LesnyRumcajs
Copy link
Member

Specs for reference:

         .';:cccccccccccc:;,.            --------------------
      .;cccccccccccccccccccccc;.         OS: Fedora release 38 (Thirty Eight) x86_64
    .:cccccccccccccccccccccccccc:.       Host: Z790 UD AX
  .;ccccccccccccc;.:dddl:.;ccccccc;.     Kernel: 6.3.6-200.fc38.x86_64
 .:ccccccccccccc;OWMKOOXMWd;ccccccc:.    Uptime: 6 hours, 16 mins
.:ccccccccccccc;KMMc;cc;xMMc:ccccccc:.   Packages: 2159 (rpm), 2 (flatpak)
,cccccccccccccc;MMM.;cc;;WW::cccccccc,   Shell: fish 3.6.1
:cccccccccccccc;MMM.;cccccccccccccccc:   Resolution: 3840x2160
:ccccccc;oxOOOo;MMM0OOk.;cccccccccccc:   DE: GNOME 44.1
cccccc:0MMKxdd:;MMMkddc.;cccccccccccc;   WM: Mutter
ccccc:XM0';cccc;MMM.;cccccccccccccccc'   WM Theme: Adwaita
ccccc;MMo;ccccc;MMW.;ccccccccccccccc;    Theme: Adwaita [GTK2/3]
ccccc;0MNc.ccc.xMMd:ccccccccccccccc;     Icons: Adwaita [GTK2/3]
cccccc;dNMWXXXWM0::cccccccccccccc:,      Terminal: tmux
cccccccc;.:odl:.;cccccccccccccc:,.       CPU: 13th Gen Intel i9-13900KF (32) @ 5.500GHz
:cccccccccccccccccccccccccccc:'.         GPU: NVIDIA GeForce RTX 4090
.:cccccccccccccccccccccc:;,..            Memory: 5867MiB / 64119MiB

Disk is https://www.kingston.com/en/ssd/kc3000-nvme-m2-solid-state-drive (7G/s write/read)

@lemmih lemmih enabled auto-merge (squash) June 19, 2023 15:31
@lemmih lemmih merged commit fe552fa into main Jun 19, 2023
19 checks passed
@lemmih lemmih deleted the lemmih/parity-preimage branch June 19, 2023 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants