Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(archive): allow request concurrency towards PostgreSQL #1604

Closed
5 tasks done
Tracked by #1888
jm-clius opened this issue Mar 14, 2023 · 1 comment
Closed
5 tasks done
Tracked by #1888

chore(archive): allow request concurrency towards PostgreSQL #1604

jm-clius opened this issue Mar 14, 2023 · 1 comment
Assignees
Labels
E:PostgreSQL See https://github.com/waku-org/pm/issues/84 for details good first issue Good for newcomers

Comments

@jm-clius
Copy link
Contributor

jm-clius commented Mar 14, 2023

In order to allow for a scalable and fault-resistant Waku Archive technology as set out in waku-org/pm#12, we have decided to implement a PostgreSQL backend.

Work on adding basic support for PostgreSQL has started: #1590

A related work item, required for scalability, is to allow processing store requests concurrently. The standard library integration will, similar to the previous SQLite-based implementation, block on processing a store query, which is not scalable at high store query rates.

A simple solution may be to adapt https://github.com/treeform/pg to work with nim-chronos, i.e. an async wrapper functioning as an extension of the std library API.

cc @cammellos @LNSD

Tasks

  • Allow to put and retrieve messages from PostgreSQL database.
  • Make sure the interaction with the PostgreSQL is asynchronous.
  • Implement the whole "waku archive" through the postgresql driver.
  • Add the config option to allow wakunode2 interact with postgres database instead of sqlite.
  • [ ] Keep the sql queries in external .sql plain text files ( comment ) . Also, important to pay attention to the @LNSD comment: Add postgres basic support #1590 (comment) -> created a separate issue
  • [ ] Optimize the database. We need a rapid response when duplicate msgs happens. -> created a separate issue
  • Run the current sqlite unittest set for the postgresql driver.
  • [ ] Apply @Menduist 's enhancement suggestions for a more appropriate asynchronous handling: feat(common): added postgress async pool wrapper #1631 (comment) -> created a separate issue
  • [ ] Protect the nwaku compilation process so that the user has the libpq library properly installed. -> created a separate issue
  • [ ] Create a docker file to bring nwaku + PostgreSQL simultaneously. Ping @LordGhostX when completed. -> created a separate issue
@Ivansete-status
Copy link
Collaborator

This is Done.

@Ivansete-status Ivansete-status mentioned this issue Aug 7, 2023
8 tasks
@fryorcraken fryorcraken added the E:PostgreSQL See https://github.com/waku-org/pm/issues/84 for details label Sep 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
E:PostgreSQL See https://github.com/waku-org/pm/issues/84 for details good first issue Good for newcomers
Projects
Archived in project
Development

No branches or pull requests

3 participants