refactor(archive): Moving waku archive logic from app.nim to the archive module #1817

Ivansete-status · 2023-06-22T11:22:52Z

Description

This PR performs a refactoring to simplify the app.nim, and move all the waku archive logic into the archive.nim module.

The App object shouldn't directly know about the "archive" protocol or the "retention policy". IMHO, these concepts should be managed by the archive module.

This movement is interesting from the perspective of new incoming archive drivers.

This is how I see a natural chain of responsibility:

app -> node -> wakuArchive -> driver (sqlite now but postgres in the next PR to be merged)

SionoiS

LGTM

SionoiS · 2023-06-22T20:30:00Z

tests/v2/waku_archive/test_waku_archive.nim

@@ -467,7 +440,6 @@ procSuite "Waku Archive - find messages":
      response.cursor.isNone()

  test "handle temporal history query with a valid time window":
-    ## Given


Why remove this?

ah yes, this is totally beyond the scope of this PR. I will set it back

SionoiS · 2023-06-22T20:45:38Z

waku/v2/node/waku_node.nim


-  discard setTimer(Moment.fromNow(interval), executeRetentionPolicy)
+    node.wakuArchive.startMessageRetentionPolicyPeriodicTask(30.minutes)


Hardcoded 30m? Do we have an issue for making this configurable?

We don't have an issue for that but I think is a good idea indeed :)

jm-clius

Ah! Apologies for not being able to approve this yet. :) Many of the changes seem reasonable, I mainly think my issue here is that the archive driver interface is designed to be generic and to leave the choice and set up of specific driver backends to the application. I agree that we can move the driver-specific setup code to not clutter app.nim, but I'd put them in a place close to the application (or at least not in the generic archive.nim - which should be seen as similar to an interface). I hope my comments below make sense, but happy to discuss in more detail if perspectives diverge a lot here. :)

jm-clius · 2023-06-23T09:29:39Z

waku/v2/waku_archive/archive.nim

+          storeMessageDbUrl: string,
+          storeMessageDbVacuum: bool = false,
+          storeMessageDbMigration: bool = false,
+          storeMessageRetentionPolicy: string = "none"):


I disagree that the Archive Driver should have any knowledge about underlying implementations. An Archive does not even have to be a DB at all - it could simply be an internal memory queue. The point of having a separate archive driver is exactly so that applications can set up and provide their own archive backend to the generic archive driver, with no information about application specific choices (do I have harddrives? do I have my own proprietary DB?) being passed down.

jm-clius · 2023-06-23T09:31:44Z

waku/v2/node/waku_node.nim

+                   storeMessageDbUrl: string,
+                   storeMessageDbVacuum: bool = false,
+                   storeMessageDbMigration: bool = false,
+                   storeMessageRetentionPolicy: string = "none"):


The node does not receive a "DB" it receives an Archive as provided by the application - it should not have to have any knowledge about this external component. This may in fact not even be a DB, but simply a chunk of memory.

jm-clius · 2023-06-23T09:39:44Z

waku/v2/waku_archive/archive.nim

+
+  discard setTimer(Moment.fromNow(interval), executeRetentionPolicy)
+
+proc setupWakuArchiveDriver(url: string, vacuum: bool, migrate: bool):


This is application logic: decisions about how the configured url should be interpreted and which specific backend storage systems should be set up for the application lies with the application. When adding new drivers (e.g. someone now supports Codex distributed storage) the point of a driver interface is exactly that the backend operation would not have to be changed or even be aware of this fact. In this case, the application would have to request changes in the setup instructions which couples the driver interface with application-defined underlying implementations.

jm-clius · 2023-06-23T09:57:04Z

The App object shouldn't directly know about the "archive" protocol or the "retention policy".

I disagree. I think only the App should know what e.g. "postgresql" means. Database technologies (and other technologies for storage) are not Waku protocols so a Waku p2p node should not know about the internals here. It simply provides a plugin interface and an application can then configure a service of its choice that works with the plugin.

or the "retention policy".

Mmm. There may indeed be some generic concepts here that can be further abstracted from the application. Thanks for pointing this out! I do think, however, that the "simplest" change here may be to keep the setup code here similarly close to the application and ensure that it works with postgresql as well. But happy to consider other ideas here. :)

app -> node -> wakuArchive -> driver

<-- I think this is true. What shouldn't be added here is that the archive is aware of different driver implementations, which moving the implementation-specific setup code into the archive interface would achieve. :)

Ivansete-status · 2023-06-26T07:27:47Z

I've re-submitted the PR seeking to follow your suggestions, @jm-clius :) Thanks for them!

jm-clius

Thanks! I think this is a much better pattern than we had before. Thanks for your patience and taking the time to clean this up. Note my comments/nitpicks below. One comment is that this signifies a significant number of changes (also for SQLite DBs) which we'll only be able to fully test in a production environment. When rolling this out in a next release, I'd suggest (1) testing basic functions in collaboration with go-waku and (2) deploying only to a few production nodes at first.

jm-clius · 2023-06-26T13:14:37Z

waku/v2/node/waku_node.nim


-  discard setTimer(Moment.fromNow(interval), executeRetentionPolicy)
+    node.wakuArchive.startMessageRetentionPolicyPeriodicTask(30.minutes)


Nit: Better to extract this to a const just before the function than to make this a "magic value" inline with the code: easier to debug; clear where to add config if needed.

jm-clius · 2023-06-26T13:15:43Z

apps/wakunode2/app.nim

 proc setupPeerStorage(): AppResult[Option[WakuPeerStorage]] =
-  let db = ?setupDatabaseConnection(PeerPersistenceDbUrl)
+  let db = ? SqliteDatabase.new("peers.db")


Nit: Better to extract this to a const just before the function than to make this a "magic value" inline with the code: easier to debug; clear where to add config if needed.

jm-clius · 2023-06-26T13:19:39Z

waku/v2/waku_archive/driver/postgres_driver/postgres_driver.nim

@@ -11,7 +11,7 @@ import
 import
  ../../../waku_core,
  ../../common,
-  ../../driver,
+  ../../driver_base,


Nit: I wouldn't call this _base. We didn't previously follow this pattern (see e.g. "node" and "builder"). It's clear enough that the builder for the driver is distinct from the driver. I'd either go for driver.nim and driver_builder or, preferably, driver.nim and builder.nim as it's fully qualified in import path: driver/driver.nim and driver/builder.nim. The latter would be the established pattern.

jm-clius · 2023-06-26T13:20:17Z

waku/v2/waku_archive/retention_policy/retention_policy_capacity.nim

-  ../driver,
-  ../retention_policy
+  ../driver_base,
+  ../retention_policy_base


Nit: I wouldn't call this _base. See my comment elsewhere on the established pattern here.

…lders

jakubgs · 2023-06-27T13:36:06Z

Looks like this resulted in all nodes on wakuv2.test restarting due to:

admin@node-01.do-ams3.wakuv2.test:~ % d run --rm -it statusteam/nim-waku:deploy-wakuv2-test --version
could not load: libpq.so(.5|)
(compile with -d:nimDebugDlOpen for more information)

So I've made a PR to sanity-check the Docker image startup:

fix: sanity-check the docker image start #1825

Ivansete-status force-pushed the refactor-archive branch 4 times, most recently from cbab1f4 to 7a8d08c Compare June 22, 2023 12:02

Ivansete-status changed the title ~~refactor(app/archive) Moving waku archive logic from app.nim to the archive module~~ refactor(app/archive): Moving waku archive logic from app.nim to the archive module Jun 22, 2023

Ivansete-status changed the title ~~refactor(app/archive): Moving waku archive logic from app.nim to the archive module~~ refactor(archive) Moving waku archive logic from app.nim to the archive module Jun 22, 2023

Ivansete-status changed the title ~~refactor(archive) Moving waku archive logic from app.nim to the archive module~~ refactor(archive): Moving waku archive logic from app.nim to the archive module Jun 22, 2023

Ivansete-status force-pushed the refactor-archive branch 2 times, most recently from 5ecf5c0 to 8e04200 Compare June 22, 2023 13:04

Ivansete-status self-assigned this Jun 22, 2023

Ivansete-status marked this pull request as ready for review June 22, 2023 13:40

Ivansete-status requested review from jm-clius, alrevuelta and SionoiS June 22, 2023 13:40

SionoiS approved these changes Jun 22, 2023

View reviewed changes

jm-clius reviewed Jun 23, 2023

View reviewed changes

Ivansete-status added 5 commits June 23, 2023 23:11

Refactoring the Waku Archive. Simplifying the app.nim

e0f1e35

Restoring back deleted comment in 'test_waku_archive.nim'

78c902c

Better refactoring of archive and retention policy

5425992

app.nim: fix after wrong automatic merge results

8b77a78

Restoring back the tests

b60d6e3

Ivansete-status force-pushed the refactor-archive branch from 8e04200 to b60d6e3 Compare June 25, 2023 21:19

jm-clius approved these changes Jun 26, 2023

View reviewed changes

Ivansete-status added 3 commits June 26, 2023 17:00

waku_node: restoring back WakuArchiveDefaultRetentionPolicyInterval

14fa00a

Use of more standard naming for driver/retention_policy and their bui…

63040aa

…lders

app.nim: restoring 'const PeerPersistenceDbUrl' back

650da24

Ivansete-status merged commit 52894a8 into master Jun 27, 2023
15 checks passed

Ivansete-status deleted the refactor-archive branch June 27, 2023 11:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(archive): Moving waku archive logic from app.nim to the archive module #1817

refactor(archive): Moving waku archive logic from app.nim to the archive module #1817

Ivansete-status commented Jun 22, 2023 •

edited

SionoiS left a comment

SionoiS Jun 22, 2023

Ivansete-status Jun 23, 2023

SionoiS Jun 22, 2023

Ivansete-status Jun 23, 2023

jm-clius left a comment

jm-clius Jun 23, 2023

jm-clius Jun 23, 2023

jm-clius Jun 23, 2023

jm-clius commented Jun 23, 2023

Ivansete-status commented Jun 26, 2023

jm-clius left a comment

jm-clius Jun 26, 2023

jm-clius Jun 26, 2023

jm-clius Jun 26, 2023

jm-clius Jun 26, 2023

jakubgs commented Jun 27, 2023


		discard setTimer(Moment.fromNow(interval), executeRetentionPolicy)
		node.wakuArchive.startMessageRetentionPolicyPeriodicTask(30.minutes)


		discard setTimer(Moment.fromNow(interval), executeRetentionPolicy)

		proc setupWakuArchiveDriver(url: string, vacuum: bool, migrate: bool):

refactor(archive): Moving waku archive logic from app.nim to the archive module #1817

refactor(archive): Moving waku archive logic from app.nim to the archive module #1817

Conversation

Ivansete-status commented Jun 22, 2023 • edited

Description

SionoiS left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jm-clius left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jm-clius commented Jun 23, 2023

Ivansete-status commented Jun 26, 2023

jm-clius left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakubgs commented Jun 27, 2023

Ivansete-status commented Jun 22, 2023 •

edited