From 7848d62e46188da927fb7478c201512df5aee061 Mon Sep 17 00:00:00 2001 From: Dotun Jolaoso Date: Sat, 25 Jun 2022 19:00:58 +0100 Subject: [PATCH 1/7] chore: update readme with latest image (#805) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b30dbf7c81..2e51f6331f 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ $ docker run \ -p 5005:5005 \ --name convoy-server \ -v `pwd`/convoy.json:/convoy.json \ - packages.getconvoy.io/frain-dev/convoy:v0.4.9 + docker.cloudsmith.io/convoy/convoy/frain-dev/convoy:v0.6.0-rc.1 ``` You can view a sample configuration here - [convoy.json](https://github.com/frain-dev/convoy/blob/main/convoy.json.example). From 7d54b600d9e6818fdb6066da7909470a1e9d4e62 Mon Sep 17 00:00:00 2001 From: Raymond Tukpe Date: Mon, 27 Jun 2022 10:47:22 +0100 Subject: [PATCH 2/7] Merge Release 0.6 into main (#804) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: update changelog * Bump version to v0.6.0-rc * Release 0.6.0-rc.1 (#802) * Megre release-0.6 into main (#682) * feat: update changelog * Bump version to v0.6.0-rc * Updated Projects API (#684) * Update Groups API * Deprecate DefaultGroup * Update how integration tests are run * Deprecate immune tests * Change Duration to duration Co-authored-by: Raymond Tukpe Co-authored-by: Raymond Tukpe * Updated API to use defaults for both incoming and outgoing (#686) * Update Groups API * Deprecate DefaultGroup * Update how integration tests are run * Deprecate immune tests * Updated API to use defaults for both incoming and outgoing * Removed badger files * Setup cron system (#691) * Setup cron system * removed retry events schedule * add support for jwt authentication (#690) * add support for jwt authentication * feat: add support for blacklisting tokens and logout endpoint * chore: update route * add test for user repository * add user service test * add user integration test * add jwt test * add jwt realm test * chore: fix failing test * chore: add docs * chore: refactor jwt config to be standalone * update routes * chore: add duration comment * add ReplayAppEvent endpoint (#672) * feat: add ReplayAppEvent endpoint * tests: add tests * fix: specify wantErr * fix: change url to /events/{eventID/replay * tests: fix testdb.SeedEvent call Co-authored-by: Subomi Oluwalana * add crud for organisations (#685) * feat: add crud for organisations * feat: implement crud for organisations * feat: implement crud for organisations in BadgerDB * tests: add tests * tests: add integration build tags * tests: fix sorting * tests: fix sorting * tests: fix TestDeleteOrganisation * tests: fix tests * tests: fix tests * tests: add tests for organisation_service.go * tests: add organisation_integration_test.go * tests: add integration build tag * fix: add requireOrganisation to organisation sub route * tests: fix tests tests: add Test_UpdateOrganisation_EmptyOrganisationName & Test_CreateOrganisation_EmptyOrganisationName * tests: fix tests * fix: implement suggestions * tests: fix urls * fix initRealmChain call * docs: regenerate docs * Add missing comma to convoy.json.example (#700) * Generic Mailers Setup (#707) * Refactored mails * Removed test line * Removed hardcoded prefix * Enhancement: Refactor to use asynq queues (#704) * Feat: Refactor to use asynq queues * Add retry mechanism, clean up * Fix CI lint * Fix: parse redis url * Fix: redis dsn in client test * Add queue prometheus metrics * Chore: redis address in queue options * Chore: clean up consumer API * chore: add err checks Co-authored-by: Raymond Tukpe * Add subscriptions and sources (#683) * feat: add subscription resource * feat:m add subsciption repo tests * Merge branch 'main' of https://github.com/frain-dev/convoy into raymond/feat/add-subscriptions * feat: update subscription * chore: remove endpoint events * feat: update subscription schema and model * feat: use noemalised data modelling * feat: test outgoing event flow end to end * chore: add project/group type validation * complete UI rebuild (#681) * Ingestion API (#688) * Setup Ingestion API * Remove unnecessary line * check source type is HTTP before proceeding * feat: fix flow for incoming and outgoing events * Update process event delivery tests (#694) * chore: update process event delivery tests * chore: fix CI lint * chore: remove badger tests * chore: fix group repo tests * chore: update event delivery tests * chore: fix retry event delivery tests * chore: update FillGroupsStatistics and ForceRetryEventDeliveries tests * chore: add integration tag build directive * chore: fix integration tests * chore: implement PR feedback * chore: rename tests * Update subscription tests (#699) * chore: add LoadSubscriptionsPaged tests * feat: update mongodb indexes * chore: added create and update scubscription tests * chore: add delete scubscription tests * Add subscription integration tests (#701) * chore: add update and delete subscription tests * chore: add fetch all subscriptions test * Emmanuel/feature/new project structure (#689) * complete UI rebuild * connect apis for source * updated app and events for v2 * events/apps integration for v2 (#695) * events/apps integration for v2 * additional updates for app/events v2 * additional updates * additional ui updates for apps v2 * projects integration v2 * additional ui updates for v2 * Update projects page (#702) * show project type on projects page * undo style link * added copies for tooltips * updated copies for create projects * updated create app component * updated create projects * projects integration v2 (#696) * projects integration v2 * additional ui updates for v2 * Update projects page (#702) * show project type on projects page * undo style link * added copies for tooltips * updated copies for create projects * updated create app component * updated create projects * add spaces, 🥺 Co-authored-by: Emmanuel Aina * update on new app endpoint * additional fixes for apps * fixed events input focus * took out event required tag * Subscriptions UI (#705) * build subscriptions and create subscrips move project id to private service level * build subscriptions details * commit fix 1 * conflict fix * clean up * fixed create project json structure * build settings page (#706) Co-authored-by: Emmanuel Aina Co-authored-by: Pelumi Honey * chore: remove extra subrouter * Merge main (#708) * Setup cron system (#691) * Setup cron system * removed retry events schedule * add support for jwt authentication (#690) * add support for jwt authentication * feat: add support for blacklisting tokens and logout endpoint * chore: update route * add test for user repository * add user service test * add user integration test * add jwt test * add jwt realm test * chore: fix failing test * chore: add docs * chore: refactor jwt config to be standalone * update routes * chore: add duration comment * add ReplayAppEvent endpoint (#672) * feat: add ReplayAppEvent endpoint * tests: add tests * fix: specify wantErr * fix: change url to /events/{eventID/replay * tests: fix testdb.SeedEvent call Co-authored-by: Subomi Oluwalana * add crud for organisations (#685) * feat: add crud for organisations * feat: implement crud for organisations * feat: implement crud for organisations in BadgerDB * tests: add tests * tests: add integration build tags * tests: fix sorting * tests: fix sorting * tests: fix TestDeleteOrganisation * tests: fix tests * tests: fix tests * tests: add tests for organisation_service.go * tests: add organisation_integration_test.go * tests: add integration build tag * fix: add requireOrganisation to organisation sub route * tests: fix tests tests: add Test_UpdateOrganisation_EmptyOrganisationName & Test_CreateOrganisation_EmptyOrganisationName * tests: fix tests * fix: implement suggestions * tests: fix urls * fix initRealmChain call * docs: regenerate docs * Add missing comma to convoy.json.example (#700) * chore: fixed tests and build Co-authored-by: Subomi Oluwalana Co-authored-by: Dotun Jolaoso Co-authored-by: Daniel Oluojomu Co-authored-by: Daniel Perrefort * Update date and time filters (#709) * Update styles source (#710) * Update date and time filters * update style source * chore: merge main into subscripions branch * chore: delete org tests Co-authored-by: Emmanuel Aina Co-authored-by: Subomi Oluwalana Co-authored-by: Pelumi Honey Co-authored-by: Dotun Jolaoso Co-authored-by: Daniel Oluojomu Co-authored-by: Daniel Perrefort * Enhancement: Asynq scheduler (#711) * Add asynq scheduler * Fix typo: ScheduleQueue * stop scheduler consumer * Combine task and task handler registration * Tie Users into organisations (#697) * feat: implement organisation member repository * feat: add metadata to auth.AuthenticatedUser * feat: add OrganisationInviteRepository * feat: add OrganisationMemberService & OrganisationInviteService * feat: add ErrEmptyBody * feat: add organisation invite handlers * tests: add tests for organisation_invite.go & organisation_member.go * fix: fix badger & orgMemberRepo * fix: fix badger * fix: initialise new repo * fix: fix organisation_service_test.go * tests: fix tests * tests: fix tests * tests: fix auth header * tests: fix OrganisationIntegrationTestSuite * tests: fix TestDeleteOrganisationInvite * tests: add organisation_member_service_test.go * fix: split ProcessOrganisationMemberInvite * tests: add more test cases to TestOrganisationInviteService_AcceptOrganisationMemberInvite * tests: add more test cases to TestOrganisationInviteService_AcceptOrganisationMemberInvite * fix: add organisationID param * fix: add organisation member routes * fix: seed superuser member for organisation owner * fix: fix doc * tests: add organisation_member_integration_test.go * fix: pass orgMemberRepo * fix: add pagination middleware to GetOrganisationMembers route * fix: add organisation_invite_integration_test.go * docs: fix docs * docs: fix owner_id json tag * fix: fix app queue param * tests: fix tests * fix: add unique email index for UserCollection * tests: seed organisation members * tests: panic on db connect error * tests: fix tests * tests: fix badger & DeleteOrganisationMember calls * tests: fix tests * fix: add LoadUserOrganisationsPaged * fix: fix LoadUserOrganisationsPaged pipleine * tests: fix Test_GetOrganisations * fix: require only token to process organisation invite * tests: increase TestOrganisationInviteService_ProcessOrganisationMemberInvite coverage * fix: validate new user * fix: move process invite route to ui * docs: define Organisation tag * feat: add invite expiry * tests: fix tests * Fixed Dockerfile (#722) * Update events and event deliveries payload (#720) * feat: add app metadata when fetching applications * feat: add metadata fields to event deliveries endpoint * chore: update tests * auth integration (#716) * auth integration * updates on organisation dropdwon * fixed http service bug * fixed http service bug * fix filters in events and event deliveries (#718) * fix filters in events and event deliveries * small touches * fix style source * fix all loaders in project page (#724) * Organisation settings ui (#719) * auth integration * organisations ui * organisations ui * updates on organisation dropdwon * updated settings page * fixed http service bug * fixed http service bug * took out billing * added delete organisation * added delete modal * chore: deprecate require auth (#728) * add user settings endpoints (#723) * feat: add user settings endpoints * chore: add user repository update test * chore: add user service test * chore: add user integration test * chore: add docs * feat: add user exists endpoint * Fixed ui build script (#730) * integrate email sender with organisation invite (#717) * feat: add invite email sending logic * feat: add organisation.invite.html * feat: - add TemplateName type - add more fields to Notification * feat: - add requireBaseUrl to /organisations route - use getUserFromContext * fix: move emailNotificationSender into preRun * feat: send invite email in new goroutine * feat: add notification task * feat: add notification task * tests: fix tests * refactor ui routes (#733) * refactor ui routes * add user routes * chore: fix failing test * chore: remove unused middleware for now * feat: add support for finding user by invite token * chore: remove user exists endpoint * chore: generate docs * chore: refactor user routes * chore: generate docs * Add subscription metadata (#738) * feat: add subscription metadata * feat: add metadata fields to event deliery * chore: remove useless internal function * feat: fix ingester event flow (#744) * fix: return correct error message for duplicate group name (#741) * Chore: remove example scheduler task (#745) * remove example scheduler task * Remove unwanted lines * Fix project creation onboarding (#726) * method for closing subscripton modal * add create subscription to project onboarding * update flow * connect project onboarding * fix project onboarding flow * update private services url * fix: source url (#747) * Fix update group reosurce and group service tests (#746) * feat: fix update group and service tests * feat: config is not required when creating a new project * patch: show app event count (#748) * Implement delete project (UI) (#749) * delete project update project onboarding * fix style source * updated subscriptions (#731) * updated subscriptions * updates on subscriptions * made subscription configs optional * review updates * tiny fixes * undo angular.json changes Co-authored-by: horlah * Enhancement: Alphanumeric secrets (#751) * Update GenerateRandomString * Add comments * Fix verifier config (#752) * prevent nil source verifier * changed ApiKey field names * apply validation to updatesource * fix: return correct error message for duplicate group name (#753) * link groups to organisations (#740) * feat: link groups to organisations * tests: fix tests * tests: fix tests * tests: fix tests * tests: fix tests * fix: - move security routes into ui - require organisation membership in order to create api key for a group - fix tests * fix: - fix security_integration_test.go - add orgID to SeedDefaultGroup * tests: fix tests * tests: fix tests * tests: fix tests Co-authored-by: Subomi Oluwalana * chore: add metadata to get one subscription (#757) * add tests for security logic change (#756) * tests: increase coverage * fix: call requireGroup middleware before requirePermission * doc: add doc duplicate * fix: - check GroupID in Test_CreateAppPortalAPIKey - add a descriptor to duplicate doc * fix: move key generation logic into SeedAPIKey * test: add Test_CreateAppPortalAPIKey_AppDoesNotBelongToGroup * fetch organisation member user info (#729) * feat: fetch organisation member user info * tests: fix tests * tests: fix tests * tests: fix tests * tests: fix tests * tests: fix tests * fix: unify loop * tests: defer closeFn * update subscriptions list (#755) * update subscriptions list * build update update subscription * Change "base_url" to "host" (#754) * feat: change "base_url" to "host" * chore: force commit * chore: update other places * chore: rename context function names * chore: fix typo * add fetch pending invites api (#736) * feat: add fetch pending invites api * feat: add fetch pending invites api * tests: fix tests Co-authored-by: Subomi Oluwalana * add support for analytics (#698) * feat: add support for analytics * feat: add support for configuration resource * chore: remove unused import * chore: add cron for tracking analytcis * feat: add config repository test * chore: add configuration service test * chore: add configuration integration test * feat: add api version to configuration response * chore: add analytics test * chore: generate docs * feat: use separate tokens based on environment * chore: add cron schedule * feat: add support for analytics source * chore: remove repeated org repo * feat: add org name to event details * feat: add support for tracking host * Feat: Forgot Password (#742) * add verification repo * merge main * Add password reset email notification * Add integration tests * update email template * set subject on notifications * Fix: Template params * Fix: correct url in email * Update internal/email/templates/reset.password.html Co-authored-by: Raymond Tukpe * Initialize reset token same line Co-authored-by: Raymond Tukpe * fixed basic auth check (#759) * app endpoint edit/update feature (#737) * app endpoint edit/update feature * updated apps and app-details * added empty state img for projects * tiny update * review updates * took out commented code * fixed oboarding flow * took out console.log Co-authored-by: horlah * Update source (#758) * event delivery batch retry (#735) * event delivery batch retry * fix for endpoint error * updated event deliveries * added tab to url * took out commented code * Fix event deliveries and deliveries attempt page (#763) * fix event deliveries and deliveries attempt page fix event deliveries in event page update text in create source form * delete commented code * fix: return EndpointError from SendNotification (#764) * fix: add invite token to find user by token response (#765) * fix: fix invite status update (#767) * New Relic Integration (#739) * New Relic Integration * removed temporary fix in middleware.go * Fixed redis instrumentation * Fixed failing test * fix subscription source fetching for outgoing group (#769) * fix: fix subscription source fetching for outgoing group * tests: fix tests * tests: fix tests * Fix: add subscription id on delete endpoint (#771) * Added missing HttpTimeout field (#762) * scope unique group names to organisation (#761) * fix: scope unique group names to organisation * tests: fix Test_CreateGroup * fix outgoing group event type matching (#768) * fix: fix outgoing group behaviour with * event type * tests: add TestProcessEventCreated * tests: fix TestProcessEventCreated * tests: fix TestProcessEventCreated * fix: fix badger subscriptionRepo * Teams Ui (#712) * teams ui * teams ui * invite team member integration * team management flow * took out projects from invite team member page * took out creatProjectComponent * change in adding filter to url * updated css link * added updates for teams * fixed space issue on table * dropdown fix * Fix roles and scope groups API (#773) * Fixed UI permissions * Fixed failing tests * Fixed Makefile * fixed go lint * User management (#766) * added public components, completed accept invite flow * forgot password flow * reset password service * added reset password * updated reset password * added account settings * updated accept invite * additional changes * added button loader gif * ui changes * added accepted inite status, added forgot password on login * Fixed Group Filters (#774) * fix organisation invite template (#760) * docs" fix docs * fix: template * fix: fix error messages * add document_status filter to all find queries (#725) * fix: add document_status to all find queries * tests: fix tests * remove mongo dsn * tests: fix tests * tests: fix tests Co-authored-by: Subomi Oluwalana * fix organisation switching (#775) delete organisation in localstorage on logout refresh page on create new organisation to set that organisation fix create source * Fe bug fixes (#772) * bug fixes for apps and subscruptions * added loaders for app/event-delivery details * fixed app details loader * review fixes * took out async * Chore: clean up reset-password (#776) * Remove email from request, make reset_password_token unique * Fix CI tests * ignore token json * Build new app portal page (#770) * build new app portal page * enable delete and update subscription on app portal * fix event deliveries clear filter * build support with app portal sdk * reset style source * fix: use correct arguments for api key verifier (#779) * validate app, endpoint & source when creating subscription (#778) * fix: validate app & endpoint when creating subscription * tests: add tests * tests: fix tests * fix: validate incoming group source when creating subscription * tests: fix tests * tests: fix tests * fix: prevent organisation owner from being deactivated (#781) * feat: set default event types when filter config is nil (#783) * Add resend and cancel org invite endpoints (#782) * Add resend and cancel invite endpoints * Scope endpoints to invite resource * Move website (#785) * delete website * move website out * add toggle subscription status endpoint (#784) * feat: add toggle subscription status endpoint * feat: add toggle subscription status endpoint * fix: add default config for subscription (#787) * chore: change app portal link (#788) * Revert "chore: change app portal link (#788)" (#790) This reverts commit afdf3f2b84508af8f4e0b58442a462f89407aff9. * Add Custom source: Github (#791) * Added custom source: Github * Fixed failing tests * Removed unwanted line * support github custom ingester (#792) * Fixed a race condition that could occur when creating an application endpoint (#793) * fix: fixed a race condition that could occur when creating an application endpoint * chore: update tests * bugfixes for v6 (#786) * fixed project update error * removed email from reset password * copy changes, forgot password validation * updated create subscription form * added analytics * user redirect to dashboard on accepting invite * team invites updates * additional changes * moved analytics modal to middle * moved analytics modal to middle * updated analytics modal * app delete feature * updated delete app * review updates * review updates * updated org switch * feat: updated changelog Co-authored-by: Subomi Oluwalana Co-authored-by: Dotun Jolaoso Co-authored-by: Daniel Oluojomu Co-authored-by: Daniel Perrefort Co-authored-by: Ogban Ugot Co-authored-by: Emmanuel Aina Co-authored-by: Pelumi Honey * chore: prefer npm ci when installing nodejs dependecies (#803) * Bump version to v0.6.0-rc.1 * chore: revert makefile ui build command Co-authored-by: Subomi Oluwalana Co-authored-by: Dotun Jolaoso Co-authored-by: Daniel Oluojomu Co-authored-by: Daniel Perrefort Co-authored-by: Ogban Ugot Co-authored-by: Emmanuel Aina Co-authored-by: Pelumi Honey --- CHANGELOG.md | 25 +++++++++++++++++++++++++ Makefile | 4 ++-- VERSION | 2 +- scripts/build.sh | 2 +- 4 files changed, 29 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b79bbcced4..f2646c6962 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,28 @@ +## v0.6.0-rc.1 + +- [Change] Introduce organisations to partition different sets of projects. +- [Change] Deprecate file authentication and authorisation. You no longer specify authentication credentials from convoy.json. User and permission details are now persisted to the DB and use jwt for authentication. +- [Change] All users are now super users in the OSS core. +- [Change] Sentry error tracking has been deprecated. Only New relic is supported for error tracking. +- [Change] Revamped UI. The former convoy dashboard was revamped to enable more management of several vital resources - users, projects, applications, endpoints, sources, and subscriptions. +- [Change] require_auth has been deprecated. All endpoints will now require authentication. +- [Feature] Add Github Custom source #792 #791 +- [Enhancement] Change base_url config variable to host #754 +- [Enhancement] Set default event types when filter config is nil #783 +- [Enhancement] Switched background job system to asynq. #711 +- [Enhancement] Add toggle subscription status endpoint #784 +- [Enhancement] Autogenerated webhook secrets use alphanumeric secrets #751 +- [Enhancement] Use asynq for the scheduler. #745 +- [Bugfix] Prevent an organisation owner from being deactivated #781 +- [Bugfix] Fix events ingestion to create event flow #744 +- [Bugfix] Fixed a race condition that could occur when making an application endpoint #790 +- [Bugfix] Fixed app portal link. #790 +- [Bugfix] Use correct arguments for API key verifier #779 +- [Bugfix] Fixed switching between organisations #775 +- [Bugfix] Return proper error from SendNotification #764 +- [Bugfix] Fixed filters in events and event deliveries #718 +- [Bugfix] Fixed loaders in projects page #724 + ## v0.6.0-rc - [Enhancement] Optimize group statistics query #677 diff --git a/Makefile b/Makefile index 2533623023..568e8a218d 100644 --- a/Makefile +++ b/Makefile @@ -9,9 +9,9 @@ mockgen: setup: init-hooks -ui_install: +ui_install: cd web/ui/dashboard && \ - npm install && \ + npm ci && \ npm run build integration_tests: diff --git a/VERSION b/VERSION index e575be2475..01b366027c 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -v0.6.0-rc +v0.6.0-rc.1 diff --git a/scripts/build.sh b/scripts/build.sh index 75686574e1..3b008cefe9 100755 --- a/scripts/build.sh +++ b/scripts/build.sh @@ -11,7 +11,7 @@ mkdir $UIDIR cd ./web/ui/dashboard # Install dependencies -npm install +npm ci # Run production build npm run build From f0d4c205647c81443a7452c022b07879e1e75a4c Mon Sep 17 00:00:00 2001 From: Raymond Tukpe Date: Mon, 27 Jun 2022 15:25:06 +0100 Subject: [PATCH 3/7] Refactor DB Layer (#780) * feat: add db store file * feat: refactor some groyp repo methods * Refactor event repo, group repo tests * group and event tests * fix IsValidPointer, refactor organization repos (#794) * feat: refactor config repo to use store (#798) * feat: refactor user repo to use store (#797) * refactor subscriptions repo (#795) * feat: refactor subscriptions repo * tests: fix test * tests: fix test * refactor applications repo (#799) * feat: refactor app repo * fix: fix call * fix: remove ==== * tests: fix test * refactor source repo (#796) * feat: refactor source repo to use store * fix: update find one pointer check * chore: fix lint error * fix: clean up db.go Co-authored-by: ogbanugot Co-authored-by: Dotun Jolaoso Co-authored-by: Daniel Oluojomu --- datastore/badger/subscription.go | 4 +- datastore/db.go | 326 ++++++++++++++++++ datastore/models.go | 5 + datastore/mongo/application.go | 65 +--- datastore/mongo/application_test.go | 14 +- datastore/mongo/configuration.go | 23 +- datastore/mongo/configuration_test.go | 9 +- datastore/mongo/event.go | 36 +- datastore/mongo/group.go | 86 +---- datastore/mongo/group_test.go | 20 +- datastore/mongo/mongo.go | 30 +- datastore/mongo/mongo_test.go | 9 +- datastore/mongo/organisation.go | 24 +- datastore/mongo/organisation_invite.go | 29 +- datastore/mongo/organisation_invite_test.go | 28 +- datastore/mongo/organisation_member.go | 48 +-- datastore/mongo/organisation_member_test.go | 44 ++- datastore/mongo/organisation_test.go | 16 +- datastore/mongo/source.go | 42 +-- datastore/mongo/source_test.go | 18 +- datastore/mongo/subscription.go | 102 ++---- datastore/mongo/subscription_test.go | 10 +- datastore/mongo/user.go | 39 +-- datastore/mongo/user_test.go | 15 +- datastore/repository.go | 4 +- docs/docs.go | 3 +- mocks/repository.go | 52 +-- server/middleware.go | 18 - server/organisation_integration_test.go | 9 +- .../organisation_member_integration_test.go | 9 +- worker/task/process_event_creation.go | 2 +- worker/task/process_event_creation_test.go | 2 +- 32 files changed, 689 insertions(+), 452 deletions(-) create mode 100644 datastore/db.go diff --git a/datastore/badger/subscription.go b/datastore/badger/subscription.go index 8c29a96f63..1992791d6c 100644 --- a/datastore/badger/subscription.go +++ b/datastore/badger/subscription.go @@ -19,11 +19,11 @@ func (*subscriptionRepo) UpdateSubscriptionStatus(context.Context, string, strin return nil } -func (*subscriptionRepo) FindSubscriptionBySourceIDs(context.Context, string, string) ([]datastore.Subscription, error) { +func (*subscriptionRepo) FindSubscriptionsBySourceIDs(context.Context, string, string) ([]datastore.Subscription, error) { return nil, nil } -func (*subscriptionRepo) FindSubscriptionByEventType(context.Context, string, string, datastore.EventType) ([]datastore.Subscription, error) { +func (*subscriptionRepo) FindSubscriptionsByEventType(context.Context, string, string, datastore.EventType) ([]datastore.Subscription, error) { return nil, nil } diff --git a/datastore/db.go b/datastore/db.go new file mode 100644 index 0000000000..57fbcf617e --- /dev/null +++ b/datastore/db.go @@ -0,0 +1,326 @@ +package datastore + +import ( + "context" + "errors" + "reflect" + "time" + + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" + "go.mongodb.org/mongo-driver/mongo" + "go.mongodb.org/mongo-driver/mongo/options" +) + +type mongoStore struct { + IsConnected bool + CollectionName string + Collection *mongo.Collection + Database *mongo.Database +} + +type Store interface { + Save(ctx context.Context, payload interface{}, result interface{}) error + SaveMany(ctx context.Context, payload []interface{}) error + + FindByID(ctx context.Context, id string, projection bson.M, result interface{}) error + FindOne(ctx context.Context, filter, projection bson.M, result interface{}) error + FindMany(ctx context.Context, filter, projection bson.M, sort interface{}, limit, skip int64, results interface{}) error + FindManyWithDeletedAt(ctx context.Context, filter, projection bson.M, sort interface{}, limit, skip int64, results interface{}) error + FindAll(ctx context.Context, filter bson.M, sort interface{}, projection, results interface{}) error + + UpdateByID(ctx context.Context, id string, payload interface{}) error + UpdateOne(ctx context.Context, filter bson.M, payload interface{}) error + UpdateMany(ctx context.Context, filter, payload bson.M) error + + Inc(ctx context.Context, filter bson.M, payload interface{}) error + + DeleteByID(ctx context.Context, id string) error + DeleteOne(ctx context.Context, filter bson.M) error + + Count(ctx context.Context, filter map[string]interface{}) (int64, error) + + Aggregate(ctx context.Context, pipeline mongo.Pipeline, result interface{}, allowDiskUse bool) error +} + +// mongodb driver -> store (database) -> repo -> service -> handler + +var _ Store = &mongoStore{} + +/* + * New + * This initialises a new MongoDB repo for the collection + */ +func New(database *mongo.Database, collection string) Store { + mongoStore := &mongoStore{ + IsConnected: true, + CollectionName: collection, + Collection: database.Collection(collection), + Database: database, + } + + return mongoStore +} + +var ( + ErrInvalidPtr = errors.New("out param is not a valid pointer") +) + +func IsValidPointer(i interface{}) bool { + v := reflect.ValueOf(i) + return v.Type().Kind() == reflect.Ptr && !v.IsNil() +} + +/** + * Save + * Save is used to save a record in the mongoStore + */ +func (d *mongoStore) Save(ctx context.Context, payload interface{}, out interface{}) error { + result, err := d.Collection.InsertOne(ctx, payload) + + if err != nil { + return err + } + + if out == nil { + return nil + } + + if !IsValidPointer(out) { + return ErrInvalidPtr + } + + return d.Collection.FindOne(ctx, bson.M{"_id": result.InsertedID}).Decode(out) +} + +/** + * SaveMany + * SaveMany is used to bulk insert into the mongoStore + * + * param: []interface{} payload + * return: error + */ +func (d *mongoStore) SaveMany(ctx context.Context, payload []interface{}) error { + _, err := d.Collection.InsertMany(ctx, payload) + return err +} + +/** + * FindByID + * FindByID finds a single record by id in the mongoStore + * returns nil if record is not found. + * + * param: interface{} id + * param: bson.M projection + * return: bson.M + */ +func (d *mongoStore) FindByID(ctx context.Context, id string, projection bson.M, result interface{}) error { + if !IsValidPointer(result) { + return ErrInvalidPtr + } + + ops := options.FindOne() + if projection != nil { + ops.Projection = projection + } + + return d.Collection.FindOne(ctx, bson.M{"uid": id, "document_status": ActiveDocumentStatus}, ops).Decode(result) +} + +/** + * Find One by + */ +func (d *mongoStore) FindOne(ctx context.Context, filter, projection bson.M, result interface{}) error { + if !IsValidPointer(result) { + return ErrInvalidPtr + } + + ops := options.FindOne() + ops.Projection = projection + + filter["document_status"] = ActiveDocumentStatus + + return d.Collection.FindOne(ctx, filter, ops).Decode(result) +} + +func (d *mongoStore) FindMany(ctx context.Context, filter, projection bson.M, sort interface{}, limit, skip int64, results interface{}) error { + if !IsValidPointer(results) { + return ErrInvalidPtr + } + + ops := options.Find() + if limit > 0 { + ops.Limit = &limit + } + if skip > 0 { + ops.Skip = &skip + } + if projection != nil { + ops.Projection = projection + } + if sort != nil { + ops.Sort = sort + } + + filter["document_status"] = ActiveDocumentStatus + + cursor, err := d.Collection.Find(ctx, filter, ops) + if err != nil { + return err + } + + return cursor.All(ctx, results) +} + +func (d *mongoStore) FindManyWithDeletedAt(ctx context.Context, filter, projection bson.M, sort interface{}, limit, skip int64, results interface{}) error { + if !IsValidPointer(results) { + return ErrInvalidPtr + } + + ops := options.Find() + if limit > 0 { + ops.Limit = &limit + } + if skip > 0 { + ops.Skip = &skip + } + if projection != nil { + ops.Projection = projection + } + if sort != nil { + ops.Sort = sort + } + + cursor, err := d.Collection.Find(ctx, filter, ops) + if err != nil { + return err + } + + return cursor.All(ctx, results) +} + +func (d *mongoStore) FindAll(ctx context.Context, filter bson.M, sort interface{}, projection, results interface{}) error { + if !IsValidPointer(results) { + return ErrInvalidPtr + } + + ops := options.Find() + + if projection != nil { + ops.Projection = projection + } + + if sort != nil { + ops.Sort = sort + } + + if filter == nil { + filter = bson.M{} + } + + filter["document_status"] = ActiveDocumentStatus + + cursor, err := d.Collection.Find(ctx, filter, ops) + if err != nil { + return err + } + + return cursor.All(ctx, results) +} + +/** + * UpdateByID + * Updates a single record by id in the mongoStore + * + * param: interface{} id + * param: interface{} payload + * return: error + */ +func (d *mongoStore) UpdateByID(ctx context.Context, id string, payload interface{}) error { + _, err := d.Collection.UpdateOne(ctx, bson.M{"uid": id}, bson.M{"$set": payload}, nil) + return err +} + +func (d *mongoStore) UpdateOne(ctx context.Context, filter bson.M, payload interface{}) error { + _, err := d.Collection.UpdateOne(ctx, filter, bson.M{"$set": payload}) + return err +} + +func (d *mongoStore) Inc(ctx context.Context, filter bson.M, payload interface{}) error { + _, err := d.Collection.UpdateOne(ctx, filter, bson.M{"$inc": payload}) + return err +} + +/** + * UpdateMany + * Updates many items in the collection + * `filter` this is the search criteria + * `payload` this is the update payload. + * + * param: bson.M filter + * param: interface{} payload + * return: error + */ +func (d *mongoStore) UpdateMany(ctx context.Context, filter, payload bson.M) error { + _, err := d.Collection.UpdateMany(ctx, filter, bson.M{"$set": payload}) + return err +} + +/** + * DeleteByID + * Deletes a single record by id + * where ID can be a string or whatever. + * param: interface{} id + * return: error + * The record is not completed deleted, only the status is changed. + */ +func (d *mongoStore) DeleteByID(ctx context.Context, id string) error { + payload := bson.M{ + "deleted_at": primitive.NewDateTimeFromTime(time.Now()), + "document_status": DeletedDocumentStatus, + } + + _, err := d.Collection.UpdateOne(ctx, bson.M{"uid": id}, bson.M{"$set": payload}, nil) + return err +} + +/** + * DeleteOne + * Deletes one item from the mongoStore using filter a hash map to properly filter what is to be deleted. + * + * param: bson.M filter + * return: error + * The record is not completed deleted, only the status is changed. + */ +func (d *mongoStore) DeleteOne(ctx context.Context, filter bson.M) error { + payload := bson.M{ + "deleted_at": primitive.NewDateTimeFromTime(time.Now()), + "document_status": DeletedDocumentStatus, + } + + _, err := d.Collection.UpdateOne(ctx, filter, bson.M{"$set": payload}) + return err +} + +func (d *mongoStore) Count(ctx context.Context, filter map[string]interface{}) (int64, error) { + filter["document_status"] = ActiveDocumentStatus + return d.Collection.CountDocuments(ctx, filter) +} + +func (d *mongoStore) Aggregate(ctx context.Context, pipeline mongo.Pipeline, output interface{}, allowDiskUse bool) error { + if !IsValidPointer(output) { + return ErrInvalidPtr + } + + opts := options.Aggregate() + if allowDiskUse { + opts.SetAllowDiskUse(true) + } + + cur, err := d.Collection.Aggregate(ctx, pipeline, opts) + if err != nil { + return err + } + + return cur.All(ctx, output) +} diff --git a/datastore/models.go b/datastore/models.go index 1a25702124..2fa312bd8c 100644 --- a/datastore/models.go +++ b/datastore/models.go @@ -259,6 +259,11 @@ func (g *GroupFilter) WithNamesTrimmed() *GroupFilter { return &f } +func (g *GroupFilter) ToGenericMap() map[string]interface{} { + m := map[string]interface{}{"name": g.Names} + return m +} + func (o *Group) IsDeleted() bool { return o.DeletedAt > 0 } func (o *Group) IsOwner(a *Application) bool { return o.UID == a.GroupID } diff --git a/datastore/mongo/application.go b/datastore/mongo/application.go index 9a90fc3989..7d3f22bcee 100644 --- a/datastore/mongo/application.go +++ b/datastore/mongo/application.go @@ -19,12 +19,14 @@ import ( type appRepo struct { innerDB *mongo.Database client *mongo.Collection + store datastore.Store } -func NewApplicationRepo(db *mongo.Database) datastore.ApplicationRepository { +func NewApplicationRepo(db *mongo.Database, store datastore.Store) datastore.ApplicationRepository { return &appRepo{ innerDB: db, client: db.Collection(AppCollection, nil), + store: store, } } @@ -39,7 +41,7 @@ func (db *appRepo) CreateApplication(ctx context.Context, app *datastore.Applica } app.ID = primitive.NewObjectID() - _, err = db.client.InsertOne(ctx, app) + err = db.store.Save(ctx, app, nil) return err } @@ -161,28 +163,11 @@ func (db *appRepo) SearchApplicationsByGroupId(ctx context.Context, groupId stri } apps := make([]datastore.Application, 0) - cur, err := db.client.Find(ctx, filter) + err := db.store.FindMany(ctx, filter, nil, nil, 0, 0, &apps) if err != nil { return apps, err } - for cur.Next(ctx) { - var app datastore.Application - if err := cur.Decode(&app); err != nil { - return apps, err - } - - apps = append(apps, app) - } - - if err := cur.Err(); err != nil { - return nil, err - } - - if err := cur.Close(ctx); err != nil { - return apps, err - } - msgCollection := db.innerDB.Collection(EventCollection) for i, app := range apps { filter = bson.M{"app_id": app.UID, "document_status": datastore.ActiveDocumentStatus} @@ -202,17 +187,14 @@ func (db *appRepo) FindApplicationByID(ctx context.Context, app := new(datastore.Application) - filter := bson.M{"uid": id, "document_status": datastore.ActiveDocumentStatus} - - err := db.client.FindOne(ctx, filter). - Decode(&app) + err := db.store.FindByID(ctx, id, nil, app) if errors.Is(err, mongo.ErrNoDocuments) { err = datastore.ErrApplicationNotFound return app, err } msgCollection := db.innerDB.Collection(EventCollection) - filter = bson.M{"app_id": app.UID, "document_status": datastore.ActiveDocumentStatus} + filter := bson.M{"app_id": app.UID, "document_status": datastore.ActiveDocumentStatus} count, err := msgCollection.CountDocuments(ctx, filter) if err != nil { log.WithError(err).Errorf("failed to count events in %s", app.UID) @@ -254,18 +236,15 @@ func (db *appRepo) UpdateApplication(ctx context.Context, app *datastore.Applica app.UpdatedAt = primitive.NewDateTimeFromTime(time.Now()) - filter := bson.M{"uid": app.UID, "document_status": datastore.ActiveDocumentStatus} - - update := bson.D{primitive.E{Key: "$set", Value: bson.D{ - primitive.E{Key: "endpoints", Value: app.Endpoints}, - primitive.E{Key: "updated_at", Value: app.UpdatedAt}, - primitive.E{Key: "title", Value: app.Title}, - primitive.E{Key: "support_email", Value: app.SupportEmail}, - primitive.E{Key: "is_disabled", Value: app.IsDisabled}, - }}} + update := bson.M{ + "endpoints": app.Endpoints, + "updated_at": app.UpdatedAt, + "title": app.Title, + "support_email": app.SupportEmail, + "is_disabled": app.IsDisabled, + } - _, err = db.client.UpdateOne(ctx, filter, update) - return err + return db.store.UpdateByID(ctx, app.UID, update) } func (db *appRepo) CreateApplicationEndpoint(ctx context.Context, groupID string, appID string, endpoint *datastore.Endpoint) error { @@ -284,20 +263,12 @@ func (db *appRepo) CreateApplicationEndpoint(ctx context.Context, groupID string } func (db *appRepo) DeleteGroupApps(ctx context.Context, groupID string) error { - update := bson.M{ - "$set": bson.M{ - "deleted_at": primitive.NewDateTimeFromTime(time.Now()), - "document_status": datastore.DeletedDocumentStatus, - }, - } - - _, err := db.client.UpdateMany(ctx, bson.M{"group_id": groupID}, update) - if err != nil { - return err + "deleted_at": primitive.NewDateTimeFromTime(time.Now()), + "document_status": datastore.DeletedDocumentStatus, } - return nil + return db.store.UpdateMany(ctx, bson.M{"group_id": groupID}, update) } func (db *appRepo) DeleteApplication(ctx context.Context, app *datastore.Application) error { diff --git a/datastore/mongo/application_test.go b/datastore/mongo/application_test.go index 5961f4ba3e..c3099b4b30 100644 --- a/datastore/mongo/application_test.go +++ b/datastore/mongo/application_test.go @@ -17,8 +17,8 @@ func Test_UpdateApplication(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - groupRepo := NewGroupRepo(db) - appRepo := NewApplicationRepo(db) + groupRepo := NewGroupRepo(db, getStore(db, GroupCollection)) + appRepo := NewApplicationRepo(db, getStore(db, AppCollection)) newGroup := &datastore.Group{ Name: "Random new group", @@ -61,8 +61,8 @@ func Test_CreateApplication(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - groupRepo := NewGroupRepo(db) - appRepo := NewApplicationRepo(db) + groupRepo := NewGroupRepo(db, getStore(db, GroupCollection)) + appRepo := NewApplicationRepo(db, getStore(db, AppCollection)) newOrg := &datastore.Group{ Name: "Random new group 2", @@ -95,7 +95,7 @@ func Test_LoadApplicationsPaged(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - appRepo := NewApplicationRepo(db) + appRepo := NewApplicationRepo(db, getStore(db, AppCollection)) apps, _, err := appRepo.LoadApplicationsPaged(context.Background(), "", "", datastore.Pageable{ Page: 1, @@ -110,14 +110,14 @@ func Test_FindApplicationByID(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - appRepo := NewApplicationRepo(db) + appRepo := NewApplicationRepo(db, getStore(db, AppCollection)) _, err := appRepo.FindApplicationByID(context.Background(), uuid.New().String()) require.Error(t, err) require.True(t, errors.Is(err, datastore.ErrApplicationNotFound)) - groupRepo := NewGroupRepo(db) + groupRepo := NewGroupRepo(db, getStore(db, GroupCollection)) newGroup := &datastore.Group{ Name: "Yet another Random new group", diff --git a/datastore/mongo/configuration.go b/datastore/mongo/configuration.go index 7eb524f636..77611d1d19 100644 --- a/datastore/mongo/configuration.go +++ b/datastore/mongo/configuration.go @@ -14,27 +14,30 @@ import ( type configRepo struct { innerDB *mongo.Database client *mongo.Collection + store datastore.Store } -func NewConfigRepo(db *mongo.Database) datastore.ConfigurationRepository { +func NewConfigRepo(db *mongo.Database, store datastore.Store) datastore.ConfigurationRepository { return &configRepo{ innerDB: db, - client: db.Collection(ConfigCollection)} + client: db.Collection(ConfigCollection), + store: store, + } } func (c *configRepo) CreateConfiguration(ctx context.Context, config *datastore.Configuration) error { config.ID = primitive.NewObjectID() - _, err := c.client.InsertOne(ctx, config) + err := c.store.Save(ctx, config, nil) return err } func (c *configRepo) LoadConfiguration(ctx context.Context) (*datastore.Configuration, error) { config := &datastore.Configuration{} - filter := bson.M{"document_status": datastore.ActiveDocumentStatus} + filter := bson.M{} - err := c.client.FindOne(ctx, filter).Decode(&config) + err := c.store.FindOne(ctx, filter, nil, config) if errors.Is(err, mongo.ErrNoDocuments) { return nil, datastore.ErrConfigNotFound @@ -44,15 +47,13 @@ func (c *configRepo) LoadConfiguration(ctx context.Context) (*datastore.Configur } func (c *configRepo) UpdateConfiguration(ctx context.Context, config *datastore.Configuration) error { - filter := bson.M{"uid": config.UID, "document_status": datastore.ActiveDocumentStatus} + filter := bson.M{"uid": config.UID} update := bson.D{ - primitive.E{Key: "$set", Value: bson.D{ - primitive.E{Key: "is_analytics_enabled", Value: config.IsAnalyticsEnabled}, - primitive.E{Key: "updated_at", Value: primitive.NewDateTimeFromTime(time.Now())}, - }}, + primitive.E{Key: "is_analytics_enabled", Value: config.IsAnalyticsEnabled}, + primitive.E{Key: "updated_at", Value: primitive.NewDateTimeFromTime(time.Now())}, } - _, err := c.client.UpdateOne(ctx, filter, update) + err := c.store.UpdateOne(ctx, filter, update) return err } diff --git a/datastore/mongo/configuration_test.go b/datastore/mongo/configuration_test.go index 50d03bbb2f..a2a1c0ff2b 100644 --- a/datastore/mongo/configuration_test.go +++ b/datastore/mongo/configuration_test.go @@ -17,7 +17,8 @@ func Test_CreateConfiguration(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - configRepo := NewConfigRepo(db) + store := getStore(db, ConfigCollection) + configRepo := NewConfigRepo(db, store) config := generateConfig() require.NoError(t, configRepo.CreateConfiguration(context.Background(), config)) @@ -33,7 +34,8 @@ func Test_LoadConfiguration(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - configRepo := NewConfigRepo(db) + store := getStore(db, ConfigCollection) + configRepo := NewConfigRepo(db, store) config := generateConfig() _, err := configRepo.LoadConfiguration(context.Background()) @@ -53,7 +55,8 @@ func Test_UpdateConfiguration(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - configRepo := NewConfigRepo(db) + store := getStore(db, ConfigCollection) + configRepo := NewConfigRepo(db, store) config := generateConfig() require.NoError(t, configRepo.CreateConfiguration(context.Background(), config)) diff --git a/datastore/mongo/event.go b/datastore/mongo/event.go index dde21d3989..98a05763e3 100644 --- a/datastore/mongo/event.go +++ b/datastore/mongo/event.go @@ -17,11 +17,13 @@ import ( type eventRepo struct { inner *mongo.Collection + store datastore.Store } -func NewEventRepository(db *mongo.Database) datastore.EventRepository { +func NewEventRepository(db *mongo.Database, store datastore.Store) datastore.EventRepository { return &eventRepo{ inner: db.Collection(EventCollection), + store: store, } } @@ -41,7 +43,7 @@ func (db *eventRepo) CreateEvent(ctx context.Context, message *datastore.Event) message.UID = uuid.New().String() } - _, err := db.inner.InsertOne(ctx, message) + err := db.store.Save(ctx, message, nil) return err } @@ -51,7 +53,7 @@ func (db *eventRepo) CountGroupMessages(ctx context.Context, groupID string) (in "document_status": datastore.ActiveDocumentStatus, } - count, err := db.inner.CountDocuments(ctx, filter) + count, err := db.store.Count(ctx, filter) if err != nil { log.WithError(err).Errorf("failed to count events in group %s", groupID) return 0, err @@ -61,14 +63,12 @@ func (db *eventRepo) CountGroupMessages(ctx context.Context, groupID string) (in func (db *eventRepo) DeleteGroupEvents(ctx context.Context, groupID string) error { update := bson.M{ - "$set": bson.M{ - "deleted_at": primitive.NewDateTimeFromTime(time.Now()), - "document_status": datastore.ActiveDocumentStatus, - }, + "deleted_at": primitive.NewDateTimeFromTime(time.Now()), + "document_status": datastore.ActiveDocumentStatus, } filter := bson.M{"group_id": groupID} - _, err := db.inner.UpdateMany(ctx, filter, update) + err := db.store.UpdateMany(ctx, filter, update) if err != nil { return err } @@ -130,17 +130,13 @@ func (db *eventRepo) LoadEventIntervals(ctx context.Context, groupID string, sea }, } sortStage := bson.D{{Key: "$sort", Value: bson.D{primitive.E{Key: "_id", Value: 1}}}} + var eventsIntervals []datastore.EventInterval - data, err := db.inner.Aggregate(ctx, mongo.Pipeline{matchStage, groupStage, sortStage}) + err := db.store.Aggregate(ctx, mongo.Pipeline{matchStage, groupStage, sortStage}, &eventsIntervals, false) if err != nil { log.WithError(err).Errorln("aggregate error") return nil, err } - var eventsIntervals []datastore.EventInterval - if err = data.All(ctx, &eventsIntervals); err != nil { - log.WithError(err).Error("marshal error") - return nil, err - } if eventsIntervals == nil { eventsIntervals = make([]datastore.EventInterval, 0) } @@ -151,10 +147,8 @@ func (db *eventRepo) LoadEventIntervals(ctx context.Context, groupID string, sea func (db *eventRepo) FindEventByID(ctx context.Context, id string) (*datastore.Event, error) { m := new(datastore.Event) - filter := bson.M{"uid": id, "document_status": datastore.ActiveDocumentStatus} + err := db.store.FindByID(ctx, id, nil, m) - err := db.inner.FindOne(ctx, filter). - Decode(&m) if errors.Is(err, mongo.ErrNoDocuments) { err = datastore.ErrEventNotFound } @@ -167,16 +161,10 @@ func (db *eventRepo) FindEventsByIDs(ctx context.Context, ids []string) ([]datas filter := bson.M{"uid": bson.M{"$in": ids}, "document_status": datastore.ActiveDocumentStatus} - cursor, err := db.inner.Find(ctx, filter) + err := db.store.FindMany(ctx, filter, nil, nil, 0, 0, &m) if err != nil { return nil, err } - - err = cursor.All(ctx, &m) - if err != nil { - return nil, err - } - return m, err } diff --git a/datastore/mongo/group.go b/datastore/mongo/group.go index e6906d4757..7cf7f0275e 100644 --- a/datastore/mongo/group.go +++ b/datastore/mongo/group.go @@ -12,29 +12,30 @@ import ( "go.mongodb.org/mongo-driver/bson" "go.mongodb.org/mongo-driver/bson/primitive" "go.mongodb.org/mongo-driver/mongo" - "go.mongodb.org/mongo-driver/mongo/options" ) type groupRepo struct { innerDB *mongo.Database inner *mongo.Collection + store datastore.Store } func isDuplicateNameIndex(err error) bool { return strings.Contains(err.Error(), "name") } -func NewGroupRepo(db *mongo.Database) datastore.GroupRepository { +func NewGroupRepo(db *mongo.Database, store datastore.Store) datastore.GroupRepository { return &groupRepo{ innerDB: db, inner: db.Collection(GroupCollection), + store: store, } } func (db *groupRepo) LoadGroups(ctx context.Context, f *datastore.GroupFilter) ([]*datastore.Group, error) { groups := make([]*datastore.Group, 0) - opts := &options.FindOptions{Collation: &options.Collation{Locale: "en", Strength: 2}} + // opts := &options.FindOptions{Collation: &options.Collation{Locale: "en", Strength: 2}} filter := bson.M{ "document_status": datastore.ActiveDocumentStatus, "organisation_id": f.OrgID, @@ -45,59 +46,36 @@ func (db *groupRepo) LoadGroups(ctx context.Context, f *datastore.GroupFilter) ( filter["name"] = bson.M{"$in": f.Names} } - cur, err := db.inner.Find(ctx, filter, opts) - if err != nil { - return groups, err - } - - for cur.Next(ctx) { - var group = new(datastore.Group) - if err := cur.Decode(&group); err != nil { - return groups, err - } - - groups = append(groups, group) - } + sort := bson.M{"created_at": 1} + err := db.store.FindAll(ctx, filter, sort, nil, &groups) - if err := cur.Err(); err != nil { - return nil, err - } - - if err := cur.Close(ctx); err != nil { - return groups, err - } - - return groups, nil + return groups, err } func (db *groupRepo) CreateGroup(ctx context.Context, o *datastore.Group) error { - o.ID = primitive.NewObjectID() - _, err := db.inner.InsertOne(ctx, o) + err := db.store.Save(ctx, o, nil) // check if the error string contains the index called "name" if mongo.IsDuplicateKeyError(err) && isDuplicateNameIndex(err) { return datastore.ErrDuplicateGroupName } + return err } func (db *groupRepo) UpdateGroup(ctx context.Context, o *datastore.Group) error { o.UpdatedAt = primitive.NewDateTimeFromTime(time.Now()) - - filter := bson.D{primitive.E{Key: "uid", Value: o.UID}} - - update := bson.D{primitive.E{Key: "$set", Value: bson.D{ - primitive.E{Key: "name", Value: o.Name}, + update := bson.D{primitive.E{Key: "name", Value: o.Name}, primitive.E{Key: "logo_url", Value: o.LogoURL}, primitive.E{Key: "updated_at", Value: o.UpdatedAt}, primitive.E{Key: "config", Value: o.Config}, primitive.E{Key: "rate_limit", Value: o.RateLimit}, primitive.E{Key: "rate_limit_duration", Value: o.RateLimitDuration}, - }}} + } - _, err := db.inner.UpdateOne(ctx, filter, update) + err := db.store.UpdateByID(ctx, o.UID, update) if mongo.IsDuplicateKeyError(err) && isDuplicateNameIndex(err) { return datastore.ErrDuplicateGroupName } @@ -108,12 +86,7 @@ func (db *groupRepo) UpdateGroup(ctx context.Context, o *datastore.Group) error func (db *groupRepo) FetchGroupByID(ctx context.Context, id string) (*datastore.Group, error) { group := new(datastore.Group) - filter := bson.M{ - "uid": id, - "document_status": datastore.ActiveDocumentStatus, - } - - err := db.inner.FindOne(ctx, filter).Decode(group) + err := db.store.FindByID(ctx, id, nil, group) if errors.Is(err, mongo.ErrNoDocuments) { err = datastore.ErrGroupNotFound } @@ -162,19 +135,14 @@ func (db *groupRepo) FillGroupsStatistics(ctx context.Context, groups []*datasto {Key: "messages_sent", Value: bson.D{{Key: "$size", Value: "$group_events"}}}, }}, } + var stats []datastore.GroupStatistics - data, err := db.inner.Aggregate(ctx, mongo.Pipeline{matchStage, lookupStage1, lookupStage2, projectStage}) + err := db.store.Aggregate(ctx, mongo.Pipeline{matchStage, lookupStage1, lookupStage2, projectStage}, &stats, false) if err != nil { log.WithError(err).Error("failed to run group statistics aggregation") return err } - var stats []datastore.GroupStatistics - if err = data.All(ctx, &stats); err != nil { - log.WithError(err).Error("failed to marshal group statistics") - return err - } - statsMap := map[string]*datastore.GroupStatistics{} for i, s := range stats { statsMap[s.GroupID] = &stats[i] @@ -188,14 +156,7 @@ func (db *groupRepo) FillGroupsStatistics(ctx context.Context, groups []*datasto } func (db *groupRepo) DeleteGroup(ctx context.Context, uid string) error { - update := bson.M{ - "$set": bson.M{ - "deleted_at": primitive.NewDateTimeFromTime(time.Now()), - "document_status": datastore.DeletedDocumentStatus, - }, - } - - _, err := db.inner.UpdateOne(ctx, bson.M{"uid": uid}, update) + err := db.store.DeleteByID(ctx, uid) if err != nil { return err } @@ -212,19 +173,10 @@ func (db *groupRepo) FetchGroupsByIDs(ctx context.Context, ids []string) ([]data } groups := make([]datastore.Group, 0) - - cur, err := db.inner.Find(ctx, filter, nil) + sort := bson.M{"created_at": 1} + err := db.store.FindAll(ctx, filter, sort, nil, &groups) if err != nil { - return groups, err - } - - for cur.Next(ctx) { - var group datastore.Group - if err := cur.Decode(&group); err != nil { - return groups, err - } - - groups = append(groups, group) + return nil, err } return groups, err diff --git a/datastore/mongo/group_test.go b/datastore/mongo/group_test.go index cd97f8f582..28773ebcc7 100644 --- a/datastore/mongo/group_test.go +++ b/datastore/mongo/group_test.go @@ -16,7 +16,9 @@ func Test_FetchGroupByID(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - groupRepo := NewGroupRepo(db) + store := getStore(db, GroupCollection) + + groupRepo := NewGroupRepo(db, store) newOrg := &datastore.Group{ Name: "Yet another group", @@ -37,6 +39,8 @@ func Test_CreateGroup(t *testing.T) { db, closeFn := getDB(t) defer closeFn() + store := getStore(db, GroupCollection) + tt := []struct { name string groups []datastore.Group @@ -114,7 +118,7 @@ func Test_CreateGroup(t *testing.T) { for _, tc := range tt { t.Run(tc.name, func(t *testing.T) { - groupRepo := NewGroupRepo(db) + groupRepo := NewGroupRepo(db, store) for i, group := range tc.groups { newGroup := &datastore.Group{ @@ -145,8 +149,9 @@ func Test_CreateGroup(t *testing.T) { func Test_LoadGroups(t *testing.T) { db, closeFn := getDB(t) defer closeFn() + store := getStore(db, GroupCollection) - orgRepo := NewGroupRepo(db) + orgRepo := NewGroupRepo(db, store) orgs, err := orgRepo.LoadGroups(context.Background(), &datastore.GroupFilter{}) require.NoError(t, err) @@ -158,7 +163,10 @@ func Test_FillGroupsStatistics(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - groupRepo := NewGroupRepo(db) + groupStore := getStore(db, GroupCollection) + eventStore := getStore(db, EventCollection) + + groupRepo := NewGroupRepo(db, groupStore) group1 := &datastore.Group{ Name: "group1", @@ -186,7 +194,7 @@ func Test_FillGroupsStatistics(t *testing.T) { GroupID: group2.UID, } - appRepo := NewApplicationRepo(db) + appRepo := NewApplicationRepo(db, getStore(db, AppCollection)) err = appRepo.CreateApplication(context.Background(), app1, group1.UID) require.NoError(t, err) @@ -199,7 +207,7 @@ func Test_FillGroupsStatistics(t *testing.T) { AppID: app1.UID, } - err = NewEventRepository(db).CreateEvent(context.Background(), event) + err = NewEventRepository(db, eventStore).CreateEvent(context.Background(), event) require.NoError(t, err) groups := []*datastore.Group{group1, group2} diff --git a/datastore/mongo/mongo.go b/datastore/mongo/mongo.go index cca9282bbc..1b6713552f 100644 --- a/datastore/mongo/mongo.go +++ b/datastore/mongo/mongo.go @@ -72,21 +72,31 @@ func New(cfg config.Configuration) (datastore.DatabaseClient, error) { dbName := strings.TrimPrefix(u.Path, "/") conn := client.Database(dbName, nil) + groups := datastore.New(conn, GroupCollection) + events := datastore.New(conn, EventCollection) + sources := datastore.New(conn, SourceCollection) + apps := datastore.New(conn, AppCollection) + subscriptions := datastore.New(conn, SubscriptionCollection) + orgs := datastore.New(conn, OrganisationCollection) + org_member := datastore.New(conn, OrganisationMembersCollection) + org_invite := datastore.New(conn, OrganisationInvitesCollection) + users := datastore.New(conn, UserCollection) + config := datastore.New(conn, ConfigCollection) c := &Client{ db: conn, apiKeyRepo: NewApiKeyRepo(conn), - groupRepo: NewGroupRepo(conn), - subscriptionRepo: NewSubscriptionRepo(conn), - applicationRepo: NewApplicationRepo(conn), - eventRepo: NewEventRepository(conn), + groupRepo: NewGroupRepo(conn, groups), + applicationRepo: NewApplicationRepo(conn, apps), + subscriptionRepo: NewSubscriptionRepo(conn, subscriptions), + eventRepo: NewEventRepository(conn, events), eventDeliveryRepo: NewEventDeliveryRepository(conn), - sourceRepo: NewSourceRepo(conn), - orgRepo: NewOrgRepo(conn), - orgMemberRepo: NewOrgMemberRepo(conn), - orgInviteRepo: NewOrgInviteRepo(conn), - userRepo: NewUserRepo(conn), - configRepo: NewConfigRepo(conn), + sourceRepo: NewSourceRepo(conn, sources), + orgRepo: NewOrgRepo(conn, orgs), + orgMemberRepo: NewOrgMemberRepo(conn, org_member), + orgInviteRepo: NewOrgInviteRepo(conn, org_invite), + userRepo: NewUserRepo(conn, users), + configRepo: NewConfigRepo(conn, config), } c.ensureMongoIndices() diff --git a/datastore/mongo/mongo_test.go b/datastore/mongo/mongo_test.go index 8852b5aec5..90c79d983f 100644 --- a/datastore/mongo/mongo_test.go +++ b/datastore/mongo/mongo_test.go @@ -5,10 +5,11 @@ package mongo import ( "context" - "os" + "os" "testing" "github.com/frain-dev/convoy/config" + "github.com/frain-dev/convoy/datastore" "github.com/stretchr/testify/require" "go.mongodb.org/mongo-driver/mongo" ) @@ -39,3 +40,9 @@ func getDB(t *testing.T) (*mongo.Database, func()) { require.NoError(t, db.Disconnect(context.Background())) } } + +func getStore(db *mongo.Database, collection string) datastore.Store { + + store := datastore.New(db, collection) + return store +} diff --git a/datastore/mongo/organisation.go b/datastore/mongo/organisation.go index 92812946fa..471ccb506b 100644 --- a/datastore/mongo/organisation.go +++ b/datastore/mongo/organisation.go @@ -15,12 +15,14 @@ import ( type orgRepo struct { innerDB *mongo.Database inner *mongo.Collection + store datastore.Store } -func NewOrgRepo(db *mongo.Database) datastore.OrganisationRepository { +func NewOrgRepo(db *mongo.Database, store datastore.Store) datastore.OrganisationRepository { return &orgRepo{ innerDB: db, inner: db.Collection(OrganisationCollection), + store: store, } } @@ -38,30 +40,28 @@ func (db *orgRepo) LoadOrganisationsPaged(ctx context.Context, pageable datastor func (db *orgRepo) CreateOrganisation(ctx context.Context, org *datastore.Organisation) error { org.ID = primitive.NewObjectID() - _, err := db.inner.InsertOne(ctx, org) + err := db.store.Save(ctx, org, nil) return err } func (db *orgRepo) UpdateOrganisation(ctx context.Context, org *datastore.Organisation) error { org.UpdatedAt = primitive.NewDateTimeFromTime(time.Now()) - update := bson.D{primitive.E{Key: "$set", Value: bson.D{ + update := bson.D{ primitive.E{Key: "name", Value: org.Name}, primitive.E{Key: "updated_at", Value: org.UpdatedAt}, - }}} + } - _, err := db.inner.UpdateOne(ctx, bson.M{"uid": org.UID}, update) + err := db.store.UpdateOne(ctx, bson.M{"uid": org.UID}, update) return err } func (db *orgRepo) DeleteOrganisation(ctx context.Context, uid string) error { update := bson.M{ - "$set": bson.M{ - "deleted_at": primitive.NewDateTimeFromTime(time.Now()), - "document_status": datastore.DeletedDocumentStatus, - }, + "deleted_at": primitive.NewDateTimeFromTime(time.Now()), + "document_status": datastore.DeletedDocumentStatus, } - _, err := db.inner.UpdateOne(ctx, bson.M{"uid": uid}, update) + err := db.store.UpdateOne(ctx, bson.M{"uid": uid}, update) if err != nil { return err } @@ -72,9 +72,7 @@ func (db *orgRepo) DeleteOrganisation(ctx context.Context, uid string) error { func (db *orgRepo) FetchOrganisationByID(ctx context.Context, id string) (*datastore.Organisation, error) { org := new(datastore.Organisation) - filter := bson.M{"uid": id, "document_status": datastore.ActiveDocumentStatus} - - err := db.inner.FindOne(ctx, filter).Decode(&org) + err := db.store.FindByID(ctx, id, nil, org) if errors.Is(err, mongo.ErrNoDocuments) { err = datastore.ErrOrgNotFound } diff --git a/datastore/mongo/organisation_invite.go b/datastore/mongo/organisation_invite.go index bd03fe1c48..bc3253de3c 100644 --- a/datastore/mongo/organisation_invite.go +++ b/datastore/mongo/organisation_invite.go @@ -17,12 +17,14 @@ import ( type orgInviteRepo struct { innerDB *mongo.Database inner *mongo.Collection + store datastore.Store } -func NewOrgInviteRepo(db *mongo.Database) datastore.OrganisationInviteRepository { +func NewOrgInviteRepo(db *mongo.Database, store datastore.Store) datastore.OrganisationInviteRepository { return &orgInviteRepo{ innerDB: db, inner: db.Collection(OrganisationInvitesCollection), + store: store, } } @@ -48,32 +50,30 @@ func (db *orgInviteRepo) LoadOrganisationsInvitesPaged(ctx context.Context, orgI func (db *orgInviteRepo) CreateOrganisationInvite(ctx context.Context, iv *datastore.OrganisationInvite) error { iv.ID = primitive.NewObjectID() - _, err := db.inner.InsertOne(ctx, iv) + err := db.store.Save(ctx, iv, nil) return err } func (db *orgInviteRepo) UpdateOrganisationInvite(ctx context.Context, iv *datastore.OrganisationInvite) error { iv.UpdatedAt = primitive.NewDateTimeFromTime(time.Now()) - update := bson.D{primitive.E{Key: "$set", Value: bson.D{ + update := bson.D{ primitive.E{Key: "role", Value: iv.Role}, primitive.E{Key: "status", Value: iv.Status}, primitive.E{Key: "updated_at", Value: iv.UpdatedAt}, primitive.E{Key: "expires_at", Value: iv.ExpiresAt}, - }}} + } - _, err := db.inner.UpdateOne(ctx, bson.M{"uid": iv.UID}, update) + err := db.store.UpdateOne(ctx, bson.M{"uid": iv.UID}, update) return err } func (db *orgInviteRepo) DeleteOrganisationInvite(ctx context.Context, uid string) error { update := bson.M{ - "$set": bson.M{ - "deleted_at": primitive.NewDateTimeFromTime(time.Now()), - "document_status": datastore.DeletedDocumentStatus, - }, + "deleted_at": primitive.NewDateTimeFromTime(time.Now()), + "document_status": datastore.DeletedDocumentStatus, } - _, err := db.inner.UpdateOne(ctx, bson.M{"uid": uid}, update) + err := db.store.UpdateOne(ctx, bson.M{"uid": uid}, update) if err != nil { return err } @@ -84,12 +84,7 @@ func (db *orgInviteRepo) DeleteOrganisationInvite(ctx context.Context, uid strin func (db *orgInviteRepo) FetchOrganisationInviteByID(ctx context.Context, id string) (*datastore.OrganisationInvite, error) { org := &datastore.OrganisationInvite{} - filter := bson.M{ - "uid": id, - "document_status": datastore.ActiveDocumentStatus, - } - - err := db.inner.FindOne(ctx, filter).Decode(org) + err := db.store.FindByID(ctx, id, nil, org) if errors.Is(err, mongo.ErrNoDocuments) { err = datastore.ErrOrgInviteNotFound } @@ -105,7 +100,7 @@ func (db *orgInviteRepo) FetchOrganisationInviteByToken(ctx context.Context, tok "document_status": datastore.ActiveDocumentStatus, } - err := db.inner.FindOne(ctx, filter).Decode(org) + err := db.store.FindOne(ctx, filter, nil, org) if errors.Is(err, mongo.ErrNoDocuments) { err = datastore.ErrOrgInviteNotFound } diff --git a/datastore/mongo/organisation_invite_test.go b/datastore/mongo/organisation_invite_test.go index 2ca890c8c7..3a8b46fc32 100644 --- a/datastore/mongo/organisation_invite_test.go +++ b/datastore/mongo/organisation_invite_test.go @@ -6,10 +6,11 @@ package mongo import ( "context" "fmt" - "github.com/frain-dev/convoy/auth" "testing" "time" + "github.com/frain-dev/convoy/auth" + "github.com/frain-dev/convoy/datastore" "github.com/google/uuid" "github.com/stretchr/testify/require" @@ -19,8 +20,9 @@ import ( func TestLoadOrganisationsInvitesPaged(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - - inviteRepo := NewOrgInviteRepo(db) + inviteStore := getStore(db, OrganisationInvitesCollection) + orgStore := getStore(db, OrganisationCollection) + inviteRepo := NewOrgInviteRepo(db, inviteStore) org := &datastore.Organisation{ UID: uuid.NewString(), Name: "test_org", @@ -29,7 +31,7 @@ func TestLoadOrganisationsInvitesPaged(t *testing.T) { UpdatedAt: primitive.NewDateTimeFromTime(time.Now()), } - err := NewOrgRepo(db).CreateOrganisation(context.Background(), org) + err := NewOrgRepo(db, orgStore).CreateOrganisation(context.Background(), org) require.NoError(t, err) uids := []string{} @@ -84,8 +86,8 @@ func TestCreateOrganisationInvite(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - inviteRepo := NewOrgInviteRepo(db) - + inviteStore := getStore(db, OrganisationInvitesCollection) + inviteRepo := NewOrgInviteRepo(db, inviteStore) iv := &datastore.OrganisationInvite{ UID: uuid.NewString(), InviteeEmail: fmt.Sprintf("%s@gmail.com", uuid.NewString()), @@ -111,7 +113,8 @@ func TestUpdateOrganisationInvite(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - inviteRepo := NewOrgInviteRepo(db) + inviteStore := getStore(db, OrganisationInvitesCollection) + inviteRepo := NewOrgInviteRepo(db, inviteStore) iv := &datastore.OrganisationInvite{ UID: uuid.NewString(), @@ -158,7 +161,8 @@ func TestDeleteOrganisationInvite(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - inviteRepo := NewOrgInviteRepo(db) + inviteStore := getStore(db, OrganisationInvitesCollection) + inviteRepo := NewOrgInviteRepo(db, inviteStore) org := &datastore.OrganisationInvite{ UID: uuid.NewString(), @@ -188,7 +192,9 @@ func TestFetchOrganisationInviteByID(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - inviteRepo := NewOrgInviteRepo(db) + inviteStore := getStore(db, OrganisationInvitesCollection) + inviteRepo := NewOrgInviteRepo(db, inviteStore) + iv := &datastore.OrganisationInvite{ UID: uuid.NewString(), InviteeEmail: fmt.Sprintf("%s@gmail.com", uuid.NewString()), @@ -218,7 +224,9 @@ func TestFetchOrganisationInviteByTokenAndEmail(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - inviteRepo := NewOrgInviteRepo(db) + inviteStore := getStore(db, OrganisationInvitesCollection) + inviteRepo := NewOrgInviteRepo(db, inviteStore) + iv := &datastore.OrganisationInvite{ UID: uuid.NewString(), InviteeEmail: fmt.Sprintf("%s@gmail.com", uuid.NewString()), diff --git a/datastore/mongo/organisation_member.go b/datastore/mongo/organisation_member.go index 64257701e3..069327eaaa 100644 --- a/datastore/mongo/organisation_member.go +++ b/datastore/mongo/organisation_member.go @@ -3,6 +3,8 @@ package mongo import ( "context" "errors" + "time" + "github.com/frain-dev/convoy/datastore" "github.com/frain-dev/convoy/util" pager "github.com/gobeam/mongo-go-pagination" @@ -10,18 +12,19 @@ import ( "go.mongodb.org/mongo-driver/bson" "go.mongodb.org/mongo-driver/bson/primitive" "go.mongodb.org/mongo-driver/mongo" - "time" ) type orgMemberRepo struct { innerDB *mongo.Database inner *mongo.Collection + store datastore.Store } -func NewOrgMemberRepo(db *mongo.Database) datastore.OrganisationMemberRepository { +func NewOrgMemberRepo(db *mongo.Database, store datastore.Store) datastore.OrganisationMemberRepository { return &orgMemberRepo{ innerDB: db, inner: db.Collection(OrganisationMembersCollection), + store: store, } } @@ -112,19 +115,9 @@ func (o *orgMemberRepo) LoadUserOrganisationsPaged(ctx context.Context, userID s }, }, } - - data, err := o.inner.Aggregate(ctx, mongo.Pipeline{matchStage1, sortStage, skipStage, limitStage, lookupStage, unwindStage, replaceRootStage, matchStage2}) - if err != nil { - log.WithError(err).Error("failed to run user organisations aggregation") - return nil, datastore.PaginationData{}, err - } - if err != nil { - return nil, datastore.PaginationData{}, err - } - organisations := make([]datastore.Organisation, 0) - err = data.All(ctx, &organisations) + err := o.store.Aggregate(ctx, mongo.Pipeline{matchStage1, sortStage, skipStage, limitStage, lookupStage, unwindStage, replaceRootStage, matchStage2}, &organisations, false) if err != nil { log.WithError(err).Error("failed to run user organisations aggregation") return nil, datastore.PaginationData{}, err @@ -135,28 +128,26 @@ func (o *orgMemberRepo) LoadUserOrganisationsPaged(ctx context.Context, userID s func (o *orgMemberRepo) CreateOrganisationMember(ctx context.Context, member *datastore.OrganisationMember) error { member.ID = primitive.NewObjectID() - _, err := o.inner.InsertOne(ctx, member) + err := o.store.Save(ctx, member, nil) return err } func (o *orgMemberRepo) UpdateOrganisationMember(ctx context.Context, member *datastore.OrganisationMember) error { member.UpdatedAt = primitive.NewDateTimeFromTime(time.Now()) - update := bson.D{primitive.E{Key: "$set", Value: bson.D{ + update := bson.D{ primitive.E{Key: "role", Value: member.Role}, primitive.E{Key: "updated_at", Value: member.UpdatedAt}, - }}} + } - _, err := o.inner.UpdateOne(ctx, bson.M{"uid": member.UID}, update) + err := o.store.UpdateOne(ctx, bson.M{"uid": member.UID}, update) return err } func (o *orgMemberRepo) DeleteOrganisationMember(ctx context.Context, uid, orgID string) error { update := bson.M{ - "$set": bson.M{ - "deleted_at": primitive.NewDateTimeFromTime(time.Now()), - "document_status": datastore.DeletedDocumentStatus, - }, + "deleted_at": primitive.NewDateTimeFromTime(time.Now()), + "document_status": datastore.DeletedDocumentStatus, } filter := bson.M{ @@ -164,7 +155,7 @@ func (o *orgMemberRepo) DeleteOrganisationMember(ctx context.Context, uid, orgID "organisation_id": orgID, } - _, err := o.inner.UpdateOne(ctx, filter, update) + err := o.store.UpdateOne(ctx, filter, update) if err != nil { return err } @@ -181,7 +172,7 @@ func (o *orgMemberRepo) FetchOrganisationMemberByID(ctx context.Context, uid, or "document_status": datastore.ActiveDocumentStatus, } - err := o.inner.FindOne(ctx, filter).Decode(&member) + err := o.store.FindOne(ctx, filter, nil, member) if errors.Is(err, mongo.ErrNoDocuments) { return nil, datastore.ErrOrgMemberNotFound } @@ -198,7 +189,7 @@ func (o *orgMemberRepo) FetchOrganisationMemberByUserID(ctx context.Context, use } member := new(datastore.OrganisationMember) - err := o.inner.FindOne(ctx, filter).Decode(member) + err := o.store.FindOne(ctx, filter, nil, member) if errors.Is(err, mongo.ErrNoDocuments) { return nil, datastore.ErrOrgMemberNotFound } @@ -263,19 +254,14 @@ func (o *orgMemberRepo) fillOrgMemberUserMetadata(ctx context.Context, members [ {Key: "email", Value: "$email"}, }}, } + var userMetadata []datastore.UserMetadata - data, err := o.inner.Aggregate(ctx, mongo.Pipeline{matchStage, lookupStage, projectStage1, replaceRootStage, projectStage2}) + err := o.store.Aggregate(ctx, mongo.Pipeline{matchStage, lookupStage, projectStage1, replaceRootStage, projectStage2}, &userMetadata, false) if err != nil { log.WithError(err).Error("failed to run user metadata for organisation members aggregation") return err } - var userMetadata []datastore.UserMetadata - if err = data.All(ctx, &userMetadata); err != nil { - log.WithError(err).Error("failed to marshal user metadata for organisation members") - return err - } - metaMap := map[string]*datastore.UserMetadata{} for i, s := range userMetadata { metaMap[s.UserID] = &userMetadata[i] diff --git a/datastore/mongo/organisation_member_test.go b/datastore/mongo/organisation_member_test.go index b83677b859..b36dfd7432 100644 --- a/datastore/mongo/organisation_member_test.go +++ b/datastore/mongo/organisation_member_test.go @@ -6,10 +6,11 @@ package mongo import ( "context" "fmt" - "github.com/frain-dev/convoy/auth" "testing" "time" + "github.com/frain-dev/convoy/auth" + "github.com/frain-dev/convoy/datastore" "github.com/google/uuid" "github.com/stretchr/testify/require" @@ -20,7 +21,9 @@ func TestLoadOrganisationMembersPaged(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - organisationMemberRepo := NewOrgMemberRepo(db) + store := getStore(db, OrganisationMembersCollection) + userStore := getStore(db, UserCollection) + organisationMemberRepo := NewOrgMemberRepo(db, store) orgID := uuid.NewString() userMap := map[string]*datastore.UserMetadata{} @@ -36,7 +39,7 @@ func TestLoadOrganisationMembersPaged(t *testing.T) { UpdatedAt: primitive.NewDateTimeFromTime(time.Now()), DocumentStatus: datastore.ActiveDocumentStatus, } - require.NoError(t, NewUserRepo(db).CreateUser(context.Background(), user)) + require.NoError(t, NewUserRepo(db, userStore).CreateUser(context.Background(), user)) member := &datastore.OrganisationMember{ UID: uuid.NewString(), @@ -78,7 +81,9 @@ func TestLoadUserOrganisationsPaged(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - organisationMemberRepo := NewOrgMemberRepo(db) + orgMemberStore := getStore(db, OrganisationMembersCollection) + orgStore := getStore(db, OrganisationCollection) + organisationMemberRepo := NewOrgMemberRepo(db, orgMemberStore) userID := uuid.NewString() for i := 0; i < 7; i++ { @@ -88,7 +93,7 @@ func TestLoadUserOrganisationsPaged(t *testing.T) { } org := &datastore.Organisation{UID: uuid.NewString(), DocumentStatus: status} - err := NewOrgRepo(db).CreateOrganisation(context.Background(), org) + err := NewOrgRepo(db, orgStore).CreateOrganisation(context.Background(), org) require.NoError(t, err) member := &datastore.OrganisationMember{ @@ -119,6 +124,7 @@ func TestCreateOrganisationMember(t *testing.T) { db, closeFn := getDB(t) defer closeFn() + userStore := getStore(db, UserCollection) user := &datastore.User{ UID: uuid.NewString(), FirstName: fmt.Sprintf("test-%s", uuid.NewString()), @@ -129,9 +135,10 @@ func TestCreateOrganisationMember(t *testing.T) { UpdatedAt: primitive.NewDateTimeFromTime(time.Now()), DocumentStatus: datastore.ActiveDocumentStatus, } - require.NoError(t, NewUserRepo(db).CreateUser(context.Background(), user)) + require.NoError(t, NewUserRepo(db, userStore).CreateUser(context.Background(), user)) - organisationMemberRepo := NewOrgMemberRepo(db) + orgMemberStore := getStore(db, OrganisationMembersCollection) + organisationMemberRepo := NewOrgMemberRepo(db, orgMemberStore) m := &datastore.OrganisationMember{ UID: uuid.NewString(), @@ -164,6 +171,7 @@ func TestUpdateOrganisationMember(t *testing.T) { db, closeFn := getDB(t) defer closeFn() + userStore := getStore(db, UserCollection) user := &datastore.User{ UID: uuid.NewString(), FirstName: fmt.Sprintf("test-%s", uuid.NewString()), @@ -174,10 +182,10 @@ func TestUpdateOrganisationMember(t *testing.T) { UpdatedAt: primitive.NewDateTimeFromTime(time.Now()), DocumentStatus: datastore.ActiveDocumentStatus, } - require.NoError(t, NewUserRepo(db).CreateUser(context.Background(), user)) - - organisationMemberRepo := NewOrgMemberRepo(db) + require.NoError(t, NewUserRepo(db, userStore).CreateUser(context.Background(), user)) + orgMemberStore := getStore(db, OrganisationMembersCollection) + organisationMemberRepo := NewOrgMemberRepo(db, orgMemberStore) m := &datastore.OrganisationMember{ UID: uuid.NewString(), OrganisationID: uuid.NewString(), @@ -218,7 +226,9 @@ func TestDeleteOrganisationMember(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - organisationMemberRepo := NewOrgMemberRepo(db) + orgMemberStore := getStore(db, OrganisationMembersCollection) + organisationMemberRepo := NewOrgMemberRepo(db, orgMemberStore) + m := &datastore.OrganisationMember{ UID: uuid.NewString(), OrganisationID: uuid.NewString(), @@ -243,6 +253,7 @@ func TestFetchOrganisationMemberByID(t *testing.T) { db, closeFn := getDB(t) defer closeFn() + userStore := getStore(db, UserCollection) user := &datastore.User{ UID: uuid.NewString(), FirstName: fmt.Sprintf("test-%s", uuid.NewString()), @@ -253,9 +264,10 @@ func TestFetchOrganisationMemberByID(t *testing.T) { UpdatedAt: primitive.NewDateTimeFromTime(time.Now()), DocumentStatus: datastore.ActiveDocumentStatus, } - require.NoError(t, NewUserRepo(db).CreateUser(context.Background(), user)) + require.NoError(t, NewUserRepo(db, userStore).CreateUser(context.Background(), user)) + orgMemberStore := getStore(db, OrganisationMembersCollection) + organisationMemberRepo := NewOrgMemberRepo(db, orgMemberStore) - organisationMemberRepo := NewOrgMemberRepo(db) m := &datastore.OrganisationMember{ UID: uuid.NewString(), OrganisationID: uuid.NewString(), @@ -287,6 +299,7 @@ func TestFetchOrganisationMemberByUserID(t *testing.T) { db, closeFn := getDB(t) defer closeFn() + userStore := getStore(db, UserCollection) user := &datastore.User{ UID: uuid.NewString(), FirstName: fmt.Sprintf("test-%s", uuid.NewString()), @@ -297,9 +310,10 @@ func TestFetchOrganisationMemberByUserID(t *testing.T) { UpdatedAt: primitive.NewDateTimeFromTime(time.Now()), DocumentStatus: datastore.ActiveDocumentStatus, } - require.NoError(t, NewUserRepo(db).CreateUser(context.Background(), user)) + require.NoError(t, NewUserRepo(db, userStore).CreateUser(context.Background(), user)) - organisationMemberRepo := NewOrgMemberRepo(db) + orgMemberStore := getStore(db, OrganisationMembersCollection) + organisationMemberRepo := NewOrgMemberRepo(db, orgMemberStore) m := &datastore.OrganisationMember{ UID: uuid.NewString(), OrganisationID: uuid.NewString(), diff --git a/datastore/mongo/organisation_test.go b/datastore/mongo/organisation_test.go index 7cfa711322..fd575df361 100644 --- a/datastore/mongo/organisation_test.go +++ b/datastore/mongo/organisation_test.go @@ -18,8 +18,8 @@ import ( func TestLoadOrganisationsPaged(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - - orgRepo := NewOrgRepo(db) + store := getStore(db, OrganisationCollection) + orgRepo := NewOrgRepo(db, store) for i := 1; i < 6; i++ { org := &datastore.Organisation{ @@ -48,7 +48,8 @@ func TestCreateOrganisation(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - orgRepo := NewOrgRepo(db) + store := getStore(db, OrganisationCollection) + orgRepo := NewOrgRepo(db, store) org := &datastore.Organisation{ UID: uuid.NewString(), Name: fmt.Sprintf("new org"), @@ -64,7 +65,8 @@ func TestUpdateOrganisation(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - orgRepo := NewOrgRepo(db) + store := getStore(db, OrganisationCollection) + orgRepo := NewOrgRepo(db, store) org := &datastore.Organisation{ UID: uuid.NewString(), Name: fmt.Sprintf("new org"), @@ -92,7 +94,8 @@ func TestFetchOrganisationByID(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - orgRepo := NewOrgRepo(db) + store := getStore(db, OrganisationCollection) + orgRepo := NewOrgRepo(db, store) org := &datastore.Organisation{ UID: uuid.NewString(), Name: fmt.Sprintf("new org"), @@ -114,7 +117,8 @@ func TestDeleteOrganisation(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - orgRepo := NewOrgRepo(db) + store := getStore(db, OrganisationCollection) + orgRepo := NewOrgRepo(db, store) org := &datastore.Organisation{ UID: uuid.NewString(), Name: fmt.Sprintf("new org"), diff --git a/datastore/mongo/source.go b/datastore/mongo/source.go index 0002a4643a..bf9ffa8670 100644 --- a/datastore/mongo/source.go +++ b/datastore/mongo/source.go @@ -16,19 +16,21 @@ import ( type sourceRepo struct { innerDB *mongo.Database client *mongo.Collection + store datastore.Store } -func NewSourceRepo(db *mongo.Database) datastore.SourceRepository { +func NewSourceRepo(db *mongo.Database, store datastore.Store) datastore.SourceRepository { return &sourceRepo{ innerDB: db, client: db.Collection(SourceCollection), + store: store, } } func (s *sourceRepo) CreateSource(ctx context.Context, source *datastore.Source) error { source.ID = primitive.NewObjectID() - _, err := s.client.InsertOne(ctx, source) + err := s.store.Save(ctx, source, nil) return err } @@ -36,40 +38,36 @@ func (s *sourceRepo) UpdateSource(ctx context.Context, groupId string, source *d filter := bson.M{"uid": source.UID, "group_id": groupId, "document_status": datastore.ActiveDocumentStatus} update := bson.D{ - primitive.E{Key: "$set", Value: bson.D{ - primitive.E{Key: "name", Value: source.Name}, - primitive.E{Key: "type", Value: source.Type}, - primitive.E{Key: "is_disabled", Value: source.IsDisabled}, - primitive.E{Key: "verifier", Value: source.Verifier}, - primitive.E{Key: "updated_at", Value: primitive.NewDateTimeFromTime(time.Now())}, - }}, + primitive.E{Key: "name", Value: source.Name}, + primitive.E{Key: "type", Value: source.Type}, + primitive.E{Key: "is_disabled", Value: source.IsDisabled}, + primitive.E{Key: "verifier", Value: source.Verifier}, + primitive.E{Key: "updated_at", Value: primitive.NewDateTimeFromTime(time.Now())}, } - _, err := s.client.UpdateOne(ctx, filter, update) + err := s.store.UpdateOne(ctx, filter, update) return err } func (s *sourceRepo) FindSourceByID(ctx context.Context, groupId string, id string) (*datastore.Source, error) { source := &datastore.Source{} - filter := bson.M{"uid": id, "group_id": groupId, "document_status": datastore.ActiveDocumentStatus} - - err := s.client.FindOne(ctx, filter).Decode(&source) + filter := bson.M{"uid": id, "group_id": groupId} + err := s.store.FindOne(ctx, filter, nil, source) if errors.Is(err, mongo.ErrNoDocuments) { return source, datastore.ErrSourceNotFound } - return source, nil + return source, err } func (s *sourceRepo) FindSourceByMaskID(ctx context.Context, maskId string) (*datastore.Source, error) { source := &datastore.Source{} - filter := bson.M{"mask_id": maskId, "document_status": datastore.ActiveDocumentStatus} - - err := s.client.FindOne(ctx, filter).Decode(&source) + filter := bson.M{"mask_id": maskId} + err := s.store.FindOne(ctx, filter, nil, source) if errors.Is(err, mongo.ErrNoDocuments) { return source, datastore.ErrSourceNotFound } @@ -78,16 +76,14 @@ func (s *sourceRepo) FindSourceByMaskID(ctx context.Context, maskId string) (*da } func (s *sourceRepo) DeleteSourceByID(ctx context.Context, groupId string, id string) error { - filter := bson.M{"uid": id, "group_id": groupId, "document_status": datastore.ActiveDocumentStatus} + filter := bson.M{"uid": id, "group_id": groupId} update := bson.M{ - "$set": bson.M{ - "deleted_at": primitive.NewDateTimeFromTime(time.Now()), - "document_status": datastore.DeletedDocumentStatus, - }, + "deleted_at": primitive.NewDateTimeFromTime(time.Now()), + "document_status": datastore.DeletedDocumentStatus, } - _, err := s.client.UpdateOne(ctx, filter, update) + err := s.store.UpdateOne(ctx, filter, update) return err } diff --git a/datastore/mongo/source_test.go b/datastore/mongo/source_test.go index aff4e35762..20ea93df64 100644 --- a/datastore/mongo/source_test.go +++ b/datastore/mongo/source_test.go @@ -18,7 +18,8 @@ func Test_CreateSource(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - sourceRepo := NewSourceRepo(db) + store := getStore(db, SourceCollection) + sourceRepo := NewSourceRepo(db, store) source := generateSource(t) require.NoError(t, sourceRepo.CreateSource(context.Background(), source)) @@ -36,7 +37,8 @@ func Test_FindSourceByID(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - sourceRepo := NewSourceRepo(db) + store := getStore(db, SourceCollection) + sourceRepo := NewSourceRepo(db, store) source := generateSource(t) _, err := sourceRepo.FindSourceByID(context.Background(), source.GroupID, source.UID) @@ -56,7 +58,8 @@ func Test_FindSourceByMaskID(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - sourceRepo := NewSourceRepo(db) + store := getStore(db, SourceCollection) + sourceRepo := NewSourceRepo(db, store) source := generateSource(t) _, err := sourceRepo.FindSourceByMaskID(context.Background(), source.MaskID) @@ -76,7 +79,8 @@ func Test_UpdateSource(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - sourceRepo := NewSourceRepo(db) + store := getStore(db, SourceCollection) + sourceRepo := NewSourceRepo(db, store) source := generateSource(t) require.NoError(t, sourceRepo.CreateSource(context.Background(), source)) @@ -96,7 +100,8 @@ func Test_DeleteSource(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - sourceRepo := NewSourceRepo(db) + store := getStore(db, SourceCollection) + sourceRepo := NewSourceRepo(db, store) source := generateSource(t) require.NoError(t, sourceRepo.CreateSource(context.Background(), source)) @@ -177,7 +182,8 @@ func Test_LoadSourcesPaged(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - sourceRepo := NewSourceRepo(db) + store := getStore(db, SourceCollection) + sourceRepo := NewSourceRepo(db, store) groupId := uuid.NewString() for i := 0; i < tc.count; i++ { diff --git a/datastore/mongo/subscription.go b/datastore/mongo/subscription.go index 55a27ee9b2..b9cac49adb 100644 --- a/datastore/mongo/subscription.go +++ b/datastore/mongo/subscription.go @@ -14,11 +14,13 @@ import ( type subscriptionRepo struct { client *mongo.Collection + store datastore.Store } -func NewSubscriptionRepo(db *mongo.Database) datastore.SubscriptionRepository { +func NewSubscriptionRepo(db *mongo.Database, store datastore.Store) datastore.SubscriptionRepository { return &subscriptionRepo{ client: db.Collection(SubscriptionCollection), + store: store, } } @@ -28,8 +30,7 @@ func (s *subscriptionRepo) CreateSubscription(ctx context.Context, groupId strin } subscription.ID = primitive.NewObjectID() - _, err := s.client.InsertOne(ctx, subscription) - return err + return s.store.Save(ctx, subscription, nil) } func (s *subscriptionRepo) UpdateSubscription(ctx context.Context, groupId string, subscription *datastore.Subscription) error { @@ -45,22 +46,21 @@ func (s *subscriptionRepo) UpdateSubscription(ctx context.Context, groupId strin "document_status": datastore.ActiveDocumentStatus, } - update := bson.D{primitive.E{Key: "$set", Value: bson.D{ - primitive.E{Key: "name", Value: subscription.Name}, - primitive.E{Key: "source_id", Value: subscription.SourceID}, - primitive.E{Key: "endpoint_id", Value: subscription.EndpointID}, - - primitive.E{Key: "filter_config.event_types", Value: subscription.FilterConfig.EventTypes}, + update := bson.M{ + "name": subscription.Name, + "source_id": subscription.SourceID, + "endpoint_id": subscription.EndpointID, - primitive.E{Key: "alert_config.count", Value: subscription.AlertConfig.Count}, - primitive.E{Key: "alert_config.threshold", Value: subscription.AlertConfig.Threshold}, + "filter_config.event_types": subscription.FilterConfig.EventTypes, + "alert_config.count": subscription.AlertConfig.Count, + "alert_config.threshold": subscription.AlertConfig.Threshold, - primitive.E{Key: "retry_config.type", Value: string(subscription.RetryConfig.Type)}, - primitive.E{Key: "retry_config.duration", Value: subscription.RetryConfig.Duration}, - primitive.E{Key: "retry_config.retry_count", Value: subscription.RetryConfig.RetryCount}, - }}} + "retry_config.type": string(subscription.RetryConfig.Type), + "retry_config.duration": subscription.RetryConfig.Duration, + "retry_config.retry_count": subscription.RetryConfig.RetryCount, + } - _, err := s.client.UpdateOne(ctx, filter, update) + err := s.store.UpdateOne(ctx, filter, update) return err } @@ -90,28 +90,18 @@ func (s *subscriptionRepo) DeleteSubscription(ctx context.Context, groupId strin return datastore.ErrNotAuthorisedToAccessDocument } - update := bson.M{ - "$set": bson.M{ - "deleted_at": primitive.NewDateTimeFromTime(time.Now()), - "document_status": datastore.DeletedDocumentStatus, - }, - } - - filter := bson.M{"uid": subscription.UID, "group_id": groupId} - _, err := s.client.UpdateOne(ctx, filter, update) - if err != nil { - return err + filter := bson.M{ + "uid": subscription.UID, + "group_id": groupId, } - - return nil + return s.store.DeleteOne(ctx, filter) } func (s *subscriptionRepo) FindSubscriptionByID(ctx context.Context, groupId string, uid string) (*datastore.Subscription, error) { - var subscription *datastore.Subscription + subscription := &datastore.Subscription{} filter := bson.M{"uid": uid, "group_id": groupId, "document_status": datastore.ActiveDocumentStatus} - err := s.client.FindOne(ctx, filter).Decode(&subscription) - + err := s.store.FindOne(ctx, filter, nil, subscription) if errors.Is(err, mongo.ErrNoDocuments) { err = datastore.ErrSubscriptionNotFound } @@ -119,21 +109,16 @@ func (s *subscriptionRepo) FindSubscriptionByID(ctx context.Context, groupId str return subscription, err } -func (s *subscriptionRepo) FindSubscriptionByEventType(ctx context.Context, groupId string, appId string, eventType datastore.EventType) ([]datastore.Subscription, error) { - var subscription []datastore.Subscription +func (s *subscriptionRepo) FindSubscriptionsByEventType(ctx context.Context, groupId string, appId string, eventType datastore.EventType) ([]datastore.Subscription, error) { + var subscriptions []datastore.Subscription filter := bson.M{"group_id": groupId, "app_id": appId, "filter_config.event_types": string(eventType), "document_status": datastore.ActiveDocumentStatus} - c, err := s.client.Find(ctx, filter) - if err != nil { - return nil, err - } - - err = c.All(ctx, &subscription) + err := s.store.FindMany(ctx, filter, nil, nil, 0, 0, subscriptions) if err != nil { return nil, err } - return subscription, nil + return subscriptions, nil } func (s *subscriptionRepo) FindSubscriptionsByAppID(ctx context.Context, groupId string, appID string) ([]datastore.Subscription, error) { @@ -143,35 +128,25 @@ func (s *subscriptionRepo) FindSubscriptionsByAppID(ctx context.Context, groupId "document_status": datastore.ActiveDocumentStatus, } - c, err := s.client.Find(ctx, filter) + var subscriptions []datastore.Subscription + err := s.store.FindMany(ctx, filter, nil, nil, 0, 0, subscriptions) if errors.Is(err, mongo.ErrNoDocuments) { return nil, datastore.ErrSubscriptionNotFound } - var subscription []datastore.Subscription - err = c.All(ctx, &subscription) - if err != nil { - return nil, err - } - - return subscription, nil + return subscriptions, nil } -func (s *subscriptionRepo) FindSubscriptionBySourceIDs(ctx context.Context, groupId string, sourceId string) ([]datastore.Subscription, error) { - var subscription []datastore.Subscription +func (s *subscriptionRepo) FindSubscriptionsBySourceIDs(ctx context.Context, groupId string, sourceId string) ([]datastore.Subscription, error) { + var subscriptions []datastore.Subscription filter := bson.M{"group_id": groupId, "source_id": sourceId, "document_status": datastore.ActiveDocumentStatus} - c, err := s.client.Find(ctx, filter) + err := s.store.FindMany(ctx, filter, nil, nil, 0, 0, subscriptions) if err != nil { return nil, err } - err = c.All(ctx, &subscription) - if err != nil { - return nil, err - } - - return subscription, nil + return subscriptions, nil } func (s *subscriptionRepo) UpdateSubscriptionStatus(ctx context.Context, groupId string, subscriptionId string, status datastore.SubscriptionStatus) error { @@ -181,12 +156,11 @@ func (s *subscriptionRepo) UpdateSubscriptionStatus(ctx context.Context, groupId "document_status": datastore.ActiveDocumentStatus, } - updatedAt := primitive.NewDateTimeFromTime(time.Now()) - update := bson.D{primitive.E{Key: "$set", Value: bson.D{ - primitive.E{Key: "status", Value: status}, - primitive.E{Key: "updated_at", Value: updatedAt}, - }}} + update := bson.M{ + "status": status, + "updated_at": primitive.NewDateTimeFromTime(time.Now()), + } - _, err := s.client.UpdateOne(ctx, filter, update) + err := s.store.UpdateOne(ctx, filter, update) return err } diff --git a/datastore/mongo/subscription_test.go b/datastore/mongo/subscription_test.go index 3657c4b8dd..379fc0e450 100644 --- a/datastore/mongo/subscription_test.go +++ b/datastore/mongo/subscription_test.go @@ -42,7 +42,7 @@ func Test_LoadSubscriptionsPaged(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - subRepo := NewSubscriptionRepo(db) + subRepo := NewSubscriptionRepo(db, datastore.New(db, SubscriptionCollection)) for i := 0; i < 20; i++ { subscription := &datastore.Subscription{ @@ -132,7 +132,7 @@ func Test_DeleteSubscription(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - subRepo := NewSubscriptionRepo(db) + subRepo := NewSubscriptionRepo(db, datastore.New(db, SubscriptionCollection)) newSub := createSubscription() require.NoError(t, subRepo.CreateSubscription(context.Background(), newSub.GroupID, newSub)) @@ -151,7 +151,7 @@ func Test_CreateSubscription(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - subRepo := NewSubscriptionRepo(db) + subRepo := NewSubscriptionRepo(db, datastore.New(db, SubscriptionCollection)) newSub := createSubscription() require.NoError(t, subRepo.CreateSubscription(context.Background(), newSub.GroupID, newSub)) @@ -168,7 +168,7 @@ func Test_FindSubscriptionByID(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - subRepo := NewSubscriptionRepo(db) + subRepo := NewSubscriptionRepo(db, datastore.New(db, SubscriptionCollection)) newSub := createSubscription() // Fetch sub again @@ -191,7 +191,7 @@ func Test_FindSubscriptionByAppID(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - subRepo := NewSubscriptionRepo(db) + subRepo := NewSubscriptionRepo(db, datastore.New(db, SubscriptionCollection)) for i := 0; i < 20; i++ { subscription := &datastore.Subscription{ diff --git a/datastore/mongo/user.go b/datastore/mongo/user.go index 83c88ba7c8..99eb9fb2ba 100644 --- a/datastore/mongo/user.go +++ b/datastore/mongo/user.go @@ -16,12 +16,14 @@ import ( type userRepo struct { innerDB *mongo.Database client *mongo.Collection + store datastore.Store } -func NewUserRepo(db *mongo.Database) datastore.UserRepository { +func NewUserRepo(db *mongo.Database, store datastore.Store) datastore.UserRepository { return &userRepo{ innerDB: db, client: db.Collection(UserCollection), + store: store, } } @@ -29,16 +31,16 @@ func (u *userRepo) CreateUser(ctx context.Context, user *datastore.User) error { user.ID = primitive.NewObjectID() user.ResetPasswordToken = uuid.NewString() - _, err := u.client.InsertOne(ctx, user) + err := u.store.Save(ctx, user, nil) return err } func (u *userRepo) FindUserByEmail(ctx context.Context, email string) (*datastore.User, error) { user := &datastore.User{} - filter := bson.M{"email": email, "document_status": datastore.ActiveDocumentStatus} + filter := bson.M{"email": email} - err := u.client.FindOne(ctx, filter).Decode(&user) + err := u.store.FindOne(ctx, filter, nil, user) if errors.Is(err, mongo.ErrNoDocuments) { return user, datastore.ErrUserNotFound @@ -50,9 +52,7 @@ func (u *userRepo) FindUserByEmail(ctx context.Context, email string) (*datastor func (u *userRepo) FindUserByID(ctx context.Context, id string) (*datastore.User, error) { user := &datastore.User{} - filter := bson.M{"uid": id, "document_status": datastore.ActiveDocumentStatus} - - err := u.client.FindOne(ctx, filter).Decode(&user) + err := u.store.FindByID(ctx, id, nil, user) if errors.Is(err, mongo.ErrNoDocuments) { return user, datastore.ErrUserNotFound @@ -79,31 +79,26 @@ func (u *userRepo) LoadUsersPaged(ctx context.Context, pageable datastore.Pageab } func (u *userRepo) UpdateUser(ctx context.Context, user *datastore.User) error { - filter := bson.M{"uid": user.UID, "document_status": datastore.ActiveDocumentStatus} - update := bson.D{ - primitive.E{Key: "$set", Value: bson.D{ - primitive.E{Key: "first_name", Value: user.FirstName}, - primitive.E{Key: "last_name", Value: user.LastName}, - primitive.E{Key: "email", Value: user.Email}, - primitive.E{Key: "password", Value: user.Password}, - primitive.E{Key: "updated_at", Value: primitive.NewDateTimeFromTime(time.Now())}, - primitive.E{Key: "reset_password_token", Value: user.ResetPasswordToken}, - primitive.E{Key: "reset_password_expires_at", Value: user.ResetPasswordExpiresAt}, - }}, + primitive.E{Key: "first_name", Value: user.FirstName}, + primitive.E{Key: "last_name", Value: user.LastName}, + primitive.E{Key: "email", Value: user.Email}, + primitive.E{Key: "password", Value: user.Password}, + primitive.E{Key: "updated_at", Value: primitive.NewDateTimeFromTime(time.Now())}, + primitive.E{Key: "reset_password_token", Value: user.ResetPasswordToken}, + primitive.E{Key: "reset_password_expires_at", Value: user.ResetPasswordExpiresAt}, } - _, err := u.client.UpdateOne(ctx, filter, update) + err := u.store.UpdateByID(ctx, user.UID, update) return err } func (u *userRepo) FindUserByToken(ctx context.Context, token string) (*datastore.User, error) { user := &datastore.User{} - filter := bson.M{"reset_password_token": token, "document_status": datastore.ActiveDocumentStatus} - - err := u.client.FindOne(ctx, filter).Decode(&user) + filter := bson.M{"reset_password_token": token} + err := u.store.FindOne(ctx, filter, nil, user) if errors.Is(err, mongo.ErrNoDocuments) { return user, datastore.ErrUserNotFound } diff --git a/datastore/mongo/user_test.go b/datastore/mongo/user_test.go index e19bbb8bae..6362ae5c52 100644 --- a/datastore/mongo/user_test.go +++ b/datastore/mongo/user_test.go @@ -18,7 +18,8 @@ func Test_CreateUser(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - userRepo := NewUserRepo(db) + store := getStore(db, UserCollection) + userRepo := NewUserRepo(db, store) user := generateUser(t) require.NoError(t, userRepo.CreateUser(context.Background(), user)) @@ -34,7 +35,8 @@ func Test_FindUserByEmail(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - userRepo := NewUserRepo(db) + store := getStore(db, UserCollection) + userRepo := NewUserRepo(db, store) user := generateUser(t) _, err := userRepo.FindUserByEmail(context.Background(), user.Email) @@ -55,7 +57,8 @@ func Test_FindUserByID(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - userRepo := NewUserRepo(db) + store := getStore(db, UserCollection) + userRepo := NewUserRepo(db, store) user := generateUser(t) _, err := userRepo.FindUserByID(context.Background(), user.UID) @@ -138,7 +141,8 @@ func Test_LoadUsersPaged(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - userRepo := NewUserRepo(db) + store := getStore(db, UserCollection) + userRepo := NewUserRepo(db, store) for i := 0; i < tc.count; i++ { user := &datastore.User{ UID: uuid.NewString(), @@ -165,7 +169,8 @@ func Test_UpdateUser(t *testing.T) { db, closeFn := getDB(t) defer closeFn() - userRepo := NewUserRepo(db) + store := getStore(db, UserCollection) + userRepo := NewUserRepo(db, store) user := generateUser(t) require.NoError(t, userRepo.CreateUser(context.Background(), user)) diff --git a/datastore/repository.go b/datastore/repository.go index b95db7992a..876dfb6728 100644 --- a/datastore/repository.go +++ b/datastore/repository.go @@ -95,8 +95,8 @@ type SubscriptionRepository interface { LoadSubscriptionsPaged(context.Context, string, Pageable) ([]Subscription, PaginationData, error) DeleteSubscription(context.Context, string, *Subscription) error FindSubscriptionByID(context.Context, string, string) (*Subscription, error) - FindSubscriptionByEventType(context.Context, string, string, EventType) ([]Subscription, error) - FindSubscriptionBySourceIDs(context.Context, string, string) ([]Subscription, error) + FindSubscriptionsByEventType(context.Context, string, string, EventType) ([]Subscription, error) + FindSubscriptionsBySourceIDs(context.Context, string, string) ([]Subscription, error) FindSubscriptionsByAppID(ctx context.Context, groupId string, appID string) ([]Subscription, error) UpdateSubscriptionStatus(context.Context, string, string, SubscriptionStatus) error } diff --git a/docs/docs.go b/docs/docs.go index 3e88ff98bb..0b3f4d048f 100644 --- a/docs/docs.go +++ b/docs/docs.go @@ -1,6 +1,7 @@ // Package docs GENERATED BY THE COMMAND ABOVE; DO NOT EDIT // This file was generated by swaggo/swag at -// 2022-06-23 12:56:21.496428475 +0100 WAT m=+79.618346290 +// 2022-06-23 16:01:47.052769 +0100 WAT m=+108.083357501 + package docs import ( diff --git a/mocks/repository.go b/mocks/repository.go index ef7ac79bcc..66c0106a78 100644 --- a/mocks/repository.go +++ b/mocks/repository.go @@ -1129,64 +1129,64 @@ func (mr *MockSubscriptionRepositoryMockRecorder) DeleteSubscription(arg0, arg1, return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DeleteSubscription", reflect.TypeOf((*MockSubscriptionRepository)(nil).DeleteSubscription), arg0, arg1, arg2) } -// FindSubscriptionByEventType mocks base method. -func (m *MockSubscriptionRepository) FindSubscriptionByEventType(arg0 context.Context, arg1, arg2 string, arg3 datastore.EventType) ([]datastore.Subscription, error) { +// FindSubscriptionByID mocks base method. +func (m *MockSubscriptionRepository) FindSubscriptionByID(arg0 context.Context, arg1, arg2 string) (*datastore.Subscription, error) { m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "FindSubscriptionByEventType", arg0, arg1, arg2, arg3) - ret0, _ := ret[0].([]datastore.Subscription) + ret := m.ctrl.Call(m, "FindSubscriptionByID", arg0, arg1, arg2) + ret0, _ := ret[0].(*datastore.Subscription) ret1, _ := ret[1].(error) return ret0, ret1 } -// FindSubscriptionByEventType indicates an expected call of FindSubscriptionByEventType. -func (mr *MockSubscriptionRepositoryMockRecorder) FindSubscriptionByEventType(arg0, arg1, arg2, arg3 interface{}) *gomock.Call { +// FindSubscriptionByID indicates an expected call of FindSubscriptionByID. +func (mr *MockSubscriptionRepositoryMockRecorder) FindSubscriptionByID(arg0, arg1, arg2 interface{}) *gomock.Call { mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "FindSubscriptionByEventType", reflect.TypeOf((*MockSubscriptionRepository)(nil).FindSubscriptionByEventType), arg0, arg1, arg2, arg3) + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "FindSubscriptionByID", reflect.TypeOf((*MockSubscriptionRepository)(nil).FindSubscriptionByID), arg0, arg1, arg2) } -// FindSubscriptionByID mocks base method. -func (m *MockSubscriptionRepository) FindSubscriptionByID(arg0 context.Context, arg1, arg2 string) (*datastore.Subscription, error) { +// FindSubscriptionsByAppID mocks base method. +func (m *MockSubscriptionRepository) FindSubscriptionsByAppID(ctx context.Context, groupId, appID string) ([]datastore.Subscription, error) { m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "FindSubscriptionByID", arg0, arg1, arg2) - ret0, _ := ret[0].(*datastore.Subscription) + ret := m.ctrl.Call(m, "FindSubscriptionsByAppID", ctx, groupId, appID) + ret0, _ := ret[0].([]datastore.Subscription) ret1, _ := ret[1].(error) return ret0, ret1 } -// FindSubscriptionByID indicates an expected call of FindSubscriptionByID. -func (mr *MockSubscriptionRepositoryMockRecorder) FindSubscriptionByID(arg0, arg1, arg2 interface{}) *gomock.Call { +// FindSubscriptionsByAppID indicates an expected call of FindSubscriptionsByAppID. +func (mr *MockSubscriptionRepositoryMockRecorder) FindSubscriptionsByAppID(ctx, groupId, appID interface{}) *gomock.Call { mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "FindSubscriptionByID", reflect.TypeOf((*MockSubscriptionRepository)(nil).FindSubscriptionByID), arg0, arg1, arg2) + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "FindSubscriptionsByAppID", reflect.TypeOf((*MockSubscriptionRepository)(nil).FindSubscriptionsByAppID), ctx, groupId, appID) } -// FindSubscriptionBySourceIDs mocks base method. -func (m *MockSubscriptionRepository) FindSubscriptionBySourceIDs(arg0 context.Context, arg1, arg2 string) ([]datastore.Subscription, error) { +// FindSubscriptionsByEventType mocks base method. +func (m *MockSubscriptionRepository) FindSubscriptionsByEventType(arg0 context.Context, arg1, arg2 string, arg3 datastore.EventType) ([]datastore.Subscription, error) { m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "FindSubscriptionBySourceIDs", arg0, arg1, arg2) + ret := m.ctrl.Call(m, "FindSubscriptionsByEventType", arg0, arg1, arg2, arg3) ret0, _ := ret[0].([]datastore.Subscription) ret1, _ := ret[1].(error) return ret0, ret1 } -// FindSubscriptionBySourceIDs indicates an expected call of FindSubscriptionBySourceIDs. -func (mr *MockSubscriptionRepositoryMockRecorder) FindSubscriptionBySourceIDs(arg0, arg1, arg2 interface{}) *gomock.Call { +// FindSubscriptionsByEventType indicates an expected call of FindSubscriptionsByEventType. +func (mr *MockSubscriptionRepositoryMockRecorder) FindSubscriptionsByEventType(arg0, arg1, arg2, arg3 interface{}) *gomock.Call { mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "FindSubscriptionBySourceIDs", reflect.TypeOf((*MockSubscriptionRepository)(nil).FindSubscriptionBySourceIDs), arg0, arg1, arg2) + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "FindSubscriptionsByEventType", reflect.TypeOf((*MockSubscriptionRepository)(nil).FindSubscriptionsByEventType), arg0, arg1, arg2, arg3) } -// FindSubscriptionsByAppID mocks base method. -func (m *MockSubscriptionRepository) FindSubscriptionsByAppID(ctx context.Context, groupId, appID string) ([]datastore.Subscription, error) { +// FindSubscriptionsBySourceIDs mocks base method. +func (m *MockSubscriptionRepository) FindSubscriptionsBySourceIDs(arg0 context.Context, arg1, arg2 string) ([]datastore.Subscription, error) { m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "FindSubscriptionsByAppID", ctx, groupId, appID) + ret := m.ctrl.Call(m, "FindSubscriptionsBySourceIDs", arg0, arg1, arg2) ret0, _ := ret[0].([]datastore.Subscription) ret1, _ := ret[1].(error) return ret0, ret1 } -// FindSubscriptionsByAppID indicates an expected call of FindSubscriptionsByAppID. -func (mr *MockSubscriptionRepositoryMockRecorder) FindSubscriptionsByAppID(ctx, groupId, appID interface{}) *gomock.Call { +// FindSubscriptionsBySourceIDs indicates an expected call of FindSubscriptionsBySourceIDs. +func (mr *MockSubscriptionRepositoryMockRecorder) FindSubscriptionsBySourceIDs(arg0, arg1, arg2 interface{}) *gomock.Call { mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "FindSubscriptionsByAppID", reflect.TypeOf((*MockSubscriptionRepository)(nil).FindSubscriptionsByAppID), ctx, groupId, appID) + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "FindSubscriptionsBySourceIDs", reflect.TypeOf((*MockSubscriptionRepository)(nil).FindSubscriptionsBySourceIDs), arg0, arg1, arg2) } // LoadSubscriptionsPaged mocks base method. diff --git a/server/middleware.go b/server/middleware.go index 748f8b82a6..9770365a84 100644 --- a/server/middleware.go +++ b/server/middleware.go @@ -44,13 +44,11 @@ const ( endpointCtx contextKey = "endpoint" eventCtx contextKey = "event" eventDeliveryCtx contextKey = "eventDelivery" - configCtx contextKey = "configCtx" authLoginCtx contextKey = "authLogin" authUserCtx contextKey = "authUser" userCtx contextKey = "user" pageableCtx contextKey = "pageable" pageDataCtx contextKey = "pageData" - dashboardCtx contextKey = "dashboard" deliveryAttemptsCtx contextKey = "deliveryAttempts" hostCtx contextKey = "host" appIdCtx contextKey = "appId" @@ -1092,14 +1090,6 @@ func getPaginationDataFromContext(ctx context.Context) *datastore.PaginationData return ctx.Value(pageDataCtx).(*datastore.PaginationData) } -func setDashboardSummaryInContext(ctx context.Context, d *models.DashboardSummary) context.Context { - return context.WithValue(ctx, dashboardCtx, d) -} - -func getDashboardSummaryFromContext(ctx context.Context) *models.DashboardSummary { - return ctx.Value(dashboardCtx).(*models.DashboardSummary) -} - func setDeliveryAttemptInContext(ctx context.Context, attempt *datastore.DeliveryAttempt) context.Context { return context.WithValue(ctx, deliveryAttemptsCtx, attempt) @@ -1138,14 +1128,6 @@ func getAuthLoginFromContext(ctx context.Context) *AuthorizedLogin { return ctx.Value(authLoginCtx).(*AuthorizedLogin) } -func setConfigInContext(ctx context.Context, c *ViewableConfiguration) context.Context { - return context.WithValue(ctx, configCtx, c) -} - -func getConfigFromContext(ctx context.Context) *ViewableConfiguration { - return ctx.Value(configCtx).(*ViewableConfiguration) -} - func setHostInContext(ctx context.Context, baseUrl string) context.Context { return context.WithValue(ctx, hostCtx, baseUrl) } diff --git a/server/organisation_integration_test.go b/server/organisation_integration_test.go index 561885cdad..f128207d35 100644 --- a/server/organisation_integration_test.go +++ b/server/organisation_integration_test.go @@ -6,6 +6,11 @@ package server import ( "context" "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "github.com/frain-dev/convoy/auth" "github.com/frain-dev/convoy/config" "github.com/frain-dev/convoy/datastore" @@ -14,10 +19,6 @@ import ( "github.com/google/uuid" "github.com/stretchr/testify/require" "github.com/stretchr/testify/suite" - "net/http" - "net/http/httptest" - "strings" - "testing" ) type OrganisationIntegrationTestSuite struct { diff --git a/server/organisation_member_integration_test.go b/server/organisation_member_integration_test.go index a6ca5830b1..62c0cba8f2 100644 --- a/server/organisation_member_integration_test.go +++ b/server/organisation_member_integration_test.go @@ -6,6 +6,11 @@ package server import ( "context" "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "github.com/frain-dev/convoy/auth" "github.com/frain-dev/convoy/config" "github.com/frain-dev/convoy/datastore" @@ -14,10 +19,6 @@ import ( "github.com/google/uuid" "github.com/stretchr/testify/require" "github.com/stretchr/testify/suite" - "net/http" - "net/http/httptest" - "strings" - "testing" ) type OrganisationMemberIntegrationTestSuite struct { diff --git a/worker/task/process_event_creation.go b/worker/task/process_event_creation.go index d955783956..87d5ee8933 100644 --- a/worker/task/process_event_creation.go +++ b/worker/task/process_event_creation.go @@ -76,7 +76,7 @@ func ProcessEventCreated(appRepo datastore.ApplicationRepository, eventRepo data subscriptions = matchSubscriptions(string(event.EventType), subs) } else if group.Type == datastore.IncomingGroup { - subscriptions, err = subRepo.FindSubscriptionBySourceIDs(ctx, group.UID, event.SourceID) + subscriptions, err = subRepo.FindSubscriptionsBySourceIDs(ctx, group.UID, event.SourceID) if err != nil { return &EndpointError{Err: errors.New("error fetching subscriptions for this source"), delay: 10 * time.Second} } diff --git a/worker/task/process_event_creation_test.go b/worker/task/process_event_creation_test.go index b475c934ab..e4a0863b7c 100644 --- a/worker/task/process_event_creation_test.go +++ b/worker/task/process_event_creation_test.go @@ -186,7 +186,7 @@ func TestProcessEventCreated(t *testing.T) { }, }, } - s.EXPECT().FindSubscriptionBySourceIDs(gomock.Any(), "group-id-1", "source-id-1").Times(1).Return(subscriptions, nil) + s.EXPECT().FindSubscriptionsBySourceIDs(gomock.Any(), "group-id-1", "source-id-1").Times(1).Return(subscriptions, nil) e, _ := args.eventRepo.(*mocks.MockEventRepository) e.EXPECT().CreateEvent(gomock.Any(), gomock.Any()).Times(1).Return(nil) From 6b9490c089c2d9b2c4b7626b10259407dc113b6e Mon Sep 17 00:00:00 2001 From: Raymond Tukpe Date: Mon, 27 Jun 2022 16:33:16 +0100 Subject: [PATCH 4/7] chore: rename (#806) --- .github/workflows/go.yml | 2 +- .gitignore | 1 - RELEASE.md | 19 ++++++++++++------- 3 files changed, 13 insertions(+), 9 deletions(-) diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml index e0482606b2..f1b947cef7 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yml @@ -51,7 +51,7 @@ jobs: run: go mod download && go mod verify - name: Build app to make sure there are zero issues - run: go build -o hookcamp ./cmd + run: go build -o convoy ./cmd - name: Go vet run: go vet ./... diff --git a/.gitignore b/.gitignore index 3df2122f36..665ff32c9d 100644 --- a/.gitignore +++ b/.gitignore @@ -13,7 +13,6 @@ # Dependency directories (remove the comment below to include it) # vendor/ -hookstack .idea /server/ui/build/* diff --git a/RELEASE.md b/RELEASE.md index adc4aa756e..7604a8fbaa 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,10 +1,13 @@ # Releases -This page describes the release process for hookcamp. + +This page describes the release process for convoy. ## How to cut an Individual release + These instruction is currently only valid for this repo. ### Branch management and versioning strategy + We use [Semantic Versioning](https://semver.org/). We maintain a separate branch for each minor release, named `release-.`, e.g. `release-1.1`, `release-2.0`. @@ -18,6 +21,7 @@ If a bug fix got accidentally merged into main after non-bug-fix changes in main Maintaining the release branches for older minor releases happens on a best effort basis. ### 0. Updating dependencies + A few days before a major or minor release, consider updating the dependencies. Then create a pull request against the main branch. @@ -32,9 +36,11 @@ case, you have to create an issue or pull request in the GitHub project for later follow-up. #### Updating Go dependencies + TBD. #### Updating React dependencies + TBD. ### 1. Prepare your release @@ -51,10 +57,10 @@ For release candidates still update `CHANGELOG.md`, but when you cut the final r Entries in the `CHANGELOG.md` are meant to be in this order: -* `[CHANGE]` -* `[FEATURE]` -* `[ENHANCEMENT]` -* `[BUGFIX]` +- `[CHANGE]` +- `[FEATURE]` +- `[ENHANCEMENT]` +- `[BUGFIX]` ### 2. Draft the new release @@ -75,9 +81,8 @@ Optionally, you can use this handy `.gitconfig` alias. Then release with `git tag-release`. -Once a tag is created, the release process through Github actions will take care of the rest. +Once a tag is created, the release process through Github actions will take care of the rest. TODO: A missing step here which should be later automated. A release needs to be created before the assets can be uploaded to match the tag. :) Finally, wait for the build step for the tag to finish. The point here is to wait for tarballs to be uploaded to the Github release and the container images to be pushed to the Docker Hub and Quay.io. Once that has happened, click _Publish release_, which will make the release publicly visible and create a GitHub notification. - From c70ab049bf1444ba4ddd8e40c7a88d7cf6c23155 Mon Sep 17 00:00:00 2001 From: Ogban Ugot Date: Tue, 28 Jun 2022 11:24:49 +0100 Subject: [PATCH 5/7] Fix: use redis client (#807) --- worker/scheduler.go | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/worker/scheduler.go b/worker/scheduler.go index d00d962d8f..0168277c0e 100644 --- a/worker/scheduler.go +++ b/worker/scheduler.go @@ -16,11 +16,7 @@ type Scheduler struct { } func NewScheduler(queue queue.Queuer) *Scheduler { - scheduler := asynq.NewScheduler(asynq.RedisClientOpt{ - Addr: queue.Options().RedisAddress, - Password: "", - DB: 0, - }, nil) + scheduler := asynq.NewScheduler(queue.Options().RedisClient, nil) w, err := NewConsumer(queue) if err != nil { From 73f0eb7e208ee3680a0ca292af53ff56d04195fa Mon Sep 17 00:00:00 2001 From: Raymond Tukpe Date: Tue, 28 Jun 2022 14:36:44 +0100 Subject: [PATCH 6/7] fix: use dereference slice (#808) --- datastore/mongo/subscription.go | 2 +- worker/task/process_event_creation.go | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/datastore/mongo/subscription.go b/datastore/mongo/subscription.go index b9cac49adb..2b81507134 100644 --- a/datastore/mongo/subscription.go +++ b/datastore/mongo/subscription.go @@ -141,7 +141,7 @@ func (s *subscriptionRepo) FindSubscriptionsBySourceIDs(ctx context.Context, gro var subscriptions []datastore.Subscription filter := bson.M{"group_id": groupId, "source_id": sourceId, "document_status": datastore.ActiveDocumentStatus} - err := s.store.FindMany(ctx, filter, nil, nil, 0, 0, subscriptions) + err := s.store.FindMany(ctx, filter, nil, nil, 0, 0, &subscriptions) if err != nil { return nil, err } diff --git a/worker/task/process_event_creation.go b/worker/task/process_event_creation.go index 87d5ee8933..471aa3ddd3 100644 --- a/worker/task/process_event_creation.go +++ b/worker/task/process_event_creation.go @@ -78,6 +78,7 @@ func ProcessEventCreated(appRepo datastore.ApplicationRepository, eventRepo data } else if group.Type == datastore.IncomingGroup { subscriptions, err = subRepo.FindSubscriptionsBySourceIDs(ctx, group.UID, event.SourceID) if err != nil { + log.Errorf("error fetching subscriptions for this source %s", err) return &EndpointError{Err: errors.New("error fetching subscriptions for this source"), delay: 10 * time.Second} } } From c482eda4a3c8b77aa57e96858e61b3746c2ad482 Mon Sep 17 00:00:00 2001 From: Pelumi Muyiwa-Oni Date: Tue, 28 Jun 2022 17:08:25 +0100 Subject: [PATCH 7/7] Ui Fixes (#809) * updated analytics modal * fix for creating organisation * fix fecth apps on create subscriptions oage * fix: update sources * updated create/update endpoint form * updated app details for incoming projects * updated create subscriptions for app portal * updated create projects form * removed console.log, updated project type --- .../add-analytics.component.html | 1 - .../create-app/create-app.component.html | 10 ---- .../create-app/create-app.component.ts | 4 +- .../create-project-component.component.html | 6 ++- .../create-project-component.component.ts | 46 ++++++++++++------- .../create-source/create-source.component.ts | 2 + .../create-subscription.component.ts | 9 ++-- .../app-details/app-details.component.html | 13 ++---- .../apps/app-details/app-details.component.ts | 4 ++ .../create-endpoint.component.html | 11 ----- .../create-endpoint.component.ts | 2 - .../src/app/private/private.component.html | 10 ++-- .../src/app/private/private.component.ts | 2 +- 13 files changed, 55 insertions(+), 65 deletions(-) diff --git a/web/ui/dashboard/src/app/private/components/add-analytics/add-analytics.component.html b/web/ui/dashboard/src/app/private/components/add-analytics/add-analytics.component.html index deefd06798..b8a5a89e0f 100644 --- a/web/ui/dashboard/src/app/private/components/add-analytics/add-analytics.component.html +++ b/web/ui/dashboard/src/app/private/components/add-analytics/add-analytics.component.html @@ -30,7 +30,6 @@
Would you like to enable analytics
-
diff --git a/web/ui/dashboard/src/app/private/components/create-app/create-app.component.html b/web/ui/dashboard/src/app/private/components/create-app/create-app.component.html index 41181d0ea0..97bd5986ed 100644 --- a/web/ui/dashboard/src/app/private/components/create-app/create-app.component.html +++ b/web/ui/dashboard/src/app/private/components/create-app/create-app.component.html @@ -73,16 +73,6 @@

Endpoint {{ i > 0 ? i + 1 : '' }}

Please provide an endpoint URL - -
- - -
- -
- - -
+
+ input error icon + Please select a project type +

@@ -111,7 +115,7 @@

required { - this.showSecretCopyText = false + this.showSecretCopyText = false; }, 3000); - + document.body.removeChild(el); } + + checkProjectConfig() { + const configDetails = this.projectForm.value.config; + const configKeys = Object.keys(configDetails).slice(0, -1); + configKeys.forEach(configKey => { + const configKeyValues = Object.values(configDetails[configKey]); + if (configKeyValues.every(item => item === null)) delete this.projectForm.value.config[configKey]; + if (configKey === 'strategy' && configDetails?.strategy?.duration && this.action !== 'update') { + let duration = configDetails.strategy.duration; + const [digits, word] = duration.match(/\D+|\d+/g); + word === 's' ? (duration = parseInt(digits) * 1000) : (duration = parseInt(digits) * 1000000); + this.projectForm.value.config.strategy.duration = duration; + } + }); + if (this.projectForm.value.config.disable_endpoint === null) delete this.projectForm.value.config.disable_endpoint; + } } diff --git a/web/ui/dashboard/src/app/private/components/create-source/create-source.component.ts b/web/ui/dashboard/src/app/private/components/create-source/create-source.component.ts index b9be65613e..c937db974f 100644 --- a/web/ui/dashboard/src/app/private/components/create-source/create-source.component.ts +++ b/web/ui/dashboard/src/app/private/components/create-source/create-source.component.ts @@ -58,7 +58,9 @@ export class CreateSourceComponent implements OnInit { async getSourceDetails() { try { const response = await this.createSourceService.getSourceDetails(this.sourceId); + const sourceProvider = response.data?.provider; this.sourceForm.patchValue(response.data); + if (sourceProvider === 'github') this.sourceForm.patchValue({ verifier: { type: 'github' } }); return; } catch (error) { return error; diff --git a/web/ui/dashboard/src/app/private/components/create-subscription/create-subscription.component.ts b/web/ui/dashboard/src/app/private/components/create-subscription/create-subscription.component.ts index d1adcb1fa4..c18af3921f 100644 --- a/web/ui/dashboard/src/app/private/components/create-subscription/create-subscription.component.ts +++ b/web/ui/dashboard/src/app/private/components/create-subscription/create-subscription.component.ts @@ -66,7 +66,7 @@ export class CreateSubscriptionComponent implements OnInit { try { const apps = await this.createSubscriptionService.getAppPortalApp(this.token); - this.subscriptionForm.patchValue({ app_id: apps.data.uid, group_id: apps.data.group_id }); + this.subscriptionForm.patchValue({ app_id: apps.data.uid, group_id: apps.data.group_id, type: 'outgoing' }); this.endPoints = apps.data.endpoints; this.isloadingAppPortalAppDetails = false; return; @@ -85,7 +85,7 @@ export class CreateSubscriptionComponent implements OnInit { this.subscriptionForm.patchValue({ source_id: response.data?.source_metadata?.uid, app_id: response.data?.app_metadata?.uid, endpoint_id: response.data?.endpoint_metadata?.uid }); if (!this.token) this.onUpdateAppSelection(); response.data.filter_config?.event_types ? (this.eventTags = response.data.filter_config?.event_types) : (this.eventTags = []); - + if (this.token) this.projectType = 'outgoing'; return; } catch (error) { return error; @@ -97,7 +97,6 @@ export class CreateSubscriptionComponent implements OnInit { await this.getAppPortalApp(); return; } - if (this.privateService.activeProjectDetails.type === 'incoming') return; try { const appsResponse = await this.privateService.getApps(); @@ -159,6 +158,7 @@ export class CreateSubscriptionComponent implements OnInit { ) { return this.subscriptionForm.markAllAsTouched(); } + if ( this.subscriptionForm.get('name')?.invalid || this.subscriptionForm.get('type')?.invalid || @@ -176,7 +176,6 @@ export class CreateSubscriptionComponent implements OnInit { delete subscription.retry_config; } this.isCreatingSubscription = true; - try { const response = this.action == 'update' @@ -192,7 +191,7 @@ export class CreateSubscriptionComponent implements OnInit { async onCreateNewApp(newApp: APP) { await this.getApps(); this.subscriptionForm.patchValue({ app_id: newApp.uid }); - this.onUpdateAppSelection() + this.onUpdateAppSelection(); } removeEventTag(tag: string) { diff --git a/web/ui/dashboard/src/app/private/pages/project/apps/app-details/app-details.component.html b/web/ui/dashboard/src/app/private/pages/project/apps/app-details/app-details.component.html index 8d750a9963..69ba7bb1ae 100644 --- a/web/ui/dashboard/src/app/private/pages/project/apps/app-details/app-details.component.html +++ b/web/ui/dashboard/src/app/private/pages/project/apps/app-details/app-details.component.html @@ -77,7 +77,7 @@

Overview

-->
-
+

App Event Endpoints

@@ -176,7 +176,7 @@
{{ endpoint.description }}<

No endpoint has been added for selected app yet

-
+

App Portal

    @@ -247,7 +247,7 @@

    {{ selectedEndpoint ? 'Update' : 'Add' }} Endpoint

    @@ -277,10 +277,5 @@

    Endpoint Secret

- +
diff --git a/web/ui/dashboard/src/app/private/pages/project/apps/app-details/app-details.component.ts b/web/ui/dashboard/src/app/private/pages/project/apps/app-details/app-details.component.ts index c2d2eba189..7b12185ea9 100644 --- a/web/ui/dashboard/src/app/private/pages/project/apps/app-details/app-details.component.ts +++ b/web/ui/dashboard/src/app/private/pages/project/apps/app-details/app-details.component.ts @@ -123,6 +123,10 @@ export class AppDetailsComponent implements OnInit { this.screenWidth > 1010 ? (this.shouldRenderSmallSize = false) : (this.shouldRenderSmallSize = true); } + closeEditEndpointModal(){ + this.showAddEndpointModal = false; + this.selectedEndpoint = undefined + } focusInput() { document.getElementById('tagInput')?.focus(); } diff --git a/web/ui/dashboard/src/app/private/pages/project/apps/app-details/create-endpoint/create-endpoint.component.html b/web/ui/dashboard/src/app/private/pages/project/apps/app-details/create-endpoint/create-endpoint.component.html index c5e62f7dce..60fe04eb33 100644 --- a/web/ui/dashboard/src/app/private/pages/project/apps/app-details/create-endpoint/create-endpoint.component.html +++ b/web/ui/dashboard/src/app/private/pages/project/apps/app-details/create-endpoint/create-endpoint.component.html @@ -30,17 +30,6 @@
-
-
- - -
-
- - -
-
-