Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/f4e sync event [PS-75] #1963

Merged
merged 8 commits into from
Apr 21, 2022
Merged

Feature/f4e sync event [PS-75] #1963

merged 8 commits into from
Apr 21, 2022

Conversation

MGibson1
Copy link
Member

Type of change

  • Bug fix
  • New feature development
  • Tech debt (refactoring, code cleanup, dependency upgrades, etc)
  • Build/deploy pipeline (DevOps)
  • Other

Objective

Log organization event when sponsorship sync occurs.

Code changes

  • Event/EventMessage/EventTableEntity/IEvent/: Updates models to take an installationId Note: this will necessitate updating EventsProcessor
  • EventType: Add sponsorship sync event type
  • CloudSyncCommand: I extracted the actual sync logic to a private method so we would be able to log an event for the no-sponsorships early return case as well
  • EventService: Log InstallationId for Organization events if available.

Tests

  • TestCaseHelper: This helper makes it easier to tests all combinations of options. The two helper methods yield:
    1. GetCombinations: All possible combinations of items in the given array, from none to all. Order agnostic
    2. GetCombinationsOfMultipleLists: All possible combinations of items assuming you should select exactly one option from each list.
  • TestCaseHelperTests: Tests of the above.

    Note: These tests are not run automatically as Test.Common is not a part of the main solution. I think this is OK, since these should be extremely static helper methods.

  • CommandSponsorshipTests/EventServiceTests: test functionality changes

Migrations

Migrations were generated for Dapper, MySql, and Postgres. Note, EF migrations apparently were not done for #1950

Testing requirements

Make sure an event is logged upon sponsorship sync.

Before you submit

  • I have checked for formatting errors (dotnet tool run dotnet-format --check) (required)
  • If making database changes - I have also updated Entity Framework queries and/or migrations
  • I have added unit tests where it makes sense to do so (encouraged but not required)
  • This change requires a documentation update (notify the documentation team)
  • This change has particular deployment requirements (notify the DevOps team)

justindbaur
justindbaur approved these changes Apr 21, 2022
View reviewed changes
test/Common/Helpers/TestCaseHelper.cs Outdated Show resolved Hide resolved
@MGibson1 MGibson1 merged commit 2f81a3b into feature/self-hosted-families-for-enterprise Apr 21, 2022
@MGibson1 MGibson1 deleted the feature/f4e-sync-event branch April 21, 2022 01:38
MGibson1 added a commit that referenced this pull request May 10, 2022
@MGibson1 @justindbaur
@jlf0dev @Manolachi @Hinton @sneakernuts @joseph-flinn @github-actions @eliykat @fedemkr @vincentsalucci @notnamed @kspearrin @cscharf @e271828-
Feature/self hosted families for enterprise (#1991)
c54c39b 
* Families for enterprise/split up organization sponsorship service (#1829)

* Split OrganizationSponsorshipService into commands

* Use tokenable for token validation

* Use interfaces to set up for DI

* Use commands over services

* Move service tests to command tests

* Value types can't be null

* Run dotnet format

* Update src/Core/OrganizationFeatures/OrganizationSponsorships/FamiliesForEnterprise/CancelSponsorshipCommand.cs

Co-authored-by: Justin Baur <admin@justinbaur.com>

* Fix controller tests

Co-authored-by: Justin Baur <admin@justinbaur.com>

* Families for enterprise/split up organization sponsorship service (#1875)

* Split OrganizationSponsorshipService into commands

* Use tokenable for token validation

* Use interfaces to set up for DI

* Use commands over services

* Move service tests to command tests

* Value types can't be null

* Run dotnet format

* Update src/Core/OrganizationFeatures/OrganizationSponsorships/FamiliesForEnterprise/CancelSponsorshipCommand.cs

Co-authored-by: Justin Baur <admin@justinbaur.com>

* Fix controller tests

* Split create and send sponsorships

* Split up create sponsorship

* Add self hosted commands to dependency injection

* Add field to store cloud billing sync key on self host instances

* Fix typo

* Fix data protector purpose of sponsorship offers

* Split cloud and selfhosted sponsorship offer tokenable

* Generate offer from self hosted with all necessary auth data

* Add Required properties to constructor

* Split up cancel sponsorship command

* Split revoke sponsorship command between cloud and self hosted

* Fix/f4e multiple sponsorships (#1838)

* Use sponosorship from validate to redeem

* Update tests

* Format

* Remove sponsorship service

* Run dotnet format

* Fix self hosted only controller attribute

* Clean up file structure and fixes

* Remove unneeded tokenables

* Remove obsolete commands

* Do not require file/class prefix if unnecessary

* Update Organizaiton sprocs

* Remove unnecessary models

* Fix tests

* Generalize LicenseService path calculation

Use async file read and deserialization

* Use interfaces for testability

* Remove unused usings

* Correct test direction

* Test license reading

* remove unused usings

* Format

Co-authored-by: Justin Baur <admin@justinbaur.com>

* Improve DataProtectorTokenFactory test coverage (#1884)

* Add encstring to server

* Test factory

Co-authored-by: Carlos Muentes <cmuentes@bitwarden.com>

* Format

* Remove SymmetricKeyProtectedString

Not needed

* Set ForcInvalid

Co-authored-by: Carlos Muentes <cmuentes@bitwarden.com>

* Feature/self f4e/api keys (#1896)

* Add in ApiKey

* Work on API Key table

* Work on apikey table

* Fix response model

* Work on information for UI

* Work on last sync date

* Work on sync status

* Work on auth

* Work on tokenable

* Work on merge

* Add custom requirement

* Add policy

* Run formatting

* Work on EF Migrations

* Work on OrganizationConnection

* Work on database

* Work on additional database table

* Run formatting

* Small fixes

* More cleanup

* Cleanup

* Add RevisionDate

* Add GO

* Finish Sql project

* Add newlines

* Fix stored proc file

* Fix sqlproj

* Add newlines

* Fix table

* Add navigation property

* Delete Connections when organization is deleted

* Add connection validation

* Start adding ID column

* Work on ID column

* Work on SQL migration

* Work on migrations

* Run formatting

* Fix test build

* Fix sprocs

* Work on migrations

* Fix Create table

* Fix sproc

* Add prints to migration

* Add default value

* Update EF migrations

* Formatting

* Add to integration tests

* Minor fixes

* Formatting

* Cleanup

* Address PR feedback

* Address more PR feedback

* Fix formatting

* Fix formatting

* Fix

* Address PR feedback

* Remove accidential change

* Fix SQL build

* Run formatting

* Address PR feedback

* Add sync data to OrganizationUserOrgDetails

* Add comments

* Remove OrganizationConnectionService interface

* Remove unused using

* Address PR feedback

* Formatting

* Minor fix

* Feature/self f4e/update db (#1930)

* Fix migration

* Fix TimesRenewed

* Add comments

* Make two properties non-nullable

* Remove need for SponsoredOrg on SH (#1934)

* Remove need for SponsoredOrg on SH

* Add Family prefix

* Add check for enterprise org on BillingSync key (#1936)

* [PS-10] Feature/sponsorships removed at end of term (#1938)

* Rename commands to min unique names

* Inject revoke command based on self hosting

* WIP: Remove/Revoke marks to delete

* Complete WIP

* Improve remove/revoke tests

* PR review

* Fail validation if sponsorship has failed to sync for 6 months

* Feature/do not accept old self host sponsorships (#1939)

* Do not accept >6mo old self-hosted sponsorships

* Give disabled grace period of 3 months

* Fix issues of Sql.proj differing from migration outcome (#1942)

* Fix issues of Sql.proj differing from migration outcome

* Yoink int tests

* Add missing assert helpers

* Feature/org sponsorship sync (#1922)

* Self-hosted side sync first pass

TODO:
* flush out org sponsorship model
* implement cloud side
* process cloud-side response and update self-hosted records

* sync scaffolding second pass

* remove list of Org User ids from sync and begin work on SelfHostedRevokeSponsorship

* allow authenticated http calls from server to return a result

* update models

* add logic for sync and change offer email template

* add billing sync key and hide CreateSponsorship without user

* fix tests

* add job scheduling

* add authorize attributes to endpoints

* separate models into data/model and request/response

* batch sync more, add EnableCloudCommunication for testing

* send emails in bulk

* make userId and sponsorshipType non nullable

* batch more on self hosted side of sync

* remove TODOs and formatting

* changed logic of cloud sync

* let BaseIdentityClientService handle all logging

* call sync from scheduled job on self host

* create bulk db operations for OrganizationSponsorships

* remove SponsoredOrgId from sync, return default from server http call

* validate BillingSyncKey during sync

revert changes to CreateSponsorshipCommand

* revert changes to ICreateSponsorshipCommand

* add some tests

* add DeleteExpiredSponsorshipsJob

* add cloud sync test

* remove extra method

* formatting

* prevent new sponsorships from disabled orgs

* update packages

* - pulled out send sponsorship command dependency from sync on cloud
- don't throw error when sponsorships are empty
- formatting

* formatting models

* more formatting

* remove licensingService dependency from selfhosted sync

* use installation urls and formatting

* create constructor for RequestModel and formatting

* add date parameter to OrganizationSponsorship_DeleteExpired

* add new migration

* formatting

* rename OrganizationCreateSponsorshipRequestModel to OrganizationSponsorshipCreateRequestModel

* prevent whole sync from failing if one sponsorship type is unsupported

* deserialize config and billingsynckey from org connection

* alter log message when sync disabled

* Add grace period to disabled orgs

* return early on self hosted if there are no sponsorships in database

* rename BillingSyncConfig

* send sponsorship offers from controller

* allow config to be a null object

* better exception handling in sync scheduler

* add ef migrations

* formatting

* fix tests

* fix validate test

Co-authored-by: Matt Gibson <mgibson@bitwarden.com>

* Fix OrganizationApiKey issues (#1941)

Co-authored-by: Justin Baur <admin@justinbaur.com>

* Feature/org sponsorship self hosted tests (#1947)

* Self-hosted side sync first pass

TODO:
* flush out org sponsorship model
* implement cloud side
* process cloud-side response and update self-hosted records

* sync scaffolding second pass

* remove list of Org User ids from sync and begin work on SelfHostedRevokeSponsorship

* allow authenticated http calls from server to return a result

* update models

* add logic for sync and change offer email template

* add billing sync key and hide CreateSponsorship without user

* fix tests

* add job scheduling

* add authorize attributes to endpoints

* separate models into data/model and request/response

* batch sync more, add EnableCloudCommunication for testing

* send emails in bulk

* make userId and sponsorshipType non nullable

* batch more on self hosted side of sync

* remove TODOs and formatting

* changed logic of cloud sync

* let BaseIdentityClientService handle all logging

* call sync from scheduled job on self host

* create bulk db operations for OrganizationSponsorships

* remove SponsoredOrgId from sync, return default from server http call

* validate BillingSyncKey during sync

revert changes to CreateSponsorshipCommand

* revert changes to ICreateSponsorshipCommand

* add some tests

* add DeleteExpiredSponsorshipsJob

* add cloud sync test

* remove extra method

* formatting

* prevent new sponsorships from disabled orgs

* update packages

* - pulled out send sponsorship command dependency from sync on cloud
- don't throw error when sponsorships are empty
- formatting

* formatting models

* more formatting

* remove licensingService dependency from selfhosted sync

* use installation urls and formatting

* create constructor for RequestModel and formatting

* add date parameter to OrganizationSponsorship_DeleteExpired

* add new migration

* formatting

* rename OrganizationCreateSponsorshipRequestModel to OrganizationSponsorshipCreateRequestModel

* prevent whole sync from failing if one sponsorship type is unsupported

* deserialize config and billingsynckey from org connection

* add mockHttp nuget package and use httpclientfactory

* fix current tests

* WIP of creating tests

* WIP of new self hosted tests

* WIP self hosted tests

* finish self hosted tests

* formatting

* format of interface

* remove extra config file

* added newlines

Co-authored-by: Matt Gibson <mgibson@bitwarden.com>

* Fix Organization_DeleteById (#1950)

* Fix Organization_Delete

* Fix L

* [PS-4] block enterprise user from sponsoring itself (#1943)

* [PS-248] Feature/add connections enabled endpoint (#1953)

* Move Organization models to sub namespaces

* Add Organization Connection api endpoints

* Get all connections rather than just enabled ones

* Add missing services to DI

* pluralize private api endpoints

* Add type protection to org connection request/response

* Fix route

* Use nullable Id to signify no connection

* Test Get Connections enabled

* Fix data discoverer

* Also drop this sproc for rerunning

* Id is the OUTPUT of create sprocs

* Fix connection config parsing

* Linter fixes

* update sqlproj file name

* Use param xdocs on methods

* Simplify controller path attribute

* Use JsonDocument to avoid escaped json in our response/request strings

* Fix JsonDoc tests

* Linter fixes

* Fix ApiKey Command and add tests (#1949)

* Fix ApiKey command

* Formatting

* Fix test failures introduced in #1943 (#1957)

* Remove "Did you know?" copy from emails. (#1962)

* Remove "Did you know"

* Remove jsonIf helper

* Feature/fix send single sponsorship offer email (#1956)

* Fix sponsorship offer email

* Do not sanitize org name

* PR feedback

* Feature/f4e sync event [PS-75] (#1963)

* Create sponsorship sync event type

* Add InstallationId to Event model

* Add combinatorics-based test case generators

* Log sponsorships sync event on sync

* Linter and test fixes

* Fix failing test

* Migrate sprocs and view

* Remove unused `using`s

* [PS-190] Add manual sync trigger in self hosted (#1955)

* WIP add button to admin project for billing sync

* add connection table to view page

* minor fixes for self hosted side of sync

* fixes number of bugs for cloud side of sync

* deserialize before returning for some reason

* add json attributes to return models

* list of sponsorships parameter is immutable, add secondary list

* change sproc name

* add error handling

* Fix tests

* modify call to connection

* Update src/Admin/Controllers/OrganizationsController.cs

Co-authored-by: Matt Gibson <mgibson@bitwarden.com>

* undo change to sproc name

* simplify logic

* Update src/Core/OrganizationFeatures/OrganizationSponsorships/FamiliesForEnterprise/Cloud/CloudSyncSponsorshipsCommand.cs

Co-authored-by: Matt Gibson <mgibson@bitwarden.com>

* register services despite if self hosted or cloud

* remove json properties

* revert merge conflict

Co-authored-by: Matt Gibson <mgibson@bitwarden.com>

* Update OrganizationSponsorship valid until when updating org expirati… (#1966)

* Update OrganizationSponsorship valid until when updating org expiration date

* Linter fixes

* [PS-7] change revert email copy and add ValidUntil to sponsorship (#1965)

* change revert email copy and add ValidUntil to sponsorship

* add 15 days if no ValidUntil

* Chore/merge/self hosted families for enterprise (#1972)

* Log swallowed HttpRequestExceptions (#1866)

Co-authored-by: Hinton <oscar@oscarhinton.com>

* Allow for utilization of  readonly db connection (#1937)

* Bump the pin of the download-artifacts action to bypass the broken GitHub api (#1952)

* Bumped version to 1.48.0 (#1958)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* [EC-160] Give Provider Users access to all org ciphers and collections (#1959)

* Bumped version to 1.48.1 (#1961)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Avoid sending "user need confirmation" emails when there are no org admins (#1960)

* Remove noncompliant users for new policies (#1951)

* [PS-284] Allow installation clients to not need a user. (#1968)

* Allow installation clients to not need a user.

* Run formatting

Co-authored-by: Andrei <30410186+Manolachi@users.noreply.github.com>
Co-authored-by: Hinton <oscar@oscarhinton.com>
Co-authored-by: sneakernuts <671942+sneakernuts@users.noreply.github.com>
Co-authored-by: Joseph Flinn <58369717+joseph-flinn@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Thomas Rittson <31796059+eliykat@users.noreply.github.com>
Co-authored-by: Justin Baur <136baur@gmail.com>

* Fix/license file not found (#1974)

* Handle null license

* Throw hint message if license is not found by the admin project.

* Use CloudOrganizationId from Connection config

* Change test to support change

* Fix test

Co-authored-by: Matt Gibson <mgibson@bitwarden.com>

* Feature/f4e selfhosted rename migration to .sql (#1971)

* rename migration to .sql

* format

* Add unit tests to self host F4E (#1975)

* Work on tests

* Added more tests

* Run linting

* Address PR feedback

* Fix AssertRecent

* Linting

* Fixed empty tests

* Fix/misc self hosted f4e (#1973)

* Allow setting of ApiUri

* Return updates sponsorshipsData objects

* Bind arguments by name

* Greedy load sponsorships to email.

When upsert was called, it creates Ids on _all_ records, which meant
that the lazy-evaluation from this call always returned an empty list.

* add scope for sync command DI in job. simplify error logic

* update the sync job to get CloudOrgId from the BillingSyncKey

Co-authored-by: Jacob Fink <jfink@bitwarden.com>

* Chore/merge/self hosted families for enterprise (#1987)

* Log swallowed HttpRequestExceptions (#1866)

Co-authored-by: Hinton <oscar@oscarhinton.com>

* Allow for utilization of  readonly db connection (#1937)

* Bump the pin of the download-artifacts action to bypass the broken GitHub api (#1952)

* Bumped version to 1.48.0 (#1958)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* [EC-160] Give Provider Users access to all org ciphers and collections (#1959)

* Bumped version to 1.48.1 (#1961)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Avoid sending "user need confirmation" emails when there are no org admins (#1960)

* Remove noncompliant users for new policies (#1951)

* [PS-284] Allow installation clients to not need a user. (#1968)

* Allow installation clients to not need a user.

* Run formatting

* Use accept flow for sponsorship offers (#1964)

* PS-82 check send 2FA email for new devices on TwoFactorController send-email-login (#1977)

* [Bug] Skip WebAuthn 2fa event logs during login flow (#1978)

* [Bug] Supress WebAuthn 2fa event logs during login process

* Formatting

* Simplified method call with new paramter input

* Update RealIps Description (#1980)

Describe the syntax of the real_ips configuration key with an example, to prevent type errors in the `setup` container when parsing `config.yml`

* add proper URI validation to duo host (#1984)

* captcha scores (#1967)

* captcha scores

* some api fixes

* check bot on captcha attribute

* Update src/Core/Services/Implementations/HCaptchaValidationService.cs

Co-authored-by: e271828- <e271828-@users.noreply.github.com>

Co-authored-by: Chad Scharf <3904944+cscharf@users.noreply.github.com>
Co-authored-by: e271828- <e271828-@users.noreply.github.com>

* ensure no path specific in duo host (#1985)

Co-authored-by: Andrei <30410186+Manolachi@users.noreply.github.com>
Co-authored-by: Hinton <oscar@oscarhinton.com>
Co-authored-by: sneakernuts <671942+sneakernuts@users.noreply.github.com>
Co-authored-by: Joseph Flinn <58369717+joseph-flinn@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Thomas Rittson <31796059+eliykat@users.noreply.github.com>
Co-authored-by: Justin Baur <136baur@gmail.com>
Co-authored-by: Federico Maccaroni <fedemkr@gmail.com>
Co-authored-by: Vincent Salucci <26154748+vincentsalucci@users.noreply.github.com>
Co-authored-by: Jordan Cooks <notnamed@users.noreply.github.com>
Co-authored-by: Kyle Spearrin <kspearrin@users.noreply.github.com>
Co-authored-by: Chad Scharf <3904944+cscharf@users.noreply.github.com>
Co-authored-by: e271828- <e271828-@users.noreply.github.com>

* Address feedback (#1990)

Co-authored-by: Justin Baur <admin@justinbaur.com>
Co-authored-by: Carlos Muentes <cmuentes@bitwarden.com>
Co-authored-by: Jake Fink <jfink@bitwarden.com>
Co-authored-by: Justin Baur <136baur@gmail.com>
Co-authored-by: Andrei <30410186+Manolachi@users.noreply.github.com>
Co-authored-by: Hinton <oscar@oscarhinton.com>
Co-authored-by: sneakernuts <671942+sneakernuts@users.noreply.github.com>
Co-authored-by: Joseph Flinn <58369717+joseph-flinn@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Thomas Rittson <31796059+eliykat@users.noreply.github.com>
Co-authored-by: Federico Maccaroni <fedemkr@gmail.com>
Co-authored-by: Vincent Salucci <26154748+vincentsalucci@users.noreply.github.com>
Co-authored-by: Jordan Cooks <notnamed@users.noreply.github.com>
Co-authored-by: Kyle Spearrin <kspearrin@users.noreply.github.com>
Co-authored-by: Chad Scharf <3904944+cscharf@users.noreply.github.com>
Co-authored-by: e271828- <e271828-@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@justindbaur justindbaur justindbaur approved these changes

@jlf0dev jlf0dev Awaiting requested review from jlf0dev

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@MGibson1 @justindbaur