Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cosmos DB converter for SDK-type support and samples #1406

Merged

Conversation

liliankasem
Copy link
Member

@liliankasem liliankasem commented Mar 8, 2023

Issue describing the changes in this PR

resolves #1371

old PR for reference #1109

Implementation of a Cosmos DB converter to support using SDK-type bindings and POCOs via the deferring binding model. Currently, we are only supporting input bindings as the use of deferred binding with Cosmos triggers is not possible (see discussion) and output bindings are out of scope.

In this PR I have:

  • Implemented a converter that supports the core scenarios defined in our public documentation to enable parity with in-proc
  • Implemented options to enable managed identity support
  • Added unit tests for the converter

Not cosmos related:

  • Updated blob extension to so that Constants.cs has a more specific namespace (without it causes conflicts in the test project)

Pull request checklist

  • My changes do not require documentation changes
    • Otherwise: Documentation issue linked to PR
  • My changes should not be added to the release notes for the next release
    • Otherwise: I've added my notes to release_notes.md
  • My changes do not need to be backported to a previous version
    • Otherwise: Backport tracked by issue/PR #issue_or_pr
  • I have added all required tests (Unit tests, E2E tests)

@liliankasem liliankasem changed the base branch from main to feature/sdk-type-binding March 8, 2023 18:52
@liliankasem liliankasem force-pushed the sdk-bindings/cosmos-sdk-converter branch from 3fef731 to 9ed588d Compare March 21, 2023 23:15
@liliankasem liliankasem merged commit fae3f6b into feature/sdk-type-binding Mar 21, 2023
@liliankasem liliankasem deleted the sdk-bindings/cosmos-sdk-converter branch March 21, 2023 23:55
aishwaryabh added a commit that referenced this pull request May 25, 2023
* Add E2E tests for blob SDK type bindings (#1360)

* Add analyzer for SupportsDeferredBindingAttribute (#1367)

* Update SupportsDeferredBinding diagnostic code & update docs (#1377)

* Revert "Remove types, tests & logic related to the SDK-binding feature (#1374)"

This reverts commit eac5b19.

* Added unit tests for BlobStorageConverter (#1370)

* Adding test project
* Added tests

* Initial changes

* Updating metadata

* Adding a test

* Code refactoring

* Cosmos DB converter for SDK-type support and samples (#1406)

* Execution flow added

* Update pipeline variables to include tags (#1427)

* Update extension variables to not use build number with tags (#1429)

* Execution flow

* cleanup

* Cleanup

* Cleanup

* Cleanup execution flow

* Added test

* Adding tests

* code cleanup

* Code cleanup

* code refactor

* Added logic to check type before sending to converter

* Updated tests

* Removing advertised converters from options

* Adding options back

* Changes to check type

* updating type check logic

* correction in metadata generation

* code cleanup

* Test cleanup

* Added changes to support poco

* Metadata generation changes

* Changes on Invocation side

* correcting type check and tests

* Test for poco invocation support

* Added tests for poco invocation flows

* Metadata generator minor update

* Grpc definition update

* Added poco and poco collection check

* cleanup metadata generation

* Adding support for type collection

* code cleanup

* Fixing metadata gen

* Test fix

* Grpc definition fix

* Removing BlobContainerClient advertisement

* Readding containerclient

* Updated metadata generator

* Changes as per latest design

* code cleanup

* updatin tests input

* code cleanup

* Cleanup Metadata generation

* correcting converters fallback logic

* Removing converter advertisement on cosmos trigger

* Fixing failing test

* test check in

* Undo testing changes in sample app

* spacing fix

* updated tests

* non secret changes

* Addressing PR feedback

* e2e test work

* adding comments

* Removed json deserialization attribute

* Rename to AllowConverterFallback

* Addressing feedback

* Metada generation cleanup

* Test cleanup

* Addressing feedback

* Updated grpcFunctionDefinition for multiple inputConverterAttribute

* Removing cosmos converter registration

* Address PR feedback

* Address PR feedback

* Test fix

* Minor

* addressing comments

* getting e2e working

* Implement bypass deferred binding (#1462)

Implement bypass deferred binding

* Build issue fix

* comments

* Update test/E2ETests/E2EApps/E2EApp/Table/TableInputBindingFunctions.cs

Co-authored-by: Lilian Kasem <likasem@microsoft.com>

* Adding check for PR in yml file (#1546)

* addressing some more comments

* dependency

* one more dependency

* trying to compile lol

* seeing if tables project compiles with dependency

* removing local instance

* see if this compiles

* trying to remove test

* nvm adding it back

* adding keys to nuget.config

* trying to get feed

* trying again

* more yml

* fixing spacing

* ci.yml

* Add ability to bind to SBReceivedMessage (#1313)

* another change

* trying again

* formatting changes

* changes

* updating worker.sdk package

* fixing unit test

* getting unit tests to pass

* fixing test

---------

Co-authored-by: Lilian Kasem <likasem@microsoft.com>
Co-authored-by: Surgupta <surgupta@microsoft.com>
Co-authored-by: JoshLove-msft <54595583+JoshLove-msft@users.noreply.github.com>
liliankasem added a commit that referenced this pull request Jun 21, 2023
* Add E2E tests for blob SDK type bindings (#1360)

* Add analyzer for SupportsDeferredBindingAttribute (#1367)

* Update SupportsDeferredBinding diagnostic code & update docs (#1377)

* Revert "Remove types, tests & logic related to the SDK-binding feature (#1374)"

This reverts commit eac5b19.

* Added unit tests for BlobStorageConverter (#1370)

* Adding test project
* Added tests

* Initial changes

* Updating metadata

* Adding a test

* Code refactoring

* Cosmos DB converter for SDK-type support and samples (#1406)

* Execution flow added

* Update pipeline variables to include tags (#1427)

* Update extension variables to not use build number with tags (#1429)

* Execution flow

* cleanup

* Cleanup

* Cleanup

* Cleanup execution flow

* Added test

* Adding tests

* code cleanup

* Code cleanup

* code refactor

* Added logic to check type before sending to converter

* Updated tests

* Removing advertised converters from options

* Adding options back

* Changes to check type

* updating type check logic

* correction in metadata generation

* code cleanup

* Test cleanup

* Added changes to support poco

* Metadata generation changes

* Changes on Invocation side

* correcting type check and tests

* Test for poco invocation support

* Added tests for poco invocation flows

* Metadata generator minor update

* Grpc definition update

* Added poco and poco collection check

* cleanup metadata generation

* Adding support for type collection

* code cleanup

* Fixing metadata gen

* Test fix

* Grpc definition fix

* Removing BlobContainerClient advertisement

* Readding containerclient

* Updated metadata generator

* Changes as per latest design

* code cleanup

* updatin tests input

* code cleanup

* Cleanup Metadata generation

* correcting converters fallback logic

* Removing converter advertisement on cosmos trigger

* Fixing failing test

* test check in

* Undo testing changes in sample app

* spacing fix

* updated tests

* non secret changes

* Addressing PR feedback

* e2e test work

* adding comments

* Removed json deserialization attribute

* Rename to AllowConverterFallback

* Addressing feedback

* Metada generation cleanup

* Test cleanup

* Addressing feedback

* Updated grpcFunctionDefinition for multiple inputConverterAttribute

* Removing cosmos converter registration

* Address PR feedback

* Address PR feedback

* Test fix

* Minor

* addressing comments

* getting e2e working

* Implement bypass deferred binding (#1462)

Implement bypass deferred binding

* Build issue fix

* comments

* Update test/E2ETests/E2EApps/E2EApp/Table/TableInputBindingFunctions.cs

Co-authored-by: Lilian Kasem <likasem@microsoft.com>

* Adding check for PR in yml file (#1546)

* addressing some more comments

* dependency

* one more dependency

* trying to compile lol

* seeing if tables project compiles with dependency

* removing local instance

* see if this compiles

* trying to remove test

* nvm adding it back

* adding keys to nuget.config

* trying to get feed

* trying again

* more yml

* fixing spacing

* ci.yml

* Add ability to bind to SBReceivedMessage (#1313)

* another change

* trying again

* formatting changes

* changes

* updating worker.sdk package

* fixing unit test

* getting unit tests to pass

* fixing test

---------

Co-authored-by: Lilian Kasem <likasem@microsoft.com>
Co-authored-by: Surgupta <surgupta@microsoft.com>
Co-authored-by: JoshLove-msft <54595583+JoshLove-msft@users.noreply.github.com>
liliankasem added a commit that referenced this pull request Jun 22, 2023
* Add E2E tests for blob SDK type bindings (#1360)

* Add analyzer for SupportsDeferredBindingAttribute (#1367)

* Update SupportsDeferredBinding diagnostic code & update docs (#1377)

* Revert "Remove types, tests & logic related to the SDK-binding feature (#1374)"

This reverts commit eac5b19.

* Added unit tests for BlobStorageConverter (#1370)

* Adding test project
* Added tests

* Initial changes

* Updating metadata

* Adding a test

* Code refactoring

* Cosmos DB converter for SDK-type support and samples (#1406)

* Execution flow added

* Update pipeline variables to include tags (#1427)

* Update extension variables to not use build number with tags (#1429)

* Execution flow

* cleanup

* Cleanup

* Cleanup

* Cleanup execution flow

* Added test

* Adding tests

* code cleanup

* Code cleanup

* code refactor

* Added logic to check type before sending to converter

* Updated tests

* Removing advertised converters from options

* Adding options back

* Changes to check type

* updating type check logic

* correction in metadata generation

* code cleanup

* Test cleanup

* Added changes to support poco

* Metadata generation changes

* Changes on Invocation side

* correcting type check and tests

* Test for poco invocation support

* Added tests for poco invocation flows

* Metadata generator minor update

* Grpc definition update

* Added poco and poco collection check

* cleanup metadata generation

* Adding support for type collection

* code cleanup

* Fixing metadata gen

* Test fix

* Grpc definition fix

* Removing BlobContainerClient advertisement

* Readding containerclient

* Updated metadata generator

* Changes as per latest design

* code cleanup

* updatin tests input

* code cleanup

* Cleanup Metadata generation

* correcting converters fallback logic

* Removing converter advertisement on cosmos trigger

* Fixing failing test

* test check in

* Undo testing changes in sample app

* spacing fix

* updated tests

* non secret changes

* Addressing PR feedback

* e2e test work

* adding comments

* Removed json deserialization attribute

* Rename to AllowConverterFallback

* Addressing feedback

* Metada generation cleanup

* Test cleanup

* Addressing feedback

* Updated grpcFunctionDefinition for multiple inputConverterAttribute

* Removing cosmos converter registration

* Address PR feedback

* Address PR feedback

* Test fix

* Minor

* addressing comments

* getting e2e working

* Implement bypass deferred binding (#1462)

Implement bypass deferred binding

* Build issue fix

* comments

* Update test/E2ETests/E2EApps/E2EApp/Table/TableInputBindingFunctions.cs

Co-authored-by: Lilian Kasem <likasem@microsoft.com>

* Adding check for PR in yml file (#1546)

* addressing some more comments

* dependency

* one more dependency

* trying to compile lol

* seeing if tables project compiles with dependency

* removing local instance

* see if this compiles

* trying to remove test

* nvm adding it back

* adding keys to nuget.config

* trying to get feed

* trying again

* more yml

* fixing spacing

* ci.yml

* Add ability to bind to SBReceivedMessage (#1313)

* another change

* trying again

* formatting changes

* changes

* updating worker.sdk package

* fixing unit test

* getting unit tests to pass

* fixing test

---------

Co-authored-by: Lilian Kasem <likasem@microsoft.com>
Co-authored-by: Surgupta <surgupta@microsoft.com>
Co-authored-by: JoshLove-msft <54595583+JoshLove-msft@users.noreply.github.com>
JoshLove-msft pushed a commit to JoshLove-msft/azure-functions-dotnet-worker that referenced this pull request Oct 4, 2023
JoshLove-msft added a commit to JoshLove-msft/azure-functions-dotnet-worker that referenced this pull request Oct 4, 2023
* Add E2E tests for blob SDK type bindings (Azure#1360)

* Add analyzer for SupportsDeferredBindingAttribute (Azure#1367)

* Update SupportsDeferredBinding diagnostic code & update docs (Azure#1377)

* Revert "Remove types, tests & logic related to the SDK-binding feature (Azure#1374)"

This reverts commit eac5b19.

* Added unit tests for BlobStorageConverter (Azure#1370)

* Adding test project
* Added tests

* Initial changes

* Updating metadata

* Adding a test

* Code refactoring

* Cosmos DB converter for SDK-type support and samples (Azure#1406)

* Execution flow added

* Update pipeline variables to include tags (Azure#1427)

* Update extension variables to not use build number with tags (Azure#1429)

* Execution flow

* cleanup

* Cleanup

* Cleanup

* Cleanup execution flow

* Added test

* Adding tests

* code cleanup

* Code cleanup

* code refactor

* Added logic to check type before sending to converter

* Updated tests

* Removing advertised converters from options

* Adding options back

* Changes to check type

* updating type check logic

* correction in metadata generation

* code cleanup

* Test cleanup

* Added changes to support poco

* Metadata generation changes

* Changes on Invocation side

* correcting type check and tests

* Test for poco invocation support

* Added tests for poco invocation flows

* Metadata generator minor update

* Grpc definition update

* Added poco and poco collection check

* cleanup metadata generation

* Adding support for type collection

* code cleanup

* Fixing metadata gen

* Test fix

* Grpc definition fix

* Removing BlobContainerClient advertisement

* Readding containerclient

* Updated metadata generator

* Changes as per latest design

* code cleanup

* updatin tests input

* code cleanup

* Cleanup Metadata generation

* correcting converters fallback logic

* Removing converter advertisement on cosmos trigger

* Fixing failing test

* test check in

* Undo testing changes in sample app

* spacing fix

* updated tests

* non secret changes

* Addressing PR feedback

* e2e test work

* adding comments

* Removed json deserialization attribute

* Rename to AllowConverterFallback

* Addressing feedback

* Metada generation cleanup

* Test cleanup

* Addressing feedback

* Updated grpcFunctionDefinition for multiple inputConverterAttribute

* Removing cosmos converter registration

* Address PR feedback

* Address PR feedback

* Test fix

* Minor

* addressing comments

* getting e2e working

* Implement bypass deferred binding (Azure#1462)

Implement bypass deferred binding

* Build issue fix

* comments

* Update test/E2ETests/E2EApps/E2EApp/Table/TableInputBindingFunctions.cs

Co-authored-by: Lilian Kasem <likasem@microsoft.com>

* Adding check for PR in yml file (Azure#1546)

* addressing some more comments

* dependency

* one more dependency

* trying to compile lol

* seeing if tables project compiles with dependency

* removing local instance

* see if this compiles

* trying to remove test

* nvm adding it back

* adding keys to nuget.config

* trying to get feed

* trying again

* more yml

* fixing spacing

* ci.yml

* Add ability to bind to SBReceivedMessage (Azure#1313)

* another change

* trying again

* formatting changes

* changes

* updating worker.sdk package

* fixing unit test

* getting unit tests to pass

* fixing test

---------

Co-authored-by: Lilian Kasem <likasem@microsoft.com>
Co-authored-by: Surgupta <surgupta@microsoft.com>
Co-authored-by: JoshLove-msft <54595583+JoshLove-msft@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[SDK Binding - Cosmos] Finish implementation of Cosmos Converter
5 participants