Skip to content

massdriver-cloud/azure-event-hubs

Repository files navigation

Massdriver

azure-event-hubs

Release Contributors Forks Stargazers Issues MIT License

Event Hubs is a modern big data streaming platform and event ingestion service that can seamlessly integrate with other Azure and Microsoft services. The service can process millions of events per second with low latency. The data sent to an event hub (Event Hubs instance) can be transformed and stored by using any real-time analytics providers or batching or storage adapters.


Design

For detailed information, check out our Operator Guide for this bundle.

Usage

Our bundles aren't intended to be used locally, outside of testing. Instead, our bundles are designed to be configured, connected, deployed and monitored in the Massdriver platform.

What are Bundles?

Bundles are the basic building blocks of infrastructure, applications, and architectures in Massdriver. Read more here.

Bundle

Params

Form input parameters for configuring a bundle for deployment.

View

Properties

  • capture (object)
    • arvo_encoding (string): Specifies the encoding used for the capture. Must be one of: ['Avro', 'AvroDeflate']. Default: Avro.
    • capture_buildup (integer): The amount of data built up in your Event Hub before a capture operation occurs. Minimum of 10 MiB, maximum of 500 MiB. Minimum: 10. Maximum: 500. Default: 300.
    • capture_interval (integer): The time interval, in seconds, at which the capture to Azure Data Lake will happen. Minimum of 60, maximum of 900. Minimum: 60. Maximum: 900. Default: 300.
  • hub (object)
    • message_retention (integer): The number of days to retain the events for this Event Hubs, value should be 1 to 7 days. Minimum: 1. Maximum: 7. Default: 1.
    • partition_count (integer): Minimum: 1. Maximum: 32. Default: 1.
    • sku (string): Learn more about the different features and capabilities of each pricing tier here. Cannot be changed after deployment. Must be one of: ['Standard', 'Premium']. Default: Standard.
    • throughput_units (integer): The number of throughput units allocated for the Event Hubs. Minimum of 1, maximum of 40. Learn more here. Minimum: 1. Maximum: 40.
    • zone_redundant (boolean): Enable zone redundancy for the Event Hubs. Cannot be changed after deployment. Default: False.
  • monitoring (object)
    • mode (string): Enable and customize Function App metric alarms. Default: AUTOMATED.
      • One of
        • Automated
        • Custom
        • Disabled

Examples

{
    "__name": "Development",
    "hub": {
        "partition_count": 1,
        "sku": "Standard",
        "throughput_units": 1
    }
}
{
    "__name": "Production",
    "hub": {
        "message_retention": 7,
        "partition_count": 20,
        "sku": "Premium",
        "throughput_units": 10
    }
}

Connections

Connections from other bundles that this bundle depends on.

View

Properties

  • azure_service_principal (object): . Cannot contain additional properties.
    • data (object)

      • client_id (string): A valid UUID field.

        Examples:

        "123xyz99-ab34-56cd-e7f8-456abc1q2w3e"
      • client_secret (string)

      • subscription_id (string): A valid UUID field.

        Examples:

        "123xyz99-ab34-56cd-e7f8-456abc1q2w3e"
      • tenant_id (string): A valid UUID field.

        Examples:

        "123xyz99-ab34-56cd-e7f8-456abc1q2w3e"
    • specs (object)

  • azure_storage_account_data_lake (object): . Cannot contain additional properties.
    • data (object)
      • infrastructure (object)

        • ari (string): Azure Resource ID.

          Examples:

          "/subscriptions/12345678-1234-1234-abcd-1234567890ab/resourceGroups/resource-group-name/providers/Microsoft.Network/virtualNetworks/network-name"
        • endpoint (string): Azure Storage Account endpoint authentication. Cannot contain additional properties.

          Examples:

          "https://storageaccount.blob.core.windows.net/"
          "http://storageaccount.file.core.windows.net"
          "abfs://filesystem.accountname.dfs.core.windows.net/"
          "https://storageaccount.privatelink01.queue.core.windows.net/"
      • security (object): Azure Security Configuration. Cannot contain additional properties.

        • iam (object): IAM Roles And Scopes. Cannot contain additional properties.
          • ^[a-z]+[a-z_]*[a-z]$ (object)
            • role: Azure Role.

              Examples:

              "Storage Blob Data Reader"
            • scope (string): Azure IAM Scope.

    • specs (object)
      • azure (object): .
        • region (string): Select the Azure region you'd like to provision your resources in.

Artifacts

Resources created by this bundle that can be connected to other bundles.

View

Properties

  • azure_event_hubs (object): . Cannot contain additional properties.
    • data (object)
      • infrastructure (object)

        • ari (string): Azure Resource ID.

          Examples:

          "/subscriptions/12345678-1234-1234-abcd-1234567890ab/resourceGroups/resource-group-name/providers/Microsoft.Network/virtualNetworks/network-name"
        • endpoint (string): Azure Event Hubs endpoint authentication. Cannot contain additional properties.

          Examples:

          "sb://eventhub.servicebus.windows.net/"
      • security (object): Azure Security Configuration. Cannot contain additional properties.

        • iam (object): IAM Roles And Scopes. Cannot contain additional properties.
          • ^[a-z]+[a-z_]*[a-z]$ (object)
            • role: Azure Role.

              Examples:

              "Storage Blob Data Reader"
            • scope (string): Azure IAM Scope.

    • specs (object)
      • azure (object): .
        • region (string): Select the Azure region you'd like to provision your resources in.

Contributing

Bug Reports & Feature Requests

Did we miss something? Please submit an issue to report any bugs or request additional features.

Developing

Note: Massdriver bundles are intended to be tightly use-case scoped, intention-based, reusable pieces of IaC for use in the Massdriver platform. For this reason, major feature additions that broaden the scope of an existing bundle are likely to be rejected by the community.

Still want to get involved? First check out our contribution guidelines.

Fix or Fork

If your use-case isn't covered by this bundle, you can still get involved! Massdriver is designed to be an extensible platform. Fork this bundle, or create your own bundle from scratch!

Connect

Questions? Concerns? Adulations? We'd love to hear from you!

Please connect with us!

Email GitHub LinkedIn Twitter YouTube Reddit