Welcome to the Datadog Service Catalog Metadata Provider!

The Datadog Service Catalog is a marvelous new way to track which services are in production. Your telemetry data is presented in the Service Catalog in a way where you can see the health of your services, and the health of the services that depend on them, all in one place. Most importantly, the Service Catalog helps you support your services in any environment.

With the Service Catalog registration API, you can supply the Service Catalog with all of the support information you might want:

• The name of the service
• The name of the application to which the service belongs
• The team which supports and maintains the service
• The email address of the team
• The URL of the service's documentation, which has a special subtype for runbooks (we all need more runbooks)
• The URL of the service's source code
• The URL of the service's dashboards and performance metrics
• Integration information for important services like Slack, MS Teams, PagerDuty, and Opsgenie

You can use this to register your services with the Service Catalog, and then use the Service Catalog to find the information you need to support your services.

Supporting services can be tricky, and the Datadog Service Catalog can make it easier for team members who aren't familiar with your service to support it. The Service Catalog can also help you find the information you need to support your services, and help you find the right people to support your services.

NOTE ABOUT VERSIONING: This GitHub Action uses the Service Catalog Definition APIs at version 2. Datadog is always working to improve their APIs, and this Action will be updated to support the newer versions of the API as they become available.

At this time, this GitHub Action supports the following versions of the Service Catalog schema:

• v2
• v2.1
• v2.2

Datadog already has methods for supplying this information. Why do we need another one? The answer is pretty simple: constraints.

Datadog has a super useful GitHub integration which allows you to register your service with a simple JSON file in your repository, after you install their GitHub plugin and give it read access to your repository. This is great, but if you're using GitOps for deployments and such, sometimes more integrations and webhooks can be concerning.

In order to get the benefits of the Service Catalog without having to open GitHub up to those integrations, this GitHub Action will allow you to register your services with the Service Catalog using a GitHub Action, which then allows you to have full control and visibility over this process. It also gives you full control over when this information is sent to Datadog.

Another reason to use this Action is that it permits your organization to establish controls on what must be present in your service metadata. For example, you might require that all services running in production have a runbook, or that all services have a specific tag which says where they're run. This Action gives you a way to add some controls without having to chase down every service owner and ask them to add the information. For more information on this functionality, see the section on Organization Controls below.

The registration API's links are below, and it takes input per its own JSON schema (also below), but this Action seeks to make things simpler and easier still! There are a lot of fields to choose from across the supported versions, we're going to group them into: Action Fields, Schema Fields, and Convenience Fields.

• Action Fields are fields that are specific to this GitHub Action. They are not part of the Service Catalog API, but they are used by this Action to make things easier for you.
• Schema Fields are fields that are part of the Service Catalog API. They are not specific to this GitHub Action, but they are used by this Action to make things easier for you. These may vary by schema-version.
• Convenience Fields are fields that are not part of the Service Catalog API, but they are used by this Action to make things easier for you. With these fields you need not worry about schema version, this action will map them to supported fields for you.

All of these fields are used for the functioning of this Action.

These fields directly map to the Service Catalog API. Please see the Service Catalog API Documentation for more information.

Using a field which is not supported in the schema version you've selected will result in an error.

Here I have a simple example of a service that has a single repository, and the least amount of metadata possible.

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: arcxp/datadog-service-catalog-metadata-provider@v2
        with:
          schema-version: v2.1
          datadog-hostname: api.us5.datadoghq.com
          datadog-key: ${{ secrets.DATADOG_KEY }}
          datadog-app-key: ${{ secrets.DATADOG_APP_KEY }}
          service-name: my-service
          team: my-team
          email: my-team@my-organization-which-totally-exists.com

In this example, you can see that my team is called "Global Greetings," and I have a single repository for my service. I also have a single document, and a single link.

My Slack support channel is listed by URL, and the email address is present. At the bottom you can also see the OpsGenie integration details.

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: arcxp/datadog-service-catalog-metadata-provider@v2
        with:
          schema-version: v2
          datadog-hostname: api.us5.datadoghq.com
          datadog-key: ${{ secrets.DATADOG_API_KEY }}
          datadog-app-key: ${{ secrets.DATADOG_APPLICATION_KEY }}
          service-name: hello-world
          team: Global Greetings
          email: global.greetings@fake-email-host.com
          slack-support-channel: 'https://team-name-here.slack.com/archives/ABC123'
          repos: |
            - name: hello-world
              url: https://github.com/fake-org/hello-world
              provider: github
          docs: |
            - name: outage-runbook
              url: https://fake-org.github.io/hello-world-outage-runbook
              provider: github
          integrations: |
            opsgenie:
              service-url: https://fake-org.hello-world.opsgenie.com
              region: US
          contacts: |
            - name: Product Owner
              type: email
              contact: john.doe@fake-email-host.com

This example is the same as above, except that I have a multiple repositories involved.

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: arcxp/datadog-service-catalog-metadata-provider@v2
        with:
          schema-version: v2
          datadog-hostname: api.us5.datadoghq.com
          datadog-key: ${{ secrets.DATADOG_API_KEY }}
          datadog-app-key: ${{ secrets.DATADOG_APPLICATION_KEY }}
          service-name: hello-world
          team: Global Greetings
          email: global.greetings@fake-email-host.com
          slack-support-channel: 'https://team-name-here.slack.com/archives/ABC123'
          repos: |
            - name: hello-world
              url: https://github.com/fake-org/hello-world
              provider: github
            - name: some-library
              url: https://github.com/fake-org/some-library
              provider: github
          docs: |
            - name: outage-runbook
              url: https://fake-org.github.io/hello-world-outage-runbook
              provider: github
          integrations: |
            opsgenie:
              service-url: https://fake-org.hello-world.opsgenie.com
              region: US
          contacts: |
            - name: Product Owner
              type: email
              contact: john.doe@fake-email-host.com

This is the maximal configuration you could use, in a complete workflow that you could copy-and-paste. It's a bit verbose, but it shows you all the options you have complete with "why."

---
name: Datadog Service Catalog Metadata Provider

on:
  # This will make my service definition get pushed any time I push a change
  # to the main branch of my repository.
  push:
    branches:
      - main
  # This trigger will allow me to manually run the Action in GitHub.
  workflow_dispatch:

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      # This uses the custom action to push the service definition to Datadog.
      - uses: arcxp/datadog-service-catalog-metadata-provider@v2
        with:
          service-version: v2
          # You should use GitHub's encrypted secrets feature to manage secrets for Datadog.
          # Don't store your secrets in your workflow files, and don't do anything fancy to get them.
          # GitHub already gave us a great tool for managing secrets, and it's super easy to use.
          datadog-hostname: api.us5.datadoghq.com
          datadog-key: ${{ secrets.DATADOG_API_KEY }}
          datadog-app-key: ${{ secrets.DATADOG_APPLICATION_KEY }}

          # This maps to the `dd-service` field in Datadog, it's just the name of your service.
          service-name: hello-world

          # The name of the team which owns and/or supports the service.
          team: Global Greetings

          # The email address of the team which owns and/or supports the service.
          email: global.greetings@fake-email-host.com

          # The URL of the Slack channel where support for the service is handled.
          # Keep in mind, this _must_ be a URL. To get the URL, right-click on the channel
          # in the Slack app, and select "Copy link" in the "Copy" submenu.
          slack-support-channel: 'https://team-name-here.slack.com/archives/ABC123'

          tags: |
            - isprod:true
            - lang:nodejs

          # For repos, you'll obviously want to have the repository for your service. If your service
          # is made up of multiple repositories, you can add them here as well. Note that we're using a multi-line string here. That multi-line string will be parsed as YAML, I didn't typo.
          repos: |
            - name: hello-world (primary service repo)
              url: https://github.com/fake-org/hello-world
              provider: github
            - name: some-library
              url: https://github.com/fake-org/some-library
              provider: github

          # Docs contain anything that you might need when supporting the service.
          docs: |
            - name: API Docs
              url: https://fake-org.github.io/hello-world-api-docs
              provider: github

          # Links are great for runbooks, other documentation, other services which
          # could be helpful, as well as dashboards.
          links: |
            - name: outage-runbook
              url: https://fake-org.github.io/hello-world-outage-runbook
              type: runbook
            - name: hello-world dashboard
              url: https://app.datadoghq.com/dashboard/1234567890
              type: dashboard

          # These integrations allow folks to be able to see who's on-call for the
          # service right from the Datadog Service Catalog.
          integrations: |
            opsgenie:
              service-url: https://fake-org.hello-world.opsgenie.com
              region: US
            pagerduty: https://fake-org.hello-world.pagerduty.com

          # This is the Product Owner, you should contact them if you have suggestions.
          contacts: |
            - name: Product Owner
              type: email
              contact: john.doe@fake-email-host.com

This is ported over from v2, and v2.1 fields are added.

---
name: Datadog Service Catalog Metadata Provider

on:
  # This will make my service definition get pushed any time I push a change
  # to the main branch of my repository.
  push:
    branches:
      - main
  # This trigger will allow me to manually run the Action in GitHub.
  workflow_dispatch:

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      # This uses the custom action to push the service definition to Datadog.
      - uses: arcxp/datadog-service-catalog-metadata-provider@v2
        with:
          service-version: v2.1
          # You should use GitHub's encrypted secrets feature to manage secrets for Datadog.
          # Don't store your secrets in your workflow files, and don't do anything fancy to get them.
          # GitHub already gave us a great tool for managing secrets, and it's super easy to use.
          datadog-hostname: api.us5.datadoghq.com
          datadog-key: ${{ secrets.DATADOG_API_KEY }}
          datadog-app-key: ${{ secrets.DATADOG_APPLICATION_KEY }}

          # This maps to the `dd-service` field in Datadog, it's just the name of your service.
          service-name: hello-world

          # This is the application name, which is used to group services together.
          application: hello-world

          # This application is critically important, so let's outline that with the tier field.
          tier: critical

          # This service is in production, so let's put that into the lifecycle field
          lifecycle: production

          # The name of the team which owns and/or supports the service.
          team: Global Greetings

          # The email address of the team which owns and/or supports the service. This is a convenience field
          email: global.greetings@fake-email-host.com

          # The URL of the Slack channel where support for the service is handled.
          # Keep in mind, this _must_ be a URL. To get the URL, right-click on the channel
          # in the Slack app, and select "Copy link" in the "Copy" submenu.
          slack-support-channel: 'https://team-name-here.slack.com/archives/ABC123'

          tags: |
            - isprod:true
            - lang:nodejs

          # Links are great for runbooks, other documentation, other services which
          # could be helpful, as well as dashboards. In v2.1 repos and docs move here.
          links: |
            - name: hello-world (primary service repo)
              url: https://github.com/fake-org/hello-world
              type: repo
              provider: github
            - name: some-library
              url: https://github.com/fake-org/some-library
              type: repo
              provider: github
             - name: outage-runbook
              url: https://fake-org.github.io/hello-world-outage-runbook
              type: runbook
            - name: hello-world dashboard
              url: https://app.datadoghq.com/dashboard/1234567890
              type: dashboard
            - name: API Docs
              url: https://fake-org.github.io/hello-world-api-docs
              type: doc
              provider: github

          # These integrations allow folks to be able to see who's on-call for the
          # service right from the Datadog Service Catalog.
          integrations: |
            opsgenie:
              service-url: https://fake-org.hello-world.opsgenie.com
              region: US
            pagerduty:
              service-url: https://fake-org.hello-world.pagerduty.com

          # This is the Product Owner, you should contact them if you have suggestions.
          contacts: |
            - name: Product Owner
              type: email
              contact: john.doe@fake-email-host.com

While there are a number of triggers you can use for this workflow, I recommend that you limit the triggers here to workflow_dispatch and push for your primary branch. Keep in mind that Datadog is going to always overwrite the Service Catalog definition whenever you run this action.

WARNING: If your workflow triggers trigger this Action when it's not needed, you may not only waste minutes in GitHub Actions, but you may experience rate-limiting by Datadog. If you experience rate-limiting, you may wish to restrict your triggers to only workflow_dispatch and push for your primary branch.

ANOTHER WARNING: Keep in mind that this is pushing your data up to Datadog. Please only use this Action when it's time to tell Datadog. If you're using this Action to test your configuration, you probably want to use a different Action for your live services, each with different triggers. If you don't do this, you may end up publishing service definitions which conflict with one another.

One of the challenges of running a large set of services is keeping track of what belongs to who, and enforcing all of the rules associated with that. With this GitHub Action you can use your organization's .github repository to configure controls that this Action will enforce for you. Combine that with branch rules, and you can prevent merges which would break the rules.

The place where you will configure your organization's rules is .github/service-catalog-rules.yml in your organization's .github repository. This file will be a YAML file which contains a list of rules. The syntax of this file is simple YAML, and the rules are simple as well.

For schema versioning, backwards compatibility will be maintained. If you would like to make schema version a constraint, there is a field for that in the Org Rules File.

Here's an example:

---

org: arcxp

rules:
  # This should require all service catalog definitions to
  # contain an `isprod` tag.
  - name: Simple Requirements
    selection: all
    requirements:
      tags:
      - isprod: ANY

  # Everything that runs in production must have a runbook
  - name: Prod needs a runbook
    selection:
      tags:
        isprod: "true"
    requirements:
      links:
        type: "runbook"

It should be noted that all of the key-value pairs for rules and selection criteria in an Org Rules File are case-insensitive for the values. All values will be converted to locale-sensitive lowercase before being evaluated. Keys are still case-sensitive though.

For tags, all things are lower-case.

This is the key of your org. There are two reasons why we have this key:

• It helps to have a key in here to remember which org the Org Rules File applies to.
• It's a sanity check to make sure that you're not accidentally running this Action against the wrong org. More of a misunderstanding mitigation device than anything else. If the org value doesn't match the org that the Action is running in, the Action will fail.

A quick note on versioning and backwards-compatibility: If a field is backwards-compatible, it will be marked as such in the table below. Not all fields are backwards-compatible, and if they are not then they will be skipped if the schema version is mismatched. We are intentionally using the same version numbers here that we use in the Service Catalog schema, this way there is less confusion. If you would like to make schema version a constraint, there is a field for that in the Org Rules File.

The Org Rules File is pretty light-weight, here's a breakdown of the fields:

The selection structure allows the Metadata Provider to choose which rules will apply to which services. The selection field is a list of criteria which will be used to select the services which the rule applies to. The word all for the value of this field indicates that it is applicable to all definitions for the whole org. This field must be a list, except for the case of all.

You can select on a small handful of fields on the greater service definition. These fields are:

• Tags
• Service Name
• Team Name

Wildcards are not supported, so all values must be exact matches. Wildcards may be supported in the future.

The requirements section of the Org Rules file is where you will define the requirements which must be met for the rule to pass. The requirements are a list of requirements, and each requirement is a list of fields which must be present in the service definition.

The syntax for these requirements is as follows:

---

org: "my-org"

rules:
  - name: "This is a test."
    selection:
      tags:
        isprod: "true"
    requirements:
      tags:
        - data-sensitivity:
          - critical
          - high
          - medium
          - low
          - public
        - isprod: ANY
      links:
        count: 1
        type: "runbook"
      docs:
        count: 1
      contacts:
        type: "email"
        count: 2
      repos:
        count: 1
      integrations:
        - pagerduty

This is a maximal set of requirements for v2, but here's what it means:

• The rule applies only to services which have the isprod tag, and the value of that tag is true.
• The service is required to have a tag named data-sensitivity, and the value of that tag must be one of critical, high, medium, low, or public.
• The rule requires that the service have at least one links entry with the type runbook.
• The rule requires that the service have at least one docs.
• The rule requires that the service have at least one contacts entry with the type email.
• The rule requires that the service have at least one repos.
• The rule requires that the service have at least one integrations entry called pagerduty.

It's encouraged to be judicious in how these are required. Restrictions are inherently, well, restrictive. If you're not careful, you can end up with a situation where you're not able to add new services to your org because they don't meet the requirements of a rule. It's a good idea to start with a minimal set of requirements, and then add more as you go.

Because of versioning, the above schema will break with schema v2.1. Let's take a look at what this would look like if we wanted the same rules to work across all versions of the schema:

---

org: "my-org"

rules:
  - name: "This is a test. (v2)"
    selection:
      schema-version: v2
      tags:
        isprod: "true"
    requirements:
      tags:
        - data-sensitivity:
          - critical
          - high
          - medium
          - low
          - public
        - isprod: ANY
      links:
        count: 1
        type: "runbook"
      docs:
        count: 1
      contacts:
        type: "email"
        count: 2
      repos:
        count: 1
      integrations:
        - pagerduty

  - name: "This is a test. (v2.1)"
    selection:
      schema-version: v2.1
      tags:
        isprod: "true"
    requirements:
      tags:
        - data-sensitivity:
          - critical
          - high
          - medium
          - low
          - public
        - isprod: ANY
      links:
        count: 3
        type:
          - "runbook"
          - "repo"
          - "doc"
      contacts:
        type: "email"
        count: 2
      integrations:
        - pagerduty

Since the repos and docs fields were rolled up under links in v2.1, this second rule now requires three links, at least one of type runbook, one doc, and one repo. The first rule still shows the older schema version, and works just the way it used to in v2.

• Datadog will always force your tag names and values to lowercase. The use of lower-case characters in all tags is encouraged, in order to avoid inconsistencies between your definitions and Datadog's.
• Datadog will replace any whitespace in your tag values with a -. please keep that in mind, too.
• For both of the above reasons, I encourage a more picky use of quotation marks in your YAML, just to avoid any potential parser misunderstandings.

As with any other application, there are a number of decisions that were made in the design of this Action. Here are some of the more important ones:

• For the Org Rules File, only the .github repository in the same GitHub org as the repository being processed will be used. This is to prevent a naughty user from trying to play around too much with the repositories and finding a toehold to do something malicious.
• If there is no Org Rules file, the action will still run, but it will not enforce any rules.
• If there is a file, but it does not parse, the action will fail. This is to prevent a user from accidentally breaking the Org Rules File, and then not knowing why the action is failing, but also to make sure that if someone intended for rules to be followed, we respect that.
• If there is an Org Rules File, but it does not contain any rules, the action will still run, but it will not enforce any rules (because there aren't any).

While this Custom Action does handle sensitive information (mostly your Datadog and GitHub secrets), this information is only used for the purpose of interacting with the Datadog and GitHub APIs. This information is not stored anywhere, and is not used for any other purpose. This Action is open source, and you are encouraged to review the code.

All security reports and dependency vulnerabilities via GitHub's security features are reviewed and addressed as quickly as possible. If you have more questions, please review the SECURITY.md file in this repository.

In no way does this Action phone home or otherwise send any information to any service other than GitHub or Datadog.

• Datadog Service Definition API
• JSON Schema for the Datadog Service Definition
• Working example for the Org Rules File

Unless otherwise specified, none of the authors of this project are affiliated with Datadog, Inc.

• Mike Stemle

• Thanks to The Washington Post and Arc XP for letting me work on this really cool project.
• GitHub Copilot was exceptionally helpful in the writing of tests and documentation for this program.
• Datadog personnel have been instrumental in helping me understand the API and the schema, and have been very helpful in getting this Action to a place where it's ready for public consumption.
• My own leadership, especially Jason Taylor and Jason Bartz, have been highly supportive of this project, and have provided invaluable peer review.
• Anybody who has contributed to this project, either through code, documentation, creating issues, or testing.

Thanks everybody!