Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[APM] Define data schema based on OAM #180932

Closed
cauemarcondes opened this issue Apr 16, 2024 · 1 comment · Fixed by #181387
Closed

[APM] Define data schema based on OAM #180932

cauemarcondes opened this issue Apr 16, 2024 · 1 comment · Fixed by #181387
Assignees
@cauemarcondes
Labels
enhancement New value added to drive a business result Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team

Comments

@cauemarcondes
Copy link
Contributor

cauemarcondes commented Apr 16, 2024 •
edited
Loading

Asset schema:

[
    {
    "asset": {
        "id": "payment-gateway",
        "type": "service",
        "first_seen": "2024-03-02T09:10:34.345Z",
        "last_seen": "2024-03-18T17:37:05.092Z",
        "identifying_metadata": [ "service.name" ],
        "signalTypes": {
           "has_traces": true,
        }
    },
    "service": {
        "language.name": "python",
        "name": "payment-gateway",
        "node.name": "6405c4f8-bfa6-43df-b763-36834629fa8c",
        "environment": "dev"
    }
    },
    {
        "asset": {
            "id": "payment-gateway-2",
            "type": "service",
            "first_seen": "2024-03-02T09:10:34.345Z",
            "last_seen": "2024-03-18T17:37:05.092Z",
            "identifying_metadata": [ "service.name" ],
            "has_traces": true,
           "signalTypes": {
              "has_traces": true,
              "has_logs": true,
            }
        },
        "service": {
            "language.name": "python",
            "name": "payment-gateway-2",
            "node.name": "6405c4f8-bfa6-43df-b763-36834629fa8c",
            "environment": "prod"
        }
    }
]
@cauemarcondes cauemarcondes added the Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team label Apr 16, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

@cauemarcondes cauemarcondes self-assigned this Apr 16, 2024
@cauemarcondes cauemarcondes mentioned this issue Apr 16, 2024
Closed
@cauemarcondes cauemarcondes mentioned this issue Apr 24, 2024
Merged
@smith smith added enhancement New value added to drive a business result needs-refinement Needs PM's to refine scope labels Apr 25, 2024
@cauemarcondes cauemarcondes removed the needs-refinement Needs PM's to refine scope label Apr 25, 2024
cauemarcondes added a commit that referenced this issue Apr 26, 2024
@cauemarcondes @kibanamachine
[Synthrace] Create Asset documents based on signals (Traces/Logs) (#1…
ad2af69 
…81387)

closes #180932
closes #180943

Running it:
```
node scripts/synthtrace.js traces_logs_assets.ts --clean --from=2024-04-17T08:00:00.000Z --to=2024-04-17T08:15:00.000Z --logLevel=debug
```

`assets` index output based on the scenario (traces_logs_assets):
```
"hits": [
      {
        "_index": "assets",
        "_id": "Ydx-FY8BEwTL0kdepnZC",
        "_score": 1,
        "_source": {
          "asset": {
            "id": "synth-node-0",
            "type": "service",
            "identifying_metadata": [
              "service.name"
            ],
            "first_seen": "2024-04-17T08:00:00.000Z",
            "last_seen": "2024-04-17T08:14:00.000Z",
            "signalTypes": {
              "asset.traces": true,
              "asset.logs": true
            }
          },
          "service": {
            "environment": "Synthtrace: traces_logs_assets",
            "name": "synth-node-0",
            "node": {
              "name": "instance"
            },
            "language": {}
          }
        }
      },
      {
        "_index": "assets",
        "_id": "Ytx-FY8BEwTL0kdepnZC",
        "_score": 1,
        "_source": {
          "asset": {
            "id": "synth-node-1",
            "type": "service",
            "identifying_metadata": [
              "service.name"
            ],
            "first_seen": "2024-04-17T08:00:00.000Z",
            "last_seen": "2024-04-17T08:14:00.000Z",
            "signalTypes": {
              "asset.traces": true
            }
          },
          "service": {
            "environment": "Synthtrace: traces_logs_assets",
            "name": "synth-node-1",
            "node": {
              "name": "instance"
            },
            "language": {}
          }
        }
      },
      {
        "_index": "assets",
        "_id": "Y9x-FY8BEwTL0kdepnZC",
        "_score": 1,
        "_source": {
          "asset": {
            "id": "synth-node-2",
            "type": "service",
            "identifying_metadata": [
              "service.name"
            ],
            "first_seen": "2024-04-17T08:00:00.000Z",
            "last_seen": "2024-04-17T08:14:00.000Z",
            "signalTypes": {
              "asset.traces": true
            }
          },
          "service": {
            "environment": "Synthtrace: traces_logs_assets",
            "name": "synth-node-2",
            "node": {
              "name": "instance"
            },
            "language": {}
          }
        }
      },
      {
        "_index": "assets",
        "_id": "ZNx-FY8BEwTL0kdepnZC",
        "_score": 1,
        "_source": {
          "asset": {
            "id": "synth-java",
            "type": "service",
            "identifying_metadata": [
              "service.name"
            ],
            "first_seen": "2024-04-17T08:00:00.000Z",
            "last_seen": "2024-04-17T08:14:00.000Z",
            "signalTypes": {
              "asset.logs": true
            }
          },
          "service": {
            "name": "synth-java"
          }
        }
      }
    ]
```

Updates:
- After talking to @tommyers-elastic we agreed to remove the
`asset.signalTypes` field and add `has_traces` and `has_logs`. Mainly
because updating list type field in ES is hard.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
yuliacech pushed a commit to yuliacech/kibana that referenced this issue May 3, 2024
@cauemarcondes @kibanamachine @yuliacech
[Synthrace] Create Asset documents based on signals (Traces/Logs) (el…
d644724 
…astic#181387)

closes elastic#180932
closes elastic#180943

Running it:
```
node scripts/synthtrace.js traces_logs_assets.ts --clean --from=2024-04-17T08:00:00.000Z --to=2024-04-17T08:15:00.000Z --logLevel=debug
```

`assets` index output based on the scenario (traces_logs_assets):
```
"hits": [
      {
        "_index": "assets",
        "_id": "Ydx-FY8BEwTL0kdepnZC",
        "_score": 1,
        "_source": {
          "asset": {
            "id": "synth-node-0",
            "type": "service",
            "identifying_metadata": [
              "service.name"
            ],
            "first_seen": "2024-04-17T08:00:00.000Z",
            "last_seen": "2024-04-17T08:14:00.000Z",
            "signalTypes": {
              "asset.traces": true,
              "asset.logs": true
            }
          },
          "service": {
            "environment": "Synthtrace: traces_logs_assets",
            "name": "synth-node-0",
            "node": {
              "name": "instance"
            },
            "language": {}
          }
        }
      },
      {
        "_index": "assets",
        "_id": "Ytx-FY8BEwTL0kdepnZC",
        "_score": 1,
        "_source": {
          "asset": {
            "id": "synth-node-1",
            "type": "service",
            "identifying_metadata": [
              "service.name"
            ],
            "first_seen": "2024-04-17T08:00:00.000Z",
            "last_seen": "2024-04-17T08:14:00.000Z",
            "signalTypes": {
              "asset.traces": true
            }
          },
          "service": {
            "environment": "Synthtrace: traces_logs_assets",
            "name": "synth-node-1",
            "node": {
              "name": "instance"
            },
            "language": {}
          }
        }
      },
      {
        "_index": "assets",
        "_id": "Y9x-FY8BEwTL0kdepnZC",
        "_score": 1,
        "_source": {
          "asset": {
            "id": "synth-node-2",
            "type": "service",
            "identifying_metadata": [
              "service.name"
            ],
            "first_seen": "2024-04-17T08:00:00.000Z",
            "last_seen": "2024-04-17T08:14:00.000Z",
            "signalTypes": {
              "asset.traces": true
            }
          },
          "service": {
            "environment": "Synthtrace: traces_logs_assets",
            "name": "synth-node-2",
            "node": {
              "name": "instance"
            },
            "language": {}
          }
        }
      },
      {
        "_index": "assets",
        "_id": "ZNx-FY8BEwTL0kdepnZC",
        "_score": 1,
        "_source": {
          "asset": {
            "id": "synth-java",
            "type": "service",
            "identifying_metadata": [
              "service.name"
            ],
            "first_seen": "2024-04-17T08:00:00.000Z",
            "last_seen": "2024-04-17T08:14:00.000Z",
            "signalTypes": {
              "asset.logs": true
            }
          },
          "service": {
            "name": "synth-java"
          }
        }
      }
    ]
```

Updates:
- After talking to @tommyers-elastic we agreed to remove the
`asset.signalTypes` field and add `has_traces` and `has_logs`. Mainly
because updating list type field in ES is hard.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@kpatticha kpatticha mentioned this issue May 20, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cauemarcondes cauemarcondes

Labels
enhancement New value added to drive a business result Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Synthrace] Create Asset documents based on signals (Traces/Logs) cauemarcondes/kibana
3 participants
@smith @elasticmachine @cauemarcondes