-
Notifications
You must be signed in to change notification settings - Fork 30
report metrics from third-party resource #10
Comments
That's a little finer than what I intended with this. I was more looking On Thu, Nov 17, 2016 at 10:15 PM, Brandon Philips notifications@github.com
|
@thockin well, obviously we would ask people providing these statistics to use a defined schema, likely checked in here. Essentially this is so that in Tectonic we don't have to fork Spartakus to add a few additional metrics specific to that product. |
I'm not AGAINST things like this, but there are things that a commercial On Fri, Nov 18, 2016 at 8:11 AM, Brandon Philips notifications@github.com
|
@thockin totally, I agree with that. We can totally just fork the code base to add what we need but it would make it easier to participate if there was an extension point. Another option would be another collector in the spartakus pod and a shared mountpoint. |
@thockin Any thought on this? We are happy to add the extension point as we need it. What if we added a white list as a flag? So you have to say: Then spartakus would essentially do Otherwise we can just fork the code to add the couple of metrics we need :( |
Couple clarifications: Do you intend this to be pushed to your own collector instance or to mine? To push to my collector is much more complicated because I am storing it all in BigQuery which wants a schema. If we read from an arbitrary other location and package that extra data as a JSON object in an list of extensions or something, is that sufficient? |
Our own.
For most things, yes, as long as we can optionally make some of the "extension fields" top-level fields in the DB table. b/c some will be identifiers and/or frequently used in queries. Our current thinking:
Desired end state:
|
> If we read from an arbitrary other location and package that extra data as a JSON object in an list of extensions or something, is that sufficient?
For most things, yes, as long as we can optionally make some of the "extension fields" top-level fields in the DB table. b/c some will be identifiers and/or frequently used in queries.
blech. Tradeoff complexity of merging top-level structs with a bit of
extra typing in queries...
Our current thinking:
modify the emitter to supplement payloads with additional arbitrary fields (via some json from somewhere in cluster)
modify the collector to gracefully handle these extension fields (e.g. ignore any fields not configured in a white-list for the schema, or optionally embed all the extra json into a single additional field)
Desired end state:
Users of Spartakus today would require no additional config changes for basic usage.
All additional data emission/collection is opt-in via flags.
Supplemental data generation is the responsibility of separate services, which populate to a known location (TPR?).
TPR seems heavyweight for this unless you need the data elsewhere.
Why not just run a sidecar that dumps to a file in a shared volume,
the volunteer reads the latest state from that file and reports it as
extra JSON
|
Sidecar sgtm
…On Tue, Nov 29, 2016, 12:58 PM Tim Hockin ***@***.***> wrote:
>> If we read from an arbitrary other location and package that extra data
as a JSON object in an list of extensions or something, is that sufficient?
>
> For most things, yes, as long as we can optionally make some of the
"extension fields" top-level fields in the DB table. b/c some will be
identifiers and/or frequently used in queries.
blech. Tradeoff complexity of merging top-level structs with a bit of
extra typing in queries...
> Our current thinking:
>
> modify the emitter to supplement payloads with additional arbitrary
fields (via some json from somewhere in cluster)
> modify the collector to gracefully handle these extension fields (e.g.
ignore any fields not configured in a white-list for the schema, or
optionally embed all the extra json into a single additional field)
>
> Desired end state:
>
> Users of Spartakus today would require no additional config changes for
basic usage.
> All additional data emission/collection is opt-in via flags.
> Supplemental data generation is the responsibility of separate services,
which populate to a known location (TPR?).
TPR seems heavyweight for this unless you need the data elsewhere.
Why not just run a sidecar that dumps to a file in a shared volume,
the volunteer reads the latest state from that file and reports it as
extra JSON
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#10 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AACDCFFvBxRmBjWru8SXneCENergcalFks5rDJIKgaJpZM4K2J5i>
.
|
I'm not very familiar with BigQuery, but if embedded fields can be referenced easily then we could forego this. |
Sidecar does sound like a good approach. |
Sidecar sounds good, but I think one big extra JSON string field will not be very useful. Please correct me if I'm wrong but I think since that field can be any arbitrary JSON string, BigQuery does not understand its schema and cannot search its nested fields. If we wanted to search, the best we could do would be to do string comparisons on the one big JSON dump. A better approach IMO would be to have a repeated record field called {
"version": "v1.0.0",
"timestamp": "867530909031976",
"clusterID": "2f9c93d3-156c-47aa-8802-578ffca9b50e",
"masterVersion": "v1.3.5",
"nodes": [...],
"extensions": [
{
"name": "foo",
"value": "bar",
},
...
]
} |
It would be nice for people to report additional data without changing spartakus source code. I propose that we create a new third-party resource type called "SpartakusData" that can be used to report additional arbitrary data.
It would look something like this:
This would be reported out in the payload like:
Example use cases:
cc @squat @sym3tri
The text was updated successfully, but these errors were encountered: