Bucketting of resources

[sinduri]

Bluemix BSS requires accumulation and aggregation across a group of resources(with same metrics).

For example, different versions of nodeJS (different resource-id) need to be accumulated and aggregated to the bucket(sdk-4-nodejs) or ruby, python, php runtimes usage to platform bucket.

How can we achieve this in abacus? Do we on board the metering and rating formulas to bucket-ids instead of the actual resource id?

[jsdelfino]

I'd suggest the following:

• keep the correct resource id for your buildpack (e.g. node-0.12) in the submitted usage doc;
• add an optional resource_type_id field (e.g. node) to that usage doc;
• add a bit of code to the aggregator and rating services to optionally aggregate usage by resource_type, similar to how usage is currently aggregated by consumer for example.

That resource_type_id could also be used to lookup resource configurations as an alternative to resource_id, allowing you to define a single resource configuration for node instead of having to define one for node-0.10, another for node-0.12 etc.

Submitting node as your resource_id instead of the actual/correct resource_id (is that what you meant by on boarding bucket ids?? not sure...) is probably a recipe for trouble as with that approach you won't be able to distinguish node-0.10 and node-0.12 usage (I can already guess that somebody will want that at some point) and if you don't submit your usage with the correct buildpack resource_id in the first place you won't be able to go back and figure out what buildpack was used after the fact.

So, IMO adding a resource_type_id field should be a better approach for your use case. If that sounds like a sensible solution, I'll be happy to mentor and help somebody who would like to volunteer to contribute that enhancement.

Thoughts?

[jsdelfino]

Any thoughts?

[piotrprzybylski]

Yes, I think its a good approach to have extra level of grouping resources by type (resource_type_id) and submit actual buildpack id. In addition to meeting this requirement, it will also allow to reduce number of resource definitions (as mentioned in #40). I'll be happy to work on the contribution to support that functionality.
As part of this feature, we should decide how to enable resource type in existing components, for example cf bridge.
One idea we discussed with @sinduri was to have a service which would provide mapping from the resource_id to resource_type_id. The default implementation/stub could map all the resource_ids to the type corresponding to the current resource definition for applications (linux-container)j. If specific installation would like to customize submission, it would provide different implementation of such service along with the appropriate resource definitions.

[jsdelfino]

Thanks for volunteering to work on this! I'll assign the issue to you then :)

[jsdelfino]

@piotrprzybylski Well... It doesn't look like I'm allowed to assign the issue to you, maybe only you can assign it to yourself... Not sure.

[piotrprzybylski]

I think one needs to be a committer to have issue assigned to. I was not able to do it either.

[jsdelfino]

One idea we discussed with @sinduri was to have a service which would provide mapping from the resource_id to resource_type_id.

@piotrprzybylski I'm not following that part, and in particular why can't the resource provider just send the resource_type_id along with the resource_id in the submitted usage doc, instead of not sending it and then requiring yet another service to the mapping later?

That sounds a bit complicated to me and I'm not quite understanding the value that would provide. Could you help me understand this better?

[piotrprzybylski]

The resource provider external to abacus can define and send both values, the resource_type_id and resource_id. The comment was about ability to customize resource types or grouping of resources for provider like cf bridge.

[jsdelfino]

@piotrprzybylski So if I understand what you're saying then a resource config could be associated with a resource_type_id (e.g. node) instead of a resource_id (e.g. node-0.10 and node-0.12) to cover multiple resource_ids of the same type? Did I get that right?

[piotrprzybylski]

Yes, that is correct, this could reduce number of needed resource configurations.

[jsdelfino]

Marking this as a future milestone for now, unless somebody volunteers to do it sooner.

[piotrprzybylski]

We are starting work on this contribution and will discuss details on the abacus mailing list.

[jsdelfino]

@piotrprzybylski, @sinduri, Any update on this? Do you still need any code changes in Abacus to group these resources by type?

[jsdelfino]

This is now implemented. Resource ids are mapped to resource_type ids and usage aggregated by resource_type.