[Sonic HLD] Metering requirements #313
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,7 +15,8 @@ | |
| * [1.2 CLI requirements](#12-cli-requirements) | ||
| * [1.3 Warm Restart requirements ](#13-warm-restart-requirements) | ||
| * [1.4 Scaling requirements ](#14-scaling-requirements) | ||
| * [1.5 Metering requirements ](#15-metering-requirements) | ||
| * [1.6 Design considerations ](#16-design-considerations) | ||
| * [2 Packet Flows](#2-packet-flows) | ||
| * [3 Modules Design](#3-modules-design) | ||
| * [3.1 Config DB](#31-config-db) | ||
|
|
@@ -102,8 +103,38 @@ Following are the minimal scaling requirements | |
| | ACLs per ENI | 6x10K SRC/DST ports | | ||
| | CA-PA Mappings | 10M | | ||
| | Active Connections/ENI | 1M (Bidirectional TCP or UDP) | | ||
| | Metering Buckets per ENI | 4000 | | ||
|
|
||
| ## 1.5 Metering requirements | ||
| Metering is essential for billing the customers and below are the high-level requirements. Metering/Bucket in this context is related to byte counting for billing purposes and not related to traffic policer or shaping. | ||
| - Billing shall be at per ENI level and shall be able to query metering packet bytes per ENI | ||
| - All metering buckets must be UINT64 size and start from value 0 and shall be counting number of bytes. A bucket contains 2 counters; 1 inbound (Rx) and 1 outbound (Tx). | ||
| - Implementation (a.k.a H/W pipeline implementation) must support metering at the following levels: | ||
| - Policy based metering. - E.g. For specific destinations (prefix) that must be billed separately, say action_type 'direct' | ||
| - Route table based metering - E.g. For Vnet peering cases. | ||
| - Mapping table based metering - E.g For specific destinations within the mapping table that must be billed separately | ||
| - Policy in the metering context refers to metering policy associated to Route tables. This is not related to ACL policy or any ACL packet counters. | ||
| - If packet flow hits multiple metering buckets, order of priority shall be **Policy->Route->Mapping** | ||
| - User shall be able to override the precedence between Routing and Mapping buckets by setting an _override_ flag. When policy is enabled for a route, it takes higher precedence than routing and mapping metering bucket. | ||
| - Implementation shall aggregate the counters on an "_ENI+Metering Bucket_" combination for billing: | ||
| - All traffic from an ENI to a Peered VNET | ||
| - All traffic from an ENI to a Private Link destination | ||
| - All traffic from an ENI to an internet destination | ||
| - All traffic from an ENI to cross-AZ destination (within the same VNET) | ||
| - All traffic from an ENI to cross-region destination (towards the peer VNET) | ||
| - All outbound metered traffic from an ENI | ||
| - All inbound metered traffic towards an ENI | ||
| - Customer is billed based on number of bytes sent/received separately. A distinct counter must be supported for outbound vs inbound traffic of each category. | ||
|
There was a problem hiding this comment. fix terminology: replace inbound/outbound with TX/RX ? since inbound/outbound refers to the first (initiator) packet for the flow There was a problem hiding this comment. @vijasrin , i cannot commit to this forked branch anymore after migration. Since its a minor fix, i'll update it in one of the future commits. |
||
| - For outbound flow and associated metering bucket, created as part of VM initiated traffic, the metering bucket shall account for outbound (Tx) bytes. Based on this outbound flow, pipeline shall also create a unified inbound flow. The same metering bucket shall account for the inbound (Rx) bytes for the return traffic to VM that matches this flow. | ||
|
There was a problem hiding this comment. From scale point of view, there are going be 5M connection and 10M flows (if we count inbound and outbound flows separately). Let's say we support 5M combined unified flows and if the scale required is 4K buckets per ENI, for a total of 64 ENIs, we will have 256K buckets per DPU. Considering we are sharing these buckets across ENIs and across flows, we will be limited to how much we can count separately for each of the categories you have listed above? Is this not true? There was a problem hiding this comment. Yes, it is true. But note that many connections may share same bucket id. For example, if there are 100's of mapping for a cross-Az, it will all share the same bucket. However, it is possible that in future we may have to increase the limit from 4k per ENI. |
||
| - Application shall utilize the metering hardware resource in an optimized manner by allocating meter id and deallocating when not-in-use | ||
| - Application shall bind all associated metering buckets to an ENI. During ENI deletion, all associated metering bucket binding should be auto-removed. | ||
| - A route rule table can also have a metering bucket association for explicitly accounting the inbound traffic for an ENI. | ||
|
|
||
| _Open Items_ | ||
| - Can we avoid explicit dependency between ENI's and mappings? | ||
| - Bucket id integer to be associated to an optional metadata string? | ||
|
|
||
| ## 1.6 Design Considerations | ||
|
|
||
| DASH Sonic implementation is targeted for appliance scenarios and must handles millions of updates. As such, the implementation is performance and scale focussed as compared to traditional switching and routing based solutions. The following are the design considerations. | ||
|
|
||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How is this "outbound metered traffic" different from the everything listed before it? Should this be "All other" outbound metered traffic....."? When you say implementation, which "implementation"? Is this "HW pipeline", "SW on the DPU", or "the SDN controller"? Also, you could have cross-AZ peer VNET traffic as well, would you not want to count separately as well? I guess I don't understand why you are listing these outbound traffic categories. Is it that these are the ones you want to bill separately?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All outbound metered traffic means query on
ENI + *-> Aggregated counter from all metering buckets associated with ENI. For cross-AZ peer VNET, requirement is to bill as peer VNET. But this is made generic in the modeling that if it requires to be billed differently, it is still possible. Yes, those are the cases that must be billed separately.I've updated the meaning for "Implementation" in the first usage.