Skip to content

Commit

Permalink
feat: Add the http return code to metric api_processed_total
Browse files Browse the repository at this point in the history 
Signed-off-by: Vipul Singh <vipul21sept@gmail.com>
  • Loading branch information
@vipul-21
vipul-21 committed Mar 25, 2024
1 parent a5f2a72 commit 25d6cec
Show file tree
Hide file tree
Showing 10 changed files with 120 additions and 92 deletions.
22 changes: 11 additions & 11 deletions Documentation/configuration/api-rate-limiting.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -132,17 +132,17 @@ Metrics

All API calls subject to rate limiting will expose :ref:`metrics_api_rate_limiting`. Example::

cilium_api_limiter_adjustment_factor api_call="endpoint-create" 0.695787
cilium_api_limiter_processed_requests_total api_call="endpoint-create" outcome="success" 7.000000
cilium_api_limiter_processing_duration_seconds api_call="endpoint-create" value="estimated" 2.000000
cilium_api_limiter_processing_duration_seconds api_call="endpoint-create" value="mean" 2.874443
cilium_api_limiter_rate_limit api_call="endpoint-create" value="burst" 4.000000
cilium_api_limiter_rate_limit api_call="endpoint-create" value="limit" 0.347894
cilium_api_limiter_requests_in_flight api_call="endpoint-create" value="in-flight" 0.000000
cilium_api_limiter_requests_in_flight api_call="endpoint-create" value="limit" 0.000000
cilium_api_limiter_wait_duration_seconds api_call="endpoint-create" value="max" 15.000000
cilium_api_limiter_wait_duration_seconds api_call="endpoint-create" value="mean" 0.000000
cilium_api_limiter_wait_duration_seconds api_call="endpoint-create" value="min" 0.000000
cilium_api_limiter_adjustment_factor api_call="endpoint-create" 0.695787
cilium_api_limiter_processed_requests_total api_call="endpoint-create" outcome="success" return_code="200" 7.000000
cilium_api_limiter_processing_duration_seconds api_call="endpoint-create" value="estimated" 2.000000
cilium_api_limiter_processing_duration_seconds api_call="endpoint-create" value="mean" 2.874443
cilium_api_limiter_rate_limit api_call="endpoint-create" value="burst" 4.000000
cilium_api_limiter_rate_limit api_call="endpoint-create" value="limit" 0.347894
cilium_api_limiter_requests_in_flight api_call="endpoint-create" value="in-flight" 0.000000
cilium_api_limiter_requests_in_flight api_call="endpoint-create" value="limit" 0.000000
cilium_api_limiter_wait_duration_seconds api_call="endpoint-create" value="max" 15.000000
cilium_api_limiter_wait_duration_seconds api_call="endpoint-create" value="mean" 0.000000
cilium_api_limiter_wait_duration_seconds api_call="endpoint-create" value="min" 0.000000

Understanding the log output
============================
Expand Down
58 changes: 29 additions & 29 deletions Documentation/observability/metrics.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -573,17 +573,17 @@ Name Labels Defaul
API Rate Limiting
~~~~~~~~~~~~~~~~~

============================================== ================================ ========== ========================================================
Name Labels Default Description
============================================== ================================ ========== ========================================================
``api_limiter_adjustment_factor`` ``api_call`` Enabled Most recent adjustment factor for automatic adjustment
``api_limiter_processed_requests_total`` ``api_call``, ``outcome`` Enabled Total number of API requests processed
``api_limiter_processing_duration_seconds`` ``api_call``, ``value`` Enabled Mean and estimated processing duration in seconds
``api_limiter_rate_limit`` ``api_call``, ``value`` Enabled Current rate limiting configuration (limit and burst)
``api_limiter_requests_in_flight`` ``api_call`` ``value`` Enabled Current and maximum allowed number of requests in flight
``api_limiter_wait_duration_seconds`` ``api_call``, ``value`` Enabled Mean, min, and max wait duration
``api_limiter_wait_history_duration_seconds`` ``api_call`` Disabled Histogram of wait duration per API call processed
============================================== ================================ ========== ========================================================
============================================== ========================================== ========== ========================================================
Name Labels Default Description
============================================== ========================================== ========== ========================================================
``api_limiter_adjustment_factor`` ``api_call`` Enabled Most recent adjustment factor for automatic adjustment
``api_limiter_processed_requests_total`` ``api_call``, ``outcome``, ``return_code`` Enabled Total number of API requests processed
``api_limiter_processing_duration_seconds`` ``api_call``, ``value`` Enabled Mean and estimated processing duration in seconds
``api_limiter_rate_limit`` ``api_call``, ``value`` Enabled Current rate limiting configuration (limit and burst)
``api_limiter_requests_in_flight`` ``api_call`` ``value`` Enabled Current and maximum allowed number of requests in flight
``api_limiter_wait_duration_seconds`` ``api_call``, ``value`` Enabled Mean, min, and max wait duration
``api_limiter_wait_history_duration_seconds`` ``api_call`` Disabled Histogram of wait duration per API call processed
============================================== ========================================== ========== ========================================================

cilium-operator
---------------
Expand Down Expand Up @@ -1129,15 +1129,15 @@ Name Labels
API Rate Limiting
~~~~~~~~~~~~~~~~~

============================================== ================================ ========================================================
Name Labels Description
============================================== ================================ ========================================================
``api_limiter_processed_requests_total`` ``api_call``, ``outcome`` Total number of API requests processed
``api_limiter_processing_duration_seconds`` ``api_call``, ``value`` Mean and estimated processing duration in seconds
``api_limiter_rate_limit`` ``api_call``, ``value`` Current rate limiting configuration (limit and burst)
``api_limiter_requests_in_flight`` ``api_call`` ``value`` Current and maximum allowed number of requests in flight
``api_limiter_wait_duration_seconds`` ``api_call``, ``value`` Mean, min, and max wait duration
============================================== ================================ ========================================================
============================================== ========================================== ========================================================
Name Labels Description
============================================== ========================================== ========================================================
``api_limiter_processed_requests_total`` ``api_call``, ``outcome``, ``return_code`` Total number of API requests processed
``api_limiter_processing_duration_seconds`` ``api_call``, ``value`` Mean and estimated processing duration in seconds
``api_limiter_rate_limit`` ``api_call``, ``value`` Current rate limiting configuration (limit and burst)
``api_limiter_requests_in_flight`` ``api_call`` ``value`` Current and maximum allowed number of requests in flight
``api_limiter_wait_duration_seconds`` ``api_call``, ``value`` Mean, min, and max wait duration
============================================== ========================================== ========================================================

Controllers
~~~~~~~~~~~
Expand Down Expand Up @@ -1213,15 +1213,15 @@ Name Labels
API Rate Limiting
~~~~~~~~~~~~~~~~~

============================================== ================================ ========================================================
Name Labels Description
============================================== ================================ ========================================================
``api_limiter_processed_requests_total`` ``api_call``, ``outcome`` Total number of API requests processed
``api_limiter_processing_duration_seconds`` ``api_call``, ``value`` Mean and estimated processing duration in seconds
``api_limiter_rate_limit`` ``api_call``, ``value`` Current rate limiting configuration (limit and burst)
``api_limiter_requests_in_flight`` ``api_call`` ``value`` Current and maximum allowed number of requests in flight
``api_limiter_wait_duration_seconds`` ``api_call``, ``value`` Mean, min, and max wait duration
============================================== ================================ ========================================================
============================================== ========================================== ========================================================
Name Labels Description
============================================== ========================================== ========================================================
``api_limiter_processed_requests_total`` ``api_call``, ``outcome``, ``return_code`` Total number of API requests processed
``api_limiter_processing_duration_seconds`` ``api_call``, ``value`` Mean and estimated processing duration in seconds
``api_limiter_rate_limit`` ``api_call``, ``value`` Current rate limiting configuration (limit and burst)
``api_limiter_requests_in_flight`` ``api_call`` ``value`` Current and maximum allowed number of requests in flight
``api_limiter_wait_duration_seconds`` ``api_call``, ``value`` Mean, min, and max wait duration
============================================== ========================================== ========================================================

Controllers
~~~~~~~~~~~
Expand Down
1 change: 1 addition & 0 deletions Documentation/operations/upgrade.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -493,6 +493,7 @@ Changed Metrics
* The ``cilium_cidrgroup_policies`` metric has been renamed to
``cilium_cidrgroups_referenced`` for better clarity.
* The ``cilium_cidrgroup_translation_time_stats_seconds`` metric has been disabled by default.
* The ``cilium_api_limiter_processed_requests_total`` has now label ``return_code`` to specify the http code of the request.

.. _earlier_upgrade_notes:

Expand Down
55 changes: 29 additions & 26 deletions daemon/cmd/endpoint.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ func getEndpointHandler(d *Daemon, params GetEndpointParams) middleware.Responde
resEPs := d.getEndpointList(params)

if params.Labels != nil && len(resEPs) == 0 {
r.Error(errEndpointNotFound)
r.Error(errEndpointNotFound, GetEndpointNotFoundCode)
return NewGetEndpointNotFound()
}

Expand Down Expand Up @@ -137,10 +137,11 @@ func deleteEndpointHandler(d *Daemon, params DeleteEndpointParams) middleware.Re
defer r.Done()

if nerr, err := d.deleteEndpointByContainerID(params.Endpoint.ContainerID); err != nil {
r.Error(err)
if apierr, ok := err.(*api.APIError); ok {
r.Error(err, apierr.GetCode())
return apierr
}
r.Error(err, DeleteEndpointInvalidCode)
return api.Error(DeleteEndpointInvalidCode, err)
} else if nerr > 0 {
return NewDeleteEndpointErrors().WithPayload(int64(nerr))
Expand All @@ -161,10 +162,10 @@ func getEndpointIDHandler(d *Daemon, params GetEndpointIDParams) middleware.Resp
ep, err := d.endpointManager.Lookup(params.ID)

if err != nil {
r.Error(err)
r.Error(err, GetEndpointIDInvalidCode)
return api.Error(GetEndpointIDInvalidCode, err)
} else if ep == nil {
r.Error(errEndpointNotFound)
r.Error(errEndpointNotFound, GetEndpointIDNotFoundCode)
return NewGetEndpointIDNotFound()
} else {
return NewGetEndpointIDOK().WithPayload(ep.GetModel())
Expand Down Expand Up @@ -591,7 +592,7 @@ func putEndpointIDHandler(d *Daemon, params PutEndpointIDParams) (resp middlewar

ep, code, err := d.createEndpoint(params.HTTPRequest.Context(), d, epTemplate)
if err != nil {
r.Error(err)
r.Error(err, code)
return api.Error(code, err)
}

Expand Down Expand Up @@ -643,7 +644,7 @@ func patchEndpointIDHandler(d *Daemon, params PatchEndpointIDParams) middleware.
// Note: newEp's labels are ignored.
newEp, err2 := endpoint.NewEndpointFromChangeModel(d.ctx, d, d, d.ipcache, d.l7Proxy, d.identityAllocator, epTemplate)
if err2 != nil {
r.Error(err2)
r.Error(err2, PutEndpointIDInvalidCode)
return api.Error(PutEndpointIDInvalidCode, err2)
}

Expand All @@ -659,15 +660,15 @@ func patchEndpointIDHandler(d *Daemon, params PatchEndpointIDParams) middleware.

ep, err := d.endpointManager.Lookup(params.ID)
if err != nil {
r.Error(err)
r.Error(err, GetEndpointIDInvalidCode)
return api.Error(GetEndpointIDInvalidCode, err)
}
if ep == nil {
r.Error(errEndpointNotFound)
r.Error(errEndpointNotFound, PatchEndpointIDNotFoundCode)
return NewPatchEndpointIDNotFound()
}
if err = endpoint.APICanModify(ep); err != nil {
r.Error(err)
r.Error(err, PatchEndpointIDInvalidCode)
return api.Error(PatchEndpointIDInvalidCode, err)
}

Expand All @@ -679,7 +680,7 @@ func patchEndpointIDHandler(d *Daemon, params PatchEndpointIDParams) middleware.
// Support arbitrary changes? Support only if unset?
reason, err := ep.ProcessChangeRequest(newEp, validStateTransition)
if err != nil {
r.Error(err)
r.Error(err, PatchEndpointIDNotFoundCode)
return NewPatchEndpointIDNotFound()
}

Expand All @@ -692,7 +693,7 @@ func patchEndpointIDHandler(d *Daemon, params PatchEndpointIDParams) middleware.
err := api.Error(PatchEndpointIDFailedCode,
fmt.Errorf("error while regenerating endpoint."+
" For more info run: 'cilium endpoint get %d'", ep.ID))
r.Error(err)
r.Error(err, PatchEndpointIDFailedCode)
return err
}
// FIXME: Special return code to indicate regeneration happened?
Expand Down Expand Up @@ -833,10 +834,11 @@ func deleteEndpointIDHandler(d *Daemon, params DeleteEndpointIDParams) middlewar
defer r.Done()

if nerr, err := d.DeleteEndpoint(params.ID); err != nil {
r.Error(err)
if apierr, ok := err.(*api.APIError); ok {
r.Error(err, apierr.GetCode())
return apierr
}
r.Error(err, DeleteEndpointIDErrorsCode)
return api.Error(DeleteEndpointIDErrorsCode, err)
} else if nerr > 0 {
return NewDeleteEndpointIDErrors().WithPayload(int64(nerr))
Expand Down Expand Up @@ -881,10 +883,11 @@ func patchEndpointIDConfigHandler(d *Daemon, params PatchEndpointIDConfigParams)
defer r.Done()

if err := d.EndpointUpdate(params.ID, params.EndpointConfiguration); err != nil {
r.Error(err)
if apierr, ok := err.(*api.APIError); ok {
r.Error(err, apierr.GetCode())
return apierr
}
r.Error(err, PatchEndpointIDFailedCode)
return api.Error(PatchEndpointIDFailedCode, err)
}

Expand All @@ -902,10 +905,10 @@ func getEndpointIDConfigHandler(d *Daemon, params GetEndpointIDConfigParams) mid

ep, err := d.endpointManager.Lookup(params.ID)
if err != nil {
r.Error(err)
r.Error(err, GetEndpointIDInvalidCode)
return api.Error(GetEndpointIDInvalidCode, err)
} else if ep == nil {
r.Error(errEndpointNotFound)
r.Error(errEndpointNotFound, GetEndpointIDConfigNotFoundCode)
return NewGetEndpointIDConfigNotFound()
} else {
cfgStatus := ep.GetConfigurationStatus()
Expand All @@ -925,17 +928,17 @@ func getEndpointIDLabelsHandler(d *Daemon, params GetEndpointIDLabelsParams) mid

ep, err := d.endpointManager.Lookup(params.ID)
if err != nil {
r.Error(err)
r.Error(err, GetEndpointIDInvalidCode)
return api.Error(GetEndpointIDInvalidCode, err)
}
if ep == nil {
r.Error(errEndpointNotFound)
r.Error(errEndpointNotFound, GetEndpointIDLabelsNotFoundCode)
return NewGetEndpointIDLabelsNotFound()
}

cfg, err := ep.GetLabelsModel()
if err != nil {
r.Error(err)
r.Error(err, GetEndpointIDInvalidCode)
return api.Error(GetEndpointIDInvalidCode, err)
}

Expand All @@ -954,10 +957,10 @@ func getEndpointIDLogHandler(d *Daemon, params GetEndpointIDLogParams) middlewar
ep, err := d.endpointManager.Lookup(params.ID)

if err != nil {
r.Error(err)
r.Error(err, GetEndpointIDLogInvalidCode)
return api.Error(GetEndpointIDLogInvalidCode, err)
} else if ep == nil {
r.Error(errEndpointNotFound)
r.Error(errEndpointNotFound, GetEndpointIDLogNotFoundCode)
return NewGetEndpointIDLogNotFound()
} else {
return NewGetEndpointIDLogOK().WithPayload(ep.GetStatusModel())
Expand All @@ -976,10 +979,10 @@ func getEndpointIDHealthzHandler(d *Daemon, params GetEndpointIDHealthzParams) m
ep, err := d.endpointManager.Lookup(params.ID)

if err != nil {
r.Error(err)
r.Error(err, GetEndpointIDHealthzInvalidCode)
return api.Error(GetEndpointIDHealthzInvalidCode, err)
} else if ep == nil {
r.Error(errEndpointNotFound)
r.Error(errEndpointNotFound, GetEndpointIDHealthzNotFoundCode)
return NewGetEndpointIDHealthzNotFound()
} else {
return NewGetEndpointIDHealthzOK().WithPayload(ep.GetHealthModel())
Expand Down Expand Up @@ -1034,22 +1037,22 @@ func putEndpointIDLabelsHandler(d *Daemon, params PatchEndpointIDLabelsParams) m

ep, err := d.endpointManager.Lookup(params.ID)
if err != nil {
r.Error(err)
r.Error(err, PutEndpointIDInvalidCode)
return api.Error(PutEndpointIDInvalidCode, err)
} else if ep == nil {
r.Error(errEndpointNotFound)
r.Error(errEndpointNotFound, PatchEndpointIDLabelsNotFoundCode)
return NewPatchEndpointIDLabelsNotFound()
}

add, del, err := ep.ApplyUserLabelChanges(lbls)
if err != nil {
r.Error(err)
r.Error(err, PutEndpointIDInvalidCode)
return api.Error(PutEndpointIDInvalidCode, err)
}

code, err := d.modifyEndpointIdentityLabelsFromAPI(params.ID, add, del)
if err != nil {
r.Error(err)
r.Error(err, code)
return api.Error(code, err)
}
return NewPatchEndpointIDLabelsOK()
Expand Down
5 changes: 5 additions & 0 deletions pkg/api/apierror.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@ func New(code int, msg string, args ...interface{}) *APIError {
return &APIError{code: code, msg: msg}
}

// GetCode returns the code for the API Error.
func (a *APIError) GetCode() int {
return a.code
}

// Error creates a new API error from the code and error.
func Error(code int, err error) *APIError {
if err == nil {
Expand Down

0 comments on commit 25d6cec

Please sign in to comment.