Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added rate-limiter configs in the table level #563

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Added rate-limiter configs in the table level #563

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1266b60
Added rate-limiter configs in the table level
ParthaI Apr 9, 2024
eee6bc5
Updated the tables with appropriate rate-limiter tags
ParthaI Apr 10, 2024
c9e2304
Update the gcp_cloud_asset table with correct set of limiter tags
ParthaI Apr 11, 2024
455fbac
Update the remains table with rate-limiter tag
ParthaI Apr 18, 2024
f6e65b4
Resolved conflict
ParthaI Apr 23, 2024
e2cb6e0
updated the plugin level rate limiter config for the tables
ParthaI Apr 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
16 changes: 16 additions & 0 deletions gcp/plugin.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (

"github.com/turbot/steampipe-plugin-sdk/v5/plugin"
"github.com/turbot/steampipe-plugin-sdk/v5/plugin/transform"
"github.com/turbot/steampipe-plugin-sdk/v5/rate_limiter"
)

const pluginName = "steampipe-plugin-gcp"
Expand All @@ -27,6 +28,21 @@ func Plugin(ctx context.Context) *plugin.Plugin {
DefaultIgnoreConfig: &plugin.IgnoreConfig{
ShouldIgnoreErrorFunc: shouldIgnoreErrorPluginDefault(),
},
RateLimiters: []*rate_limiter.Definition{
// A user can make up to 100 BigQuery API requests per second, per user, per project to an API method. If a user makes more than 100 requests per second to a method, then throttling can occur. This limit does not apply to
// Service API limit: https://cloud.google.com/bigquery/quotas#api_request_quotas
// https://cloud.google.com/bigquery/quotas#dataset_limits
// Configuring the rate limiter only for LIST operations will result in hitting rate limits when executing the query "select * from gcp_bigquery_dataset".
// This occurs because the table `gcp_bigquery_dataset` includes hydrated calls (getBigQueryDataset). Therefore, it is necessary to configure the rate limiter for both LIST and GET operations.
{
Name: "gcp_bigquery_list_datasets",
FillRate: 100,
BucketSize: 100,
MaxConcurrency: 50,
Scope: []string{"connection", "service", "action"},
Where: "service = 'bigquery' and (action = 'datasets.list' or action = 'datasets.get')",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ParthaI should the Where section be as follows, since you mentioned in the comment that the rate limiter would be beneficial for the LIST operations?

Where:          "service = 'bigquery' and action = 'datasets.list'",

Copy link
Contributor Author

@ParthaI ParthaI Apr 29, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@misraved, as per the comment we are still getting the rate limiter error if we run the query select * gcp_bigquery_dataset. If we are only configuring the limiter for the LIST API call. Because we are also making the GET API call as it is being used as a hydrated API call.

If we encounter a rate-limit error during a LIST API call and simultaneously make a GET API call for the same service, we will likely face the same rate limit error. Therefore, I have configured both the LIST and GET calls within the where clause to address this issue.

},
},
ConnectionKeyColumns: []plugin.ConnectionKeyColumn{
{
Name: "project",
Expand Down