Skip to content
This repository has been archived by the owner on Jul 28, 2023. It is now read-only.

query on turbot_active_grant - running out of memory in Linux #57

Closed
nivethj opened this issue Aug 11, 2022 · 17 comments
Closed

query on turbot_active_grant - running out of memory in Linux #57

nivethj opened this issue Aug 11, 2022 · 17 comments
Labels
bug Something isn't working

Comments

@nivethj
Copy link

nivethj commented Aug 11, 2022

Describe the bug
A clear and concise description of what the bug is.

I am trying to query turbot_active_grant with limit of 10 records in my turbot v5 environment which has 20+k records . i am getting out of memory error . I have 4GB Mem. I tried using latest Steampipe v0.16.0-rc.8 version . but no luck.

Steampipe version (steampipe -v) v0.16.0-rc.8
Example: v0.3.0

Plugin version (steampipe plugin list) 0.5.0
Example: v0.5.0

To reproduce
Steps to reproduce the behavior (please include relevant code and/or commands).

run below query in amazon linux 2

select grant_id,
resource_id,
identity_profile_id,
identity_display_name,
identity_email,
identity_status,
level_uri,
resource_type_uri,
workspace,
create_timestamp
from nonprod.turbot_active_grant
where resource_trunk_title = 'turbot'
and level_trunk_title = 'superuser'
limit 10;

Expected behavior
A clear and concise description of what you expected to happen.

retrun limited rows in few seconds

Additional context
Add any other context about the problem here.

@nivethj nivethj added the bug Something isn't working label Aug 11, 2022
@dboeke
Copy link
Contributor

dboeke commented Aug 11, 2022

I find when I do this query on an environment with user mode enabled that it results in many duplicate records. I wonder of that is the underlying cause of the issue?

@dboeke
Copy link
Contributor

dboeke commented Aug 11, 2022

select 
  grant_id,
  resource_id,
  identity_profile_id,
  identity_display_name,
  identity_email,
  identity_status,
  level_uri,
  resource_type_uri,
  workspace,
  create_timestamp,
  resource_trunk_title
from 
  turbot_active_grant
where 
  resource_type_uri = 'tmod:@turbot/turbot#/resource/types/turbot'
  and level_uri = 'tmod:@turbot/turbot-iam#/permission/levels/superuser'

@dboeke
Copy link
Contributor

dboeke commented Aug 11, 2022

both of the where constraints should hit indexes

@misraved
Copy link
Contributor

Hello @nivethj, I tried to replicate the issue but without much success.

To replicate the issue, could you please provide a few more details -

  • How many plugins other than turbot are you using in the Amazon Linux 2 environment?
  • Do other queries work or is it only the turbot_active_grant table queries which are failing?
  • If you restart steampipe and rerun the query, do you still see the issue?

@nivethj
Copy link
Author

nivethj commented Sep 7, 2022

Below are the requested details

How many plugins other than turbot are you using in the Amazon Linux 2 environment?
I’m only using only turbot plugin. I have AWS plugin installed , but not using it.

Do other queries work or is it only the turbot_active_grant table queries which are failing?
I haven’t worked on other tables . turbot_active_grant is working in sandbox which has few rows (<20 rows)

If you restart steampipe and rerun the query, do you still see the issue?
Restart is not helping . query runs for 15 mins and end with error

@misraved
Copy link
Contributor

Hey @nivethj, sorry for the delayed response.

Sorry to see that you are still blocked with this issue. We released a new version of the turbot plugin (v0.9.0) which includes essential memory management fixes.

Could you please upgrade the plugin and check if the issue persists?

@cbruno10
Copy link
Contributor

@nivethj To add onto @misraved 's comment, can you please also ensure that you're running Steampipe CLI v0.16.1? This CLI has some updates to allow it to better handle the memory management improvements mentioned. Thanks!

@nivethj
Copy link
Author

nivethj commented Sep 19, 2022

updated steampipe to (steampipe version 0.16.3). turbot plugin to 0.9.0.

row limit 10 is working with 60% Mem , when i update row limit 100 , max memory reached

@github-actions
Copy link

'This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

@github-actions github-actions bot added the stale No recent activity has been detected on this issue/PR and it will be closed label Oct 19, 2022
@nivethj
Copy link
Author

nivethj commented Oct 25, 2022

im following this request

@misraved misraved removed the stale No recent activity has been detected on this issue/PR and it will be closed label Oct 25, 2022
@misraved
Copy link
Contributor

misraved commented Nov 2, 2022

Hello @nivethj, we are extremely sorry for the delay.

While running the query, could you please grab the plugin logs for us to investigate? I tried to replicate this error but without any success.

Hoping to catch something unusual in those logs which might help us in debugging this issue.

@nivethj
Copy link
Author

nivethj commented Nov 30, 2022

It ran long and timed out.

Console

limit 10;
Error: Post "https://server/api/latest/graphql": net/http: TLS handshake timeout (SQLSTATE HV000)

Log file

2022-11-29 19:59:22.753 UTC [16105] STATEMENT: select grant_id,
resource_id,
identity_profile_id,
identity_display_name,
identity_email,
identity_status,
level_uri,
resource_type_uri,
workspace,
create_timestamp
from nonprod.turbot_active_grant
where resource_trunk_title = 'turbot'
and level_trunk_title = 'superuser'
limit 10;
2022-11-29 20:00:38.460 UTC [16105] LOG: disconnection: session time: 3:58:52.816 user=steampipe database=steampipe host=127.0.0.1 port=37158

@misraved
Copy link
Contributor

Apologies for the radio silence on this issue @nivethj!!

Are you still seeing this error with the new v0.19.5 CLI and the latest turbot plugin versions?

@nivethj
Copy link
Author

nivethj commented Jun 6, 2023

tried on v0.20.5 . still the same issue

Welcome to Steampipe v0.20.5
For more information, type .help
steampipe-postgres-fdw updated to 1.7.2.

=> Same query used

net/http: TLS handshake timeout (SQLSTATE HV000)

@bigdatasourav
Copy link
Contributor

Hello @nivethj, I attempted to replicate the issue, but unfortunately, I couldn't succeed. Could you kindly clarify whether by 20+k records you are referring to 20+k resources or active grants? If it pertains to 20+k resources, please provide me with the number of active grants you currently have.

@bigdatasourav
Copy link
Contributor

We have tested in a similar env with 20+K active grants, with limits 10, 100, 1000 - (Works fine)

Welcome to Steampipe v0.20.7
For more information, type .help
> select grant_id,
resource_id,
identity_status,
level_uri,
resource_type_uri,
workspace,
create_timestamp
from turbot_active_grant
limit 10;
+-----------------+-----------------------------------------------------+--------------------------------------------+----------------------+
| identity_status | level_uri                                           | resource_type_uri                          | create_timestamp     |
+-----------------+-----------------------------------------------------+--------------------------------------------+----------------------+
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2021-06-15T17:50:36Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/turbot#/resource/types/turbot | 2022-02-22T07:39:50Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2021-08-20T11:21:59Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2022-02-03T18:54:05Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2021-10-19T19:52:26Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2022-01-27T02:56:21Z |
+-----------------+-----------------------------------------------------+--------------------------------------------+----------------------+
 select
identity_status,
level_uri,
resource_type_uri,
create_timestamp
from turbot_active_grant
limit 100;
+-----------------+-----------------------------------------------------+--------------------------------------------+----------------------+
| identity_status | level_uri                                           | resource_type_uri                          | create_timestamp     |
+-----------------+-----------------------------------------------------+--------------------------------------------+----------------------+
| Inactive        | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account   | 2022-02-26T07:56:14Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2021-06-15T17:50:36Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2021-08-20T11:21:59Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2021-10-19T19:52:26Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2022-01-27T02:56:21Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2022-02-03T18:54:05Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/turbot#/resource/types/turbot | 2022-02-22T07:39:50Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Inactive        | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
 select
identity_status,
level_uri,
resource_type_uri,
create_timestamp
from turbot_active_grant
limit 1000;
+-----------------+-----------------------------------------------------+-----------------------------------------------+----------------------+
| identity_status | level_uri                                           | resource_type_uri                             | create_timestamp     |
+-----------------+-----------------------------------------------------+-----------------------------------------------+----------------------+
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:56:14Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/user     | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:05Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot    | 2021-06-15T17:50:36Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/user     | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:06Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot    | 2021-08-20T11:21:59Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot    | 2021-10-19T19:52:26Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot    | 2022-01-27T02:56:21Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:13Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot    | 2022-02-03T18:54:05Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:13Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:15Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/turbot#/resource/types/turbot    | 2022-02-22T07:39:50Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:19Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:30Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:45Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:47Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:51Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:58Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:56:00Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:56:03Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:56:04Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:56:13Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/user     | tmod:@turbot/aws#/resource/types/region       | 2022-02-26T07:56:13Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:56:14Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/user     | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |

Could you please update the CLI version to the latest and try?

@bigdatasourav
Copy link
Contributor

Hey @nivethj, We are closing this issue because we have not heard from you. Please feel free to reopen the issue if you want to share or discuss anything.

@bigdatasourav bigdatasourav closed this as not planned Won't fix, can't repro, duplicate, stale Jul 10, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants