AWS config issue

[jayeshagwan1]

I have created AWS config file as mentioned in docs and trying command piicatcher --config awsconfig.ini aws similar to db one.

piicatcher --config dbconfig.ini db works fine.

Getting below error:

Below is my config file :

[aws]
access_key="XXXXXX"
secret_key="XXXXX"
staging_dir=""
region="XXXXX"
output_format="glue"
scan_type="deep"
output="XXXXXX"
list_all=True

[vrajat]

I cannot see what is wrong with you ini file. It looks fine to me. However, the following worked for me:

[aws]
access_key="AAAA"
list_all=True
output_format="json"
region="us-east"
scan_type="deep"
secret_key="SSSS"
staging_dir="s3://dir"

I dont see a valid difference. Can you make sure you have the latest version ?

[jayeshagwan1]

Sorry my bad. Mistake from my side. Was having -(hyphen) instead of _(underscore) in params.

[jayeshagwan1]

Getting error while trying to connect :

[vrajat]

This looks like a permissions issue in AWS.

…

On Wed, Jan 29, 2020 at 2:48 PM jayeshagwan1 ***@***.***> wrote: Getting error while trying to connect : [image: image] <https://user-images.githubusercontent.com/616970/73343282-5669cc80-42a6-11ea-9f14-6ddd5b32c966.png> — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=AAMP7GSUIGFKTLFXN5CWYPDRAFCWFA5CNFSM4KMOA5G2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKGP5KQ#issuecomment-579665578>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMP7GQL7373DUKJJESWEATRAFCWFANCNFSM4KMOA5GQ> .

[jayeshagwan1]

Apart from above parms, do we need to add any specific params for permission or access ?

[vrajat]

No no. The keys dont have access to S3 bucket. So you have to fix the permissions in AWS IAM console. One way to check is to run https://github.com/tokern/piicatcher/blob/master/piicatcher/explorer/aws.py#L44 in Athena. Also run https://github.com/tokern/piicatcher/blob/master/piicatcher/explorer/aws.py#L53 for every table. My guess is that the keys you are using dont have access to one of the tables or the S3 bucket where the table data is stored.

…

On Wed, Jan 29, 2020 at 3:03 PM jayeshagwan1 ***@***.***> wrote: Apart from above parms, do we need to add any specific params for permission or access ? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=AAMP7GVQMA3AB7YEXKSY3NDRAFEPLA5CNFSM4KMOA5G2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKGRJ2I#issuecomment-579671273>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMP7GWCEGTERSWGMB6GUYLRAFEPLANCNFSM4KMOA5GQ> .

[jayeshagwan1]

L44 query results

	table_schema	table_name	column_name
	table_schema	table_name	column_name
1	sampledb	elb_logs	request_timestamp
2	sampledb	elb_logs	elb_name
3	sampledb	elb_logs	request_ip
4	sampledb	elb_logs	backend_ip
5	sampledb	elb_logs	elb_response_code
6	sampledb	elb_logs	backend_response_code
7	sampledb	elb_logs	request_verb
8	sampledb	elb_logs	url
9	sampledb	elb_logs	protocol
10	sampledb	elb_logs	user_agent
11	sampledb	elb_logs	ssl_cipher
12	sampledb	elb_logs	ssl_protocol
13	temp	pii_data	phone
14	temp	pii_data	email
15	temp	pii_data	credit_card
16	temp	pii_data	address
17	temp	pii_data	person
18	temp	pii_data	location
19	temp	pii_data	gender
20	temp	pii_data	nationality
21	temp	pii_data	ip_address
22	temp	pii_data	ssn
23	testdb	test5	tst6
24	testdb	test6	tst7
25	testdb	test8	tst2
26	testdb	test9	tst11

[vrajat]

What about the other query ? Can you run it on each of the tables ?

…

On Wed, Jan 29, 2020 at 4:02 PM jayeshagwan1 ***@***.***> wrote: L44 query results table_schema table_name column_name table_schema table_name column_name 1 sampledb elb_logs request_timestamp 2 sampledb elb_logs elb_name 3 sampledb elb_logs request_ip 4 sampledb elb_logs backend_ip 5 sampledb elb_logs elb_response_code 6 sampledb elb_logs backend_response_code 7 sampledb elb_logs request_verb 8 sampledb elb_logs url 9 sampledb elb_logs protocol 10 sampledb elb_logs user_agent 11 sampledb elb_logs ssl_cipher 12 sampledb elb_logs ssl_protocol 13 temp pii_data phone 14 temp pii_data email 15 temp pii_data credit_card 16 temp pii_data address 17 temp pii_data person 18 temp pii_data location 19 temp pii_data gender 20 temp pii_data nationality 21 temp pii_data ip_address 22 temp pii_data ssn 23 testdb test5 tst6 24 testdb test6 tst7 25 testdb test8 tst2 26 testdb test9 tst11 — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=AAMP7GSMGWH4AW5AQVT2NLTRAFLMTA5CNFSM4KMOA5G2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKGW2JI#issuecomment-579693861>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMP7GQCFVB2IBYBWG6XHCTRAFLMTANCNFSM4KMOA5GQ> .

[jayeshagwan1]

Athena is working for shallow type scan. For deep scan it throws below error:
HIVE_UNKNOWN_ERROR: All access to this object has been disabled (Service: Amazon S3; Status Code: 403; Error Code: AllAccessDisabled; Request ID:

Is there any parameter in config where we can pass specific DB to scan ?

[vrajat]

I've filed #56. I'll check the best way to support filters and add it.