Support Include/Exclude Lists

[vrajat]

Athena is working for shallow type scan. For deep scan it throws below error:
HIVE_UNKNOWN_ERROR: All access to this object has been disabled (Service: Amazon S3; Status Code: 403; Error Code: AllAccessDisabled; Request ID:

Is there any parameter in config where we can pass specific DB to scan ?

Originally posted by @jayeshagwan1 in #55 (comment)

[jayeshagwan1]

Thanks. Parameter in config will work na ?

[jayeshagwan1]

I tried providing schema and table name to check if it works. I am getting error :

raise OperationalError(query_execution.state_change_reason) pyathena.error.OperationalError: SYNTAX_ERROR: line 1:8: Column 'request_timestamp' cannot be resolved

I don't request_timestamp column.

[vrajat]

I just merged it to master. Can you downloading the code and run python setup.py develop in virtualenv ?

--help should provide enough docs. Config files are also supported

[jayeshagwan1]

After python setup.py develop, tried running help, but not getting updated help:

[vrajat]

oh - can you try piicatcher db --help ? I have forgotten to support it in aws. I'll add it asap

[jayeshagwan1]

My config, I added
list_all=True
schema="temp"
table="pii_data"

Tried running with config:

[vrajat]

Yea- I forgot to add it to aws. Fixing that now.

[jayeshagwan1]

piicatcher db --help

For db its working fine 👍

Also is the above mentioned params for config are correct ?

[vrajat]

yes. config looks good.

[jayeshagwan1]

Getting below error:
current_schema.tables.append(current_table) AttributeError: 'Schema' object has no attribute 'tables'

[vrajat]

is there a line number ? Stack trace ?

[jayeshagwan1]

[vrajat]

I found a gap in my testing. I'll add a test and fix the bug.