Deep Scan in Athena failing with HIVE_UNKNOWN_ERROR

[vrajat]

Yeah.. here its is...
debug_log.txt

Originally posted by @jayeshagwan1 in #59 (comment)

[vrajat]

@jayeshagwan1 can you run select count(*) from temp.pii_data in Athena console ?

Also the debug log seems to be truncated. Can you make sure that it contains all of the output ?

[jayeshagwan1]

I am having complete log, I have just scrapped and provided you a a certain portion. Will provide the complete log file in some time.

[vrajat]

I want to check these types of lines:

DEBUG:root:Count Query: select count(*) from temp.pii_data 
DEBUG:pyathena.common:select count(*) from temp.pii_data

[jayeshagwan1]

Not sure, but will this work ?
logs.txt

[vrajat]

I still see only select count(*) from temp.pii_data in the logs. I dont see any exceptions or errors. I want to know the last query that was run before the exception or error happened

[jayeshagwan1]

Sorry. Here is the log file if I dont hard code schema and table and just do select *
error_Log.txt

[jayeshagwan1]

Even after passing schema and table from config, its still looking into all tables. It should fetch only mentioned schema and table.

[vrajat]

Can you test this query in athena console ? select * from sampledb.elb_logs TABLESAMPLE BERNOULLI(5) limit 10 ?

sampledb.elb_logs is created automatically and I think this is not setup properly or your account doesnt have access.

can you try piicatcher ... --exclude-schema sampledb?

I'll look into "Even after passing schema and table from config, its still looking into all tables. It should fetch only mentioned schema and table." separately

[jayeshagwan1]

piicatcher ... --exclude-schema sampledb throwing error:
error.txt

[vrajat]

I just released a new version. Mostly with debug logging improvements. I do not know if it will fix all your issues. Please give it a try without any edits. Also please provide any debug logs for any issues.

On filtering tables, check my comment in #65 on the right syntax.

[jayeshagwan1]

In new version its working fine.
Some of columns are not getting identified as required PII data type.

[vrajat]

Can you provide examples in a new issue?

…

On Thu, 13 Feb 2020 at 18:23, jayeshagwan1 ***@***.***> wrote: In new version its working fine. Some of columns are not getting identified as required PII data type. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#63?email_source=notifications&email_token=AAMP7GQYMXIF7QFHAFWTUMDRCU7EZA5CNFSM4KUJXTV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELU3DXQ#issuecomment-585740766>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMP7GSYHZNTDNBSKSWS3O3RCU7EZANCNFSM4KUJXTVQ> .

[jayeshagwan1]

Will this work ? Deep scan and shallow scan results are different.
deepscan.txt
shallowscan.txt