Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CSV plugin is not reliable in loading tables in steampipe query with v0.20.x #3483

Closed
e-gineer opened this issue May 26, 2023 · 2 comments · Fixed by #3504
Closed

CSV plugin is not reliable in loading tables in steampipe query with v0.20.x #3483

e-gineer opened this issue May 26, 2023 · 2 comments · Fixed by #3504
Assignees
@kaidaguerre
Labels
bug Something isn't working

Comments

@e-gineer
Copy link
Contributor

I have a directory with a few CSV files, one of which is not valid (in fact it's JSON):

/tmp/crap $ ls
country.csv      json-not-csv.csv state.csv
/tmp/crap $ cat country.csv 
name,country
nathan,US
bob,US
/tmp/crap $ cat state.csv 
name,state
nathan,NJ
bob,PA
/tmp/crap $ cat json-not-csv.csv 
{
  "I'm": "json",
  "not": "valid CSV"
}
/tmp/crap $

I used JSON with a bad file extension just as an example. I'm not sure it matters that it is JSON. I'm not sure it even matters to have a bad CSV file for this bug report ... it's just what I was testing.

Now ... I'm finding that when I switch between directories Steampipe is unreliable at reading in the CSV plugin and tables. I see different results at different tries.

Here is a dump of the flow & attempts. Notably once it settles (which is quick) it doesn't change for that session. Sometimes it takes a moment in a session to refresh, which is understandable.

/tmp/crap $ cd ..
/tmp $ steampipe query
Welcome to Steampipe v0.20.2
For more information, type .help
> .inspect csv
+----------------+------------------------------------------+
| table          | description                              |
+----------------+------------------------------------------+
| country        | CSV file at /tmp/crap/country.csv        |
| json-not-csv-2 | CSV file at /tmp/crap/json-not-csv-2.csv |
| state          | CSV file at /tmp/crap/state.csv          |
+----------------+------------------------------------------+
> .inspect csv
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| connection | plugin                                     | schema mode | state | error | state updated             |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | ready |       | 2023-05-26T14:51:53-04:00 |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
> .inspect csv
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| connection | plugin                                     | schema mode | state | error | state updated             |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | ready |       | 2023-05-26T14:51:53-04:00 |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
> 
/tmp $ cd crap
/tmp/crap $ steampipe query
Welcome to Steampipe v0.20.2
For more information, type .help
> .inspect csv
+------------+--------------------------------------------+-------------+----------+-------+---------------------------+
| connection | plugin                                     | schema mode | state    | error | state updated             |
+------------+--------------------------------------------+-------------+----------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | updating |       | 2023-05-26T14:52:03-04:00 |
+------------+--------------------------------------------+-------------+----------+-------+---------------------------+
> .inspect csv
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| connection | plugin                                     | schema mode | state | error | state updated             |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | ready |       | 2023-05-26T14:52:03-04:00 |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
> .inspect csv
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| connection | plugin                                     | schema mode | state | error | state updated             |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | ready |       | 2023-05-26T14:52:03-04:00 |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
> .inspect csv
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| connection | plugin                                     | schema mode | state | error | state updated             |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | ready |       | 2023-05-26T14:52:03-04:00 |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
> .inspect csv
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| connection | plugin                                     | schema mode | state | error | state updated             |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | ready |       | 2023-05-26T14:52:03-04:00 |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
> .inspect csv
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| connection | plugin                                     | schema mode | state | error | state updated             |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | ready |       | 2023-05-26T14:52:03-04:00 |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
> .inspect csv
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| connection | plugin                                     | schema mode | state | error | state updated             |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | ready |       | 2023-05-26T14:52:03-04:00 |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
> 


/tmp/crap $ 
/tmp/crap $ 
/tmp/crap $ steampipe query
Welcome to Steampipe v0.20.2
For more information, type .help
> .inspect csv
+----------------+------------------------------------------+
| table          | description                              |
+----------------+------------------------------------------+
| country        | CSV file at /tmp/crap/country.csv        |
| json-not-csv-2 | CSV file at /tmp/crap/json-not-csv-2.csv |
| state          | CSV file at /tmp/crap/state.csv          |
+----------------+------------------------------------------+
> .inspect csv
+----------------+------------------------------------------+
| table          | description                              |
+----------------+------------------------------------------+
| country        | CSV file at /tmp/crap/country.csv        |
| json-not-csv-2 | CSV file at /tmp/crap/json-not-csv-2.csv |
| state          | CSV file at /tmp/crap/state.csv          |
+----------------+------------------------------------------+
> 

/tmp/crap $ 
/tmp/crap $ cd ..
/tmp $ steampipe query
Welcome to Steampipe v0.20.2
For more information, type .help
> .inspect csv
+----------------+------------------------------------------+
| table          | description                              |
+----------------+------------------------------------------+
| country        | CSV file at /tmp/crap/country.csv        |
| json-not-csv-2 | CSV file at /tmp/crap/json-not-csv-2.csv |
| state          | CSV file at /tmp/crap/state.csv          |
+----------------+------------------------------------------+
> .inspect csv
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| connection | plugin                                     | schema mode | state | error | state updated             |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | ready |       | 2023-05-26T14:52:23-04:00 |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
> .inspect csv
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| connection | plugin                                     | schema mode | state | error | state updated             |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | ready |       | 2023-05-26T14:52:23-04:00 |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
> 

/tmp $ 
/tmp $ cd crap
/tmp/crap $ steampipe query
Welcome to Steampipe v0.20.2
For more information, type .help
> .inspect csv
+------------+--------------------------------------------+-------------+----------+-------+---------------------------+
| connection | plugin                                     | schema mode | state    | error | state updated             |
+------------+--------------------------------------------+-------------+----------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | updating |       | 2023-05-26T14:52:32-04:00 |
+------------+--------------------------------------------+-------------+----------+-------+---------------------------+
> .inspect csv
+----------------+-------------+
| table          | description |
+----------------+-------------+
| country        |             |
| json-not-csv-2 |             |
| state          |             |
+----------------+-------------+
> .inspect csv
+----------------+-------------+
| table          | description |
+----------------+-------------+
| country        |             |
| json-not-csv-2 |             |
| state          |             |
+----------------+-------------+
>

Even more interesting is the connection state. Notice how in the bottom examples the hash doesn't change but the loaded tables do?

/tmp $ steampipe query
Welcome to Steampipe v0.20.2
For more information, type .help
> select * from steampipe_connection_state where name = 'csv'
+------+-------+------+---------------+--------+--------------------------------------------+-------------+----------------------------------+--------------+---------------------------+---------------------------+
| name | state | type | import_schema | error  | plugin                                     | schema_mode | schema_hash                      | comments_set | connection_mod_time       | plugin_mod_time           |
+------+-------+------+---------------+--------+--------------------------------------------+-------------+----------------------------------+--------------+---------------------------+---------------------------+
| csv  | ready |      | enabled       | <null> | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | d41d8cd98f00b204e9800998ecf8427e | true         | 2023-05-26T14:57:06-04:00 | 2023-05-15T09:04:32-04:00 |
+------+-------+------+---------------+--------+--------------------------------------------+-------------+----------------------------------+--------------+---------------------------+---------------------------+
> .inspect csv
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| connection | plugin                                     | schema mode | state | error | state updated             |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | ready |       | 2023-05-26T14:57:06-04:00 |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
> 
/tmp $ cd crap
/tmp/crap $ steampipe query
Welcome to Steampipe v0.20.2
For more information, type .help
> select * from steampipe_connection_state where name = 'csv'
+------+-------+------+---------------+--------+--------------------------------------------+-------------+----------------------------------+--------------+---------------------------+---------------------------+
| name | state | type | import_schema | error  | plugin                                     | schema_mode | schema_hash                      | comments_set | connection_mod_time       | plugin_mod_time           |
+------+-------+------+---------------+--------+--------------------------------------------+-------------+----------------------------------+--------------+---------------------------+---------------------------+
| csv  | ready |      | enabled       | <null> | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | 6c368b2c8e93381d7c35de58620b72ea | true         | 2023-05-26T14:57:20-04:00 | 2023-05-15T09:04:32-04:00 |
+------+-------+------+---------------+--------+--------------------------------------------+-------------+----------------------------------+--------------+---------------------------+---------------------------+
> .inspect csv
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| connection | plugin                                     | schema mode | state | error | state updated             |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
| csv        | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | ready |       | 2023-05-26T14:57:20-04:00 |
+------------+--------------------------------------------+-------------+-------+-------+---------------------------+
> 
/tmp/crap $ steampipe query
Welcome to Steampipe v0.20.2
For more information, type .help
> select * from steampipe_connection_state where name = 'csv'
+------+-------+------+---------------+--------+--------------------------------------------+-------------+----------------------------------+--------------+---------------------------+---------------------------+
| name | state | type | import_schema | error  | plugin                                     | schema_mode | schema_hash                      | comments_set | connection_mod_time       | plugin_mod_time           |
+------+-------+------+---------------+--------+--------------------------------------------+-------------+----------------------------------+--------------+---------------------------+---------------------------+
| csv  | ready |      | enabled       | <null> | hub.steampipe.io/plugins/turbot/csv@latest | dynamic     | 6c368b2c8e93381d7c35de58620b72ea | true         | 2023-05-26T14:57:43-04:00 | 2023-05-15T09:04:32-04:00 |
+------+-------+------+---------------+--------+--------------------------------------------+-------------+----------------------------------+--------------+---------------------------+---------------------------+
> .inspect csv
+----------------+------------------------------------------+
| table          | description                              |
+----------------+------------------------------------------+
| country        | CSV file at /tmp/crap/country.csv        |
| json-not-csv-2 | CSV file at /tmp/crap/json-not-csv-2.csv |
| state          | CSV file at /tmp/crap/state.csv          |
+----------------+------------------------------------------+
> 
/tmp/crap $
@e-gineer e-gineer added the bug Something isn't working label May 26, 2023
@e-gineer
Copy link
Contributor Author

This issue may be related to, and perhaps an easier way to reproduce, issue #3482

@e-gineer
Copy link
Contributor Author

Noting that using an empty.csv file (zero bytes) is also helpful to make this problem worse / more reproducable.

@kaidaguerre kaidaguerre linked a pull request May 31, 2023 that will close this issue
Ensure 'Ready' connections are set to 'Pending' state on startup. Fixes #3483 #3504
Merged
kaidaguerre added a commit that referenced this issue May 31, 2023
@kaidaguerre
Ensure 'Ready' connections are set to 'Pending' state on startup. Fixes
692f3e7 
#3483
kaidaguerre added a commit that referenced this issue May 31, 2023
@kaidaguerre
Ensure 'Ready' connections are set to 'Pending' state on startup. Fixes
136ab15 
#3483
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kaidaguerre kaidaguerre

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@e-gineer @kaidaguerre