🔍 Seeking Help with BigQuery Integration and Archiver Behavior

[dschwalb]

Hi,
I'm trying to load the results into BigQuery instead of a CSV file (as originally noted), but I'm encountering an issue. Here's the command I'm using:

cs_tools tools archiver identify --content ALL --recent-activity 180 --recent-modified 180 --dry-run --syncer "bigquery://syncer-bigquery.toml" --config lightricks-config

Here's the relevant part of the TOML file:

[configuration]
project_id = "ltx-dwh-stg-processed"
dataset = "dbt_dschwalb"
load_strategy = "APPEND"

And here's part of the error message:

FileNotFoundError: [Errno 2] No such file or directory: '/Users/dschwalb/Library/Application Support/cs_tools/.cs_tools/bin/python3.12'

Appreciate your help!
Also, I have a couple of questions about the cs_tools tools archiver remove command:

Does it permanently delete files with no recovery option, or is there some kind of archive or backup?

Does it remove only Liveboards and Answers, or can it also delete models/tables?

I ran the identify command and then opened a Liveboard that had been marked as unseen and unmodified. After that, I ran the command again an hour later, but the same Liveboard was still included in the results, which was unexpected. Could you clarify this behavior?

[TS-sumeet-jain]

Hi @dschwalb ,

Can you check the python version you are using? And try to reinstall the cstools seems like it is not able to find the python.

For the archiver, It only supports liveboards and answers. you can use --export-only option to get the TML files as backup and it generates the file with author information too.

[dschwalb]

hey, My Python Version is 3.12.8

thank you for the help, so the data should be updated immediately or does it take time for the cstools to update?
and theres no place that TS is storing the removed items (archive or something) so we need to take care of that?

the --export-only seems to not work/ exist
[14:01:32] ERROR No such option: --export-only main.py:86

how can i upload the result into big query?

[TS-sumeet-jain]

Can you share the full command you are using?
For uploading the result to BigQuery; you will have to define the biquery.toml and add it to syncer --syncer bigquery://syncer-overwrite.toml

Your TOML file will look like:
[configuration] project_id = "thoughtspot" dataset = "cs_tools" credentials_keyfile = "/path/to/cs_tools/bigquery/credentials.json" load_strategy = "TRUNCATE"

For more info: https://thoughtspot.github.io/cs_tools/syncer/bigquery/

[dschwalb]

Hi, I was able to upload the data to BigQuery, but I haven’t been able to locate the --export-only option. Could you please help me with that?

[TS-sumeet-jain]

The command for exporting the TMLs should be like cstools tools archiver remove --export-only --directory <directory-to-export-TMLs> --config <cstools-config-name>

[dschwalb]

so --export-only is only available in the cstools tools archiver remove command?