feat: add google cloud storage connector #746
Conversation
|
oh, also, some additions to the base class came in very recently. these will be really important for future cases where we need to be thoughtful about how/when things get reprocessed as well as giving users more access to that information. we haven't added support in any of the connectors yet so there aren't good references, but want to blaze the trail on adding these? Specifically, properties supporting: date_modified, date_processed, exists, record_locator, source_url, and version? |
I can take a look at these. But I want to finish this PR first. |
| #!/usr/bin/env bash | ||
|
|
||
| # Processes several files in a nested folder structure from gs://unstructured_public/ | ||
| # through Unstructured's library in 2 processes. | ||
|
|
||
| # Structured outputs are stored in gcs-output/ | ||
|
|
||
| SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) | ||
| cd "$SCRIPT_DIR"/../../.. || exit 1 | ||
|
|
||
| PYTHONPATH=. ./unstructured/ingest/main.py \ | ||
| --remote-url gs://unstructured_public/ \ | ||
| --structured-output-dir gcs-output \ | ||
| --num-processes 2 \ | ||
| --recursive \ | ||
| --verbose |
There was a problem hiding this comment.
I pushed up a gcs test bucket with private and public paths, we should update to use that. I'll follow up with details directly.
There was a problem hiding this comment.
let's use gs://utic-test-ingest-fixtures
There was a problem hiding this comment.
This would require the user to have authentication token. Is that what we really want? Or do we want a no auth example?
There was a problem hiding this comment.
oh sorry, this is for the example. yes, no auth. I'll see if I can set up a public bucket for this.
unstructured/ingest/main.py
Outdated
| @click.option( | ||
| "--gcs-token", | ||
| default=None, | ||
| help="Token used to access Google Cloud. By default shouldn't be needed.", |
There was a problem hiding this comment.
rather than "By default shouldn't be needed" might drop more info that you had in comment below. around using default gcloud credentials - attempt to get credentials from the google metadata service - fall back to anonymous access.
| PYTHONPATH=. ./unstructured/ingest/main.py \ | ||
| --metadata-exclude filename,file_directory,metadata.data_source.date_processed \ | ||
| --remote-url gs://unstructured_public/ \ |
There was a problem hiding this comment.
let's use this test folder: gs://utic-test-ingest-fixtures
it has two folders public and private, where private will require that the auth token is used. hopefully can leverage what was done in the google drive test that I just added?
There was a problem hiding this comment.
Worked. But had to cat the cred info into the tempfile instead of echo is that ok? Could be an OSX bash thing.
There was a problem hiding this comment.
huh, weird. yea, as long as it works locally and in CI, that works for me.
There was a problem hiding this comment.
looking at CI, it doesn't look like linux is happy about the cat change....taking a look
cat: '***'$'\n'' ***'$'\n'' ***'$'\n'' ***'$'\n'' ***'$'\n'' ***'$'\n'' ***'$'\n'' ***'$'\n'' ***'$'\n'' ***'$'\n'' ***'$'\n'' ***'$'\n''***'$'\n': File name too long
https://github.com/Unstructured-IO/unstructured/actions/runs/5330890729/jobs/9658103678
There was a problem hiding this comment.
ah, right, I see the confusion here. (should have noticed before). I think this wasn't working for you locally because you were setting GCP_INGEST_SERVICE_KEY to a filepath. GCP_INGEST_SERVICE_KEY is the actual key itself. This should works with echo on any platform.
bump expected output
update expected files
ryannikolaidis
changed the title
Closes 301
Note: properties supporting date_modified, date_processed, exists, record_locator, source_url, and version will be handled in a follow-on PR.