Connector for Google Drive #294

HAKSOAT · 2023-02-27T01:32:50Z

Implements enhancement #244

HAKSOAT · 2023-03-03T00:19:52Z

Hi, @cragwolfe I'd appreciate it if a review could begin on my PR.

Everything works as expected from running it, but I am looking to do two more things:

Integrate Google's Auth ID (so files can be downloaded without making them available for everyone to view) and find a way to allow downloads on public files without a token.

I'm not sure why the CI CD checks are not running at the moment.

No tests at the moment since API keys are needed (I'm not sure how to go about creating a test for that without exposing keys).

cragwolfe · 2023-03-03T08:15:56Z

Hi, @cragwolfe I'd appreciate it if a review could begin on my PR.

Hi @HAKSOAT , happy to! Great to hear you got this working.

Everything works as expected from running it, but I am looking to do two more things:

Integrate Google's Auth ID (so files can be downloaded without making them available for everyone to view) and find a way to allow downloads on public files without a token.

Does that mean following the credentials.json flow described here: https://developers.google.com/drive/api/quickstart/python ?

I'm not sure why the CI CD checks are not running at the moment.

I think it was mostly the gihub outage. I just merged into Main to help trigger that (hope that's OK!).

No tests at the moment since API keys are needed (I'm not sure how to go about creating a test for that without exposing keys).

How does one go about getting an API Key?

A few more things to address:

Even though it needs to be verified manually, please add a test script under https://github.com/Unstructured-IO/unstructured/tree/main/examples/ingest/google-drive/ . The script itself can provide guidance on how to get authentication credentials. [it's there, missed it!]
Please update CHANGELOG.md and unstructured/__version__.py.
Running make tidy and make check should help you past the CI lint issue.

cragwolfe · 2023-03-03T08:20:25Z

unstructured/ingest/connector/google_drive.py

+            with open(self.filename, "wb") as handler:
+                handler.write(file.getbuffer())
+                if self.config.verbose:
+                    print(f"File downloaded: {self.filename}.")


what would be the else condition (i.e. if not file)? should we print a warning if unexpected? (someday soon this would be "log a warning" but logging in multiprocessing is not working yet).

unstructured/ingest/connector/google_drive.py

HAKSOAT · 2023-03-03T22:39:51Z

Hi @cragwolfe

Thanks for the review. I'll go through your comments now.

Does that mean following the credentials.json flow described here: https://developers.google.com/drive/api/quickstart/python ?

More like here:

https://developers.google.com/workspace/guides/create-credentials

I believe a service account is better as it is made for robot accounts. The reason is that the regular API key gets throttled as I experienced an error after a couple of requests that says:

We're sorry...
... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.
See Google Help for more information.

…ured into ingest_connector/gdrive

HAKSOAT · 2023-03-04T02:24:12Z

Hi @cragwolfe

I think I just hit a roadblock for service accounts. Looks like there's an issue with using the service account approach with multiprocessing.

Traceback (most recent call last):
  File "/mnt/c/Users/HAKS/dev/unstructured/./unstructured/ingest/main.py", line 362, in <module>
    main()
  File "/home/haks/Envs/unstructured/lib/python3.9/site-packages/click/core.py", line 1126, in __call__
    return self.main(*args, **kwargs)
  File "/home/haks/Envs/unstructured/lib/python3.9/site-packages/click/core.py", line 1051, in main
    rv = self.invoke(ctx)
  File "/home/haks/Envs/unstructured/lib/python3.9/site-packages/click/core.py", line 1393, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/haks/Envs/unstructured/lib/python3.9/site-packages/click/core.py", line 752, in invoke
    return __callback(*args, **kwargs)
  File "/mnt/c/Users/HAKS/dev/unstructured/./unstructured/ingest/main.py", line 353, in main
    MainProcess(
  File "/mnt/c/Users/HAKS/dev/unstructured/./unstructured/ingest/main.py", line 73, in run
    results = pool.map(self.doc_processor_fn, docs)  # noqa: F841
  File "/home/haks/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/haks/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/haks/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/pool.py", line 537, in _handle_tasks
    put(task)
  File "/home/haks/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/connection.py", line 211, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/haks/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_cffi_backend.FFI' object

The traceback isn't so helpful either. Just putting this here to serve as an update as I have tried to find a fix but I am yet to find one. I will take another look in the morning here.

cragwolfe · 2023-03-04T23:00:39Z

I think I just hit a roadblock for service accounts. Looks like there's an issue with using the service account approach with multiprocessing.

Yes, looks that way. A way around that is to remove self.service from the config object, and only create the service object on demand (it looks like the api_key, and drive_id are sufficient to build the service object).

HAKSOAT · 2023-03-05T19:28:10Z

Yes, looks that way. A way around that is to remove self.service from the config object, and only create the service object on demand (it looks like the api_key, and drive_id are sufficient to build the service object).

Thanks for the hint @cragwolfe It works now. I think a second review will be great so I can fix any other issue.

HAKSOAT · 2023-03-07T00:16:19Z

Hi @cragwolfe please can you help take a look at this when chanced?

cragwolfe · 2023-03-07T00:35:29Z

Hi @cragwolfe please can you help take a look at this when chanced?
For sure, on my list for today!

cragwolfe

Works very nicely! 🚀

Though, I did get tripped up by not reading the Note in ingest.sh carefully enough the first time:

# NOTE: Using the Service Account key only works when the file or folder
# is shared atleast with permission for "Anyone with the link" to view
# OR the email address for the service account is given access to the file
# or folder.

service account emails do not look like user emails.

HAKSOAT added 10 commits February 27, 2023 01:27

Setup interface functions to Google Drive

f71c6da

Merge branch 'main' into ingest_connector/gdrive

49c2931

Setup ingestdoc class

5517280

Added support for recursive file downloads

aa3f180

Fixed errors and put up ingest script

a388a37

Setup dependencies and Makefile commands

243fc7a

Merge branch 'main' into ingest_connector/gdrive

8135db2

Setup requires_dependencies

edbc820

Added file export functionality

3d463cd

Added extension filter

189dd22

HAKSOAT marked this pull request as ready for review March 3, 2023 00:12

HAKSOAT and others added 2 commits March 3, 2023 00:25

Added Google filetypes

c7feb81

Merge branch 'main' into ingest_connector/gdrive

19ae943

cragwolfe reviewed Mar 3, 2023

View reviewed changes

unstructured/ingest/connector/google_drive.py Show resolved Hide resolved

cragwolfe reviewed Mar 3, 2023

View reviewed changes

unstructured/ingest/connector/google_drive.py Outdated Show resolved Hide resolved

HAKSOAT added 3 commits March 4, 2023 01:29

Fixed linting errors

b6987be

Merge branch 'ingest_connector/gdrive' of github.com:HAKSOAT/unstruct…

672678b

…ured into ingest_connector/gdrive

Updated the changelog

d85aec1

Fixed issue with pickling and service object

a60697c

Merge branch 'main' into ingest_connector/gdrive

d8cbe75

cragwolfe approved these changes Mar 7, 2023

View reviewed changes

Merge branch 'main' into ingest_connector/gdrive

4d52d39

cragwolfe enabled auto-merge (squash) March 7, 2023 05:52

cragwolfe merged commit 4117f57 into Unstructured-IO:main Mar 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connector for Google Drive #294

Connector for Google Drive #294

HAKSOAT commented Feb 27, 2023

HAKSOAT commented Mar 3, 2023

cragwolfe commented Mar 3, 2023 •

edited

Loading

cragwolfe Mar 3, 2023 •

edited

Loading

HAKSOAT commented Mar 3, 2023

We're sorry...

HAKSOAT commented Mar 4, 2023

cragwolfe commented Mar 4, 2023

HAKSOAT commented Mar 5, 2023

HAKSOAT commented Mar 7, 2023

cragwolfe commented Mar 7, 2023

cragwolfe left a comment

Connector for Google Drive #294

Connector for Google Drive #294

Conversation

HAKSOAT commented Feb 27, 2023

HAKSOAT commented Mar 3, 2023

cragwolfe commented Mar 3, 2023 • edited Loading

cragwolfe Mar 3, 2023 • edited Loading

Choose a reason for hiding this comment

HAKSOAT commented Mar 3, 2023

We're sorry...

HAKSOAT commented Mar 4, 2023

cragwolfe commented Mar 4, 2023

HAKSOAT commented Mar 5, 2023

HAKSOAT commented Mar 7, 2023

cragwolfe commented Mar 7, 2023

cragwolfe left a comment

Choose a reason for hiding this comment

cragwolfe commented Mar 3, 2023 •

edited

Loading

cragwolfe Mar 3, 2023 •

edited

Loading