Add output to Google Drive option #67

aguilerapy · 2020-01-10T16:56:40Z

In #42 we consider adding two input/output options, in this pull request we have the output to google drive option, I will create a different one for the URL input option.

For this to work it's necessary to create credentials for the Google Drive API with an OCP account. Here we have a guide.

Then just configure the environment variable: OCDS_TOUCAN_CREDENTIALS_DRIVE in settings.py to point to a JSON file that contains the credentials.

We can get this JSON file from here:

# Conflicts: # locale/es/LC_MESSAGES/django.po

jpmckinney · 2020-01-10T17:54:11Z

What's the Google integration look like? I figured Toucan would use OAuth to be able to create a spreadsheet in the user's Google account.

aguilerapy · 2020-01-24T14:15:23Z

Yes, Toucan needs OAuth and some other libraries. I add those libraries in requirements.in and I still have the same problem, what can I do? @jpmckinney

jpmckinney · 2020-01-27T17:40:04Z

@aguilerapy You need to run:

pip-compile
pip-compile requirements_dev.in

To update the requirements*.txt files.

https://ocds-standard-development-handbook.readthedocs.io/en/latest/coding/index.html#id1

romifz

There are a few things to improve, but in general looks good!

default/templates/default/base-uploader.html

default/drive_options.py

default/urls.py

romifz · 2020-07-09T02:25:52Z

Regarding credentials, I've created an OAuth2 Client ID selecting the "Web application" type. I'm including here a screenshot for the configuration I used:

The redirect URI must work over https.

Once the user has identified themselves, the credentials are saved for the session.

the code needs to be reviewed again

romifz · 2020-07-09T02:29:55Z

@aguilerapy can you please have a look at the code, and test it locally? I'll send you the test credentials

aguilerapy · 2020-07-16T02:48:33Z

default/google_drive.py

+def upload_to_drive(request):
+
+    datafile = DataFile(**request.session['google_drive_file'])
+    credentials = get_credentials_from_session(request)


Once a credential is entered into toucan, it cannot be changed, we could create an issue to add an option in which the user can change accounts.

Yes, we can leave that for another issue.

aguilerapy · 2020-07-16T02:49:11Z

default/google_drive.py

+
+    try:
+        if credentials:
+            if credentials.expired:


If the credentials expire, it automatically restarts, then the user never enters the credentials again, it's okay?

Yes, the credentials are refreshed. The user does not need to give explicit approval again (as long as their credentials are stored in the session).

aguilerapy · 2020-07-16T02:50:03Z

default/google_drive.py

+        })
+
+    except Exception as e:
+        print(e)


This print is correct here?

No, this should return some kind of error message, I'll fix it.

aguilerapy · 2020-07-16T02:52:39Z

default/static/css/custom.css

 .content-wrap.ocp-content {
    padding-bottom: 2.5rem;
 }
+.link-disabled {


I think this class is no longer used.

Ok! I'll remove it.

aguilerapy · 2020-07-16T02:59:53Z

default/templates/default/snippets/input-options/input_description.html

+<ul class="list-unstyled">
+    <li>
+        <strong>{% trans "Upload a file:" %}</strong>
+        {% trans "Upload no more than one file by dragging and dropping in the box below or using the <strong>Add a file</strong> link." %}


This message is shown in all tools, in some functions we can upload more than one file.

I'll check.

aguilerapy · 2020-07-16T03:00:32Z

default/templates/default/snippets/input-options/input_description.html

+    </li>
+    <li>
+        <strong>{% trans "Provide a URL:" %}</strong>
+        {% trans "Provide a single URL to a file." %}


Same here, in some functions we can provide more than one URL.

aguilerapy · 2020-07-16T03:17:06Z

default/views.py

        prefix = 'flatten-csv'
        ext = '.zip'
-        filename = 'result-csv.zip'
+        filename = 'result.csv.zip'


We must update the file name 'result-csv.zip' in the message box.

aguilerapy · 2020-07-16T03:19:24Z

locale/es/LC_MESSAGES/django.po

+"There was an authorization issue when saving the file to Google Drive, "
+"please try again."
+msgstr "Se encontró un problema de autorización al guardar el archivo en Google Drive, "
+"favor intente de nuevo más tarde"


Debemos sacar "mas tarde" o agregar "later" en la traducción?

Creo que hay que decir "please try again later"

aguilerapy · 2020-07-16T03:25:15Z

locale/es/LC_MESSAGES/django.po

+#: default/templates/default/mapping_sheet.html:53
+#: default/templates/default/to-spreadsheet.html:25
+#: default/templates/default/to-spreadsheet.html:38
+msgid "File saved on Google Drive"


It's better to say "in"?

La verdad no estaba segura, así que le pregunté a Teté y me dijo que es "on".

aguilerapy · 2020-07-24T01:54:42Z

I reviewed the code and tried the new features and I have no further comments.

Do you want to look at it? @jpmckinney

It's convenient to install the sslserver package.

jpmckinney · 2020-07-25T00:13:26Z

I'm happy to trust that it works, given the attention paid to this feature!

For deployment, please provide specific, step-by-step instructions for anything needed from Google.

romifz · 2020-07-27T22:18:10Z

Hi @jpmckinney, I've followed the instructions here to create the credentials ("Enable APIs for your project" and "Create authorization credentials" sections).

In step 4 for "Enable APIs for your project", look for the "Google Drive API" and enable it.
In step 4 of the "Create authorization credentials" section, add the following URI to the "Redirect URIs" section:

https://toucan.open-contracting.org/googleapi-auth-response

After downloading the client_secret.json file, save it in the root directory for the project with the name "googleapi_credentials.json", or set the OCDS_TOUCAN_GOOGLE_API_CREDENTIALS_FILE envvar with the path to the file.

Let me know if you need anything else.

I think we want to follow the verification process since unverified apps have a limit of 100 new users. The only scope we use is https://www.googleapis.com/auth/drive.file.

jpmckinney · 2020-08-08T02:03:21Z

It works! 🎉 🎊

jpmckinney · 2020-08-08T02:07:21Z

I think we want to follow the verification process since unverified apps have a limit of 100 new users. The only scope we use is https://www.googleapis.com/auth/drive.file.

I was able to use the same OAuth secret I had created earlier around May 20, for which I had requested verification. On https://console.cloud.google.com/apis/credentials/consent?project=toucan-277800 it says:

Verification Status: Being verified
The Trust and Safety team has received your form. They will reach out to you via your contact email if needed. The review process can take up to 4-6 weeks. Your last approved consent screen is still in use.

aguilerapy added 4 commits January 9, 2020 09:41

Add output to google drive option

367444e

Merge branch 'master' into 42-in_out_options

a724546

Add upload to google drive options

630130b

Add translations

7a54a7f

aguilerapy requested a review from romifz January 10, 2020 16:56

aguilerapy self-assigned this Jan 10, 2020

aguilerapy changed the title ~~42 in out options~~ Add output to Google Drive option Jan 10, 2020

aguilerapy added 3 commits January 10, 2020 14:01

Merge branch 'master' into 42-in_out_options

5d92613

# Conflicts: # locale/es/LC_MESSAGES/django.po

Fix imports

3f0d734

Fix warning

ebe8ea1

aguilerapy added 3 commits January 24, 2020 09:33

Add translations

930143e

Add module google

5cb1549

Fix large line

7cb11ca

Update requeriments

416c418

romifz suggested changes Jan 31, 2020

View reviewed changes

aguilerapy added 9 commits February 13, 2020 08:48

Update translations

27066d4

Change URL format and methods

9f0693f

Add mime types and JsonResponse

0cce70d

Add drive button and new functions

6e497a1

Update fails

3e279b0

Change saveDrive

91ae56f

Add new translations

e45ae8b

Add tests

302786a

Update tests

a1aba26

aguilerapy requested a review from romifz February 17, 2020 11:37

aguilerapy requested review from romifz and removed request for romifz February 28, 2020 17:11

aguilerapy mentioned this pull request Jun 18, 2020

Remove code repetition #101

Closed

3 tasks

romifz added 6 commits June 26, 2020 18:06

Merge branch 'master' into 42-in_out_options

8ada444

Save to Drive, and code refactors

689a622

Fix quality issues

3f064f1

Fix sorting issues

1ba2720

Read secrets file outside settings.py

4b6c88f

Remove sslserver and old credentials.json

62989c8

romifz unassigned aguilerapy Jul 9, 2020

aguilerapy commented Jul 16, 2020

View reviewed changes

romifz added 4 commits July 21, 2020 10:18

Merge branch 'master' into 42-in_out_options

0879133

Minor fixes after review

58d6559

Fix quality issues

6b930df

Fix test issue

79745f2

aguilerapy mentioned this pull request Jul 24, 2020

Add login and logout options for Google Drive accounts #109

Open

jpmckinney merged commit dba699e into master Aug 8, 2020

CDS 2020-05/2021-02 automation moved this from In progress [6 max] to Done Aug 8, 2020

jpmckinney deleted the 42-in_out_options branch August 8, 2020 01:57

jpmckinney removed this from Done in CDS 2020-05/2021-02 Dec 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add output to Google Drive option #67

Add output to Google Drive option #67

aguilerapy commented Jan 10, 2020

jpmckinney commented Jan 10, 2020

aguilerapy commented Jan 24, 2020 •

edited

Loading

jpmckinney commented Jan 27, 2020

romifz left a comment

romifz commented Jul 9, 2020

romifz commented Jul 9, 2020

aguilerapy Jul 16, 2020

romifz Jul 16, 2020

aguilerapy Jul 16, 2020

romifz Jul 16, 2020

aguilerapy Jul 16, 2020

romifz Jul 16, 2020

aguilerapy Jul 16, 2020

romifz Jul 16, 2020

aguilerapy Jul 16, 2020

romifz Jul 16, 2020

aguilerapy Jul 16, 2020

aguilerapy Jul 16, 2020

aguilerapy Jul 16, 2020

romifz Jul 16, 2020

aguilerapy Jul 16, 2020

romifz Jul 21, 2020

aguilerapy commented Jul 24, 2020

jpmckinney commented Jul 25, 2020

romifz commented Jul 27, 2020

jpmckinney commented Aug 8, 2020

jpmckinney commented Aug 8, 2020

Add output to Google Drive option #67

Add output to Google Drive option #67

Conversation

aguilerapy commented Jan 10, 2020

jpmckinney commented Jan 10, 2020

aguilerapy commented Jan 24, 2020 • edited Loading

jpmckinney commented Jan 27, 2020

romifz left a comment

Choose a reason for hiding this comment

romifz commented Jul 9, 2020

romifz commented Jul 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aguilerapy commented Jul 24, 2020

jpmckinney commented Jul 25, 2020

romifz commented Jul 27, 2020

jpmckinney commented Aug 8, 2020

jpmckinney commented Aug 8, 2020

aguilerapy commented Jan 24, 2020 •

edited

Loading