[REM] google_{drive,spreadsheet}: remove because of API auth change

[kbapt]

Google is disabling their OOB Oauth Flow on 3rd October 2022 1, as
adapting the modules would require too much work (either setup of a proxy
domain to redirect the user to their own instance or have the admins to
setup a dedicated app on their own Google Console), it was decided to
drop those modules.

task-2867215

Description of the issue/feature this PR addresses:

Current behavior before PR:

Desired behavior after PR is merged:

--
I confirm I have signed the CLA and read the PR guidelines at www.odoo.com/submit-pr

[robodoo]

[tivisse]

@robodoo r+

[robodoo]

Linked pull request(s) odoo/upgrade#3633 not ready. Linked PRs are not staged until all of them are ready.

[bouvyd]

Hello,

I can see a twitter storm brewing, so I'll provide a more in-depth rationale for this feature drop.

The 2 modules are impacted by different problems.

google_drive

For Google Drive, the problem lies indeed with the deprecation of an authorization mechanism provided by Google - the so-called "out-of-band" flow where after authorizing access to their Google Drive account to an Odoo-managed "Google Drive app", the user would be given a unique authorization code to copy/paste into their Odoo instance. This flow will cease to work in October, for reasons detailed by Google here. To keep this module working, we would need to set up a proxy-like service where users would be redirected after giving authorization; this proxy would have to be controlled by us (since we manage the app and the redirect url at the end of the oauth flow is fixed and cannot set dynamically) and would then need to redirect the user to the final Odoo instance (on-premise or cloud). Such a setup would require a rather big development effort with security implications that we do not want to manage and that on-premise maintainers may not want either (having Odoo community features and data go through an invisible proxy under Odoo S.A.'s control would not be something I'd look at with a good eye as an Odoo instance maintainer). This is why we will drop this feature.

Note that it is entirely possible to somewhat "resurect" the module in the OCA (or even the OCB), since you could create your own "Google Drive app" with a fixed redirect url that is handled by the odoo instance itself. We do not think that asking end-users to create an Oauth App access in their Google Workspace account (if they have one) and configure it properly is something that should be done for a feature with such low usefulness, and it would still break in current installation without complex configuration changes.

google_spreadsheet

The problem is distinct for the Spreadsheet integration, as it is not related to changes in Google but in Odoo - and in the way security in general is becoming a more important concern. Indeed, we are now in a position where the "AppScript" template we provide for this feature is no longer sufficient and its base concept is faulty.
To make it short: when invoked, the code would fetch a "template spreadsheet" hosted on our own Google Drive, duplicate it for the current user's Google Drive account, insert their login and password into the sheet and use an "AppScript" extension (hosted within the sheet) to communicate with the Odoo instance and fetch the data.
You can already see that this is a terrible, terrible idea, since end users would then share these documents internally or externally without ever realizing that it contained their credentials.
It no longer works in supported version because Odoo has no way to read a password at all, therefore the sheets created would not work out of the box and the user would need to re-set the password through a weird menu. In case 2 factor authentication is enabled for that user (Odoo 15.0 and up), they would need to use an API key instead of their password (which was not made clear in the "UI" of the AppScript).
Obviously, storing such passwords/API keys into the sheet is a terrible idea as well. The "correct" way to go would be to develop a proper Google Spreadsheet plugin, which hosts such credentials securely in its own infrastructure, something we do no wish to do - but which has been done by e.g. IdPact, though I do not know what data they store, nor did I check their security and privacy policies.

Since the feature is broken anyway (the password cannot be inserted into the sheet automatically since v13 at least AFAICT) and given that its basic concept is such a security nightmare and it frankly should never have existed in this form in the first place, we've decided to drop it as well.
To be absolutely clear: we (=Odoo S.A.) were never in possession, or had any way to access, the credentials hosted within the spreadsheets created by this integration. Once it was duplicated from our template, we had no way to read the content whatsoever (I don't think this was ever in question, but since this is a sensitive topic I'd rather be crystal clear). And we do not want to, hence why it's unlikely we'll ever develop a proper Google Spreadsheet/Google Workspace Add-on.

I hope this clarifies the reasoning behind this feature drop.

Good day to you all

Damien B.
Product Owner

[jakubsmolka]

Hello,

I can see a twitter storm brewing, so I'll provide a more in-depth rationale for this feature drop.

The 2 modules are impacted by different problems.

google_drive

For Google Drive, the problem lies indeed with the deprecation of an authorization mechanism provided by Google - the so-called "out-of-band" flow where after authorizing access to their Google Drive account to an Odoo-managed "Google Drive app", the user would be given a unique authorization code to copy/paste into their Odoo instance. This flow will cease to work in October, for reasons detailed by Google here. To keep this module working, we would need to set up a proxy-like service where users would be redirected after giving authorization; this proxy would have to be controlled by us (since we manage the app and the redirect url at the end of the oauth flow is fixed and cannot set dynamically) and would then need to redirect the user to the final Odoo instance (on-premise or cloud). Such a setup would require a rather big development effort with security implications that we do not want to manage and that on-premise maintainers may not want either (having Odoo community features and data go through an invisible proxy under Odoo S.A.'s control would not be something I'd look at with a good eye as an Odoo instance maintainer). This is why we will drop this feature.

Note that it is entirely possible to somewhat "resurect" the module in the OCA (or even the OCB), since you could create your own "Google Drive app" with a fixed redirect url that is handled by the odoo instance itself. We do not think that asking end-users to create an Oauth App access in their Google Workspace account (if they have one) and configure it properly is something that should be done for a feature with such low usefulness, and it would still break in current installation without complex configuration changes.

google_spreadsheet

The problem is distinct for the Spreadsheet integration, as it is not related to changes in Google but in Odoo - and in the way security in general is becoming a more important concern. Indeed, we are now in a position where the "AppScript" template we provide for this feature is no longer sufficient and its base concept is faulty. To make it short: when invoked, the code would fetch a "template spreadsheet" hosted on our own Google Drive, duplicate it for the current user's Google Drive account, insert their login and password into the sheet and use an "AppScript" extension (hosted within the sheet) to communicate with the Odoo instance and fetch the data. You can already see that this is a terrible, terrible idea, since end users would then share these documents internally or externally without ever realizing that it contained their credentials. It no longer works in supported version because Odoo has no way to read a password at all, therefore the sheets created would not work out of the box and the user would need to re-set the password through a weird menu. In case 2 factor authentication is enabled for that user (Odoo 15.0 and up), they would need to use an API key instead of their password (which was not made clear in the "UI" of the AppScript). Obviously, storing such passwords/API keys into the sheet is a terrible idea as well. The "correct" way to go would be to develop a proper Google Spreadsheet plugin, which hosts such credentials securely in its own infrastructure, something we do no wish to do - but which has been done by e.g. IdPact, though I do not know what data they store, nor did I check their security and privacy policies.

Since the feature is broken anyway (the password cannot be inserted into the sheet automatically since v13 at least AFAICT) and given that its basic concept is such a security nightmare and it frankly should never have existed in this form in the first place, we've decided to drop it as well. To be absolutely clear: we (=Odoo S.A.) were never in possession, or had any way to access, the credentials hosted within the spreadsheets created by this integration. Once it was duplicated from our template, we had no way to read the content whatsoever (I don't think this was ever in question, but since this is a sensitive topic I'd rather be crystal clear). And we do not want to, hence why it's unlikely we'll ever develop a proper Google Spreadsheet/Google Workspace Add-on.

I hope this clarifies the reasoning behind this feature drop.

Good day to you all

Damien B. Product Owner

Does this clarify things a little? yes

Does it help? not at all

Considering how many times you are saying that the things implemented in the spreadsheet integartion should have never existed in the first place (which is in no way a fault of users) and now you are saying you will discontinue a functionalitiy with SIGNIFICANT business value without any work around or replacement, and actually WITHOUT INFORMING most customers (even paying ones - no, putting a label in the config field in base odoo code from middle of july is not a sufficient communication for such a big change... you are not avoiding shitstorm. It is still coming, despite all the nice security explanations, the communication is still lacking, and reason for no functional replacement of a valuable integration, when Odoo had more than half a year to come up with something is just laughable.

Any follow up on that?

[bouvyd]

Hello again,

First, I want to be sure that we are clear on what is changing in which version:

• In future versions (16.0 and up): Both Google integrations are gone, as described here
• In stable versions (13.0 to 15.X): The Google Drive integration is gone, the Google Spreadsheet integration remains untouched (as you can check in PR [FIX] google_drive: deprecate module #94747: it only affects google_drive in 13.0 and the subsequent forward-ports)

The "Google Drive" integration is one where you can configure a "template" document that gets duplicated when you click on an action menu item, possibly named according to the record and added as a virtual attachment to the document. This will no longer work at all, for any version, starting on October 3rd for the reasons described above. Nobody seems worried about that one, so I'll leave it at that. I honestly believe the feature is more akin to a shortcut than to an actual bona fide feature (you can still do exactly what the module does by creating your google document manually and adding a link in the record somehow, e.g. an "Internal note" field or the chatter). Is it annoying for those who are used to using it? Sure. It requires a few more steps to get the same results. But nobody is prevented from working or indeed has to modify their workflow extensively.

The "Google Spreadsheet" integration will keep working in stable versions. Only when upgrading will the feature be removed, and existing sheets will keep working (since they use xmlrpc to fetch the data). I do not believe we need to actively warn customers that if/when they upgrade, this will no longer be available for the same reasons that I do not believe actively warning users of what is, in general, changing in future version before they plan on using it is useful: nobody will care or remember what they read 6 months prior when they upgrade their database.

I do agree we need to warn them of the feature change when they upgrade, therefore:

• the feature drop will be mentioned in the release notes (this was planned from the start)
• the feature drop will be announced in the upgrade message that will await administrators in Discuss when they test their upgrades (I've asked this to be done now, since indeed for some customers this can be quite a disruptive change)

I do not believe this needs more communication than that - changing versions can always come with feature changes and, less frequently, feature drops. Customers don't really care why - I don't think sending a lengthy email explaining why we are dropping support for a current feature in future Odoo versions will interest them, especially since they do not have any real way to act upon it besides changing their workflow, which is something they will do when they upgrade anyway (and then it makes sense to warn them indeed, as you suggest).

[jakubsmolka]

Hello again,

First, I want to be sure that we are clear on what is changing in which version:

• In future versions (16.0 and up): Both Google integrations are gone, as described here
• In stable versions (13.0 to 15.X): The Google Drive integration is gone, the Google Spreadsheet integration remains untouched (as you can check in PR [FIX] google_drive: deprecate module #94747: it only affects google_drive in 13.0 and the subsequent forward-ports)

The "Google Drive" integration is one where you can configure a "template" document that gets duplicated when you click on an action menu item, possibly named according to the record and added as a virtual attachment to the document. This will no longer work at all, for any version, starting on October 3rd for the reasons described above. Nobody seems worried about that one, so I'll leave it at that. I honestly believe the feature is more akin to a shortcut than to an actual bona fide feature (you can still do exactly what the module does by creating your google document manually and adding a link in the record somehow, e.g. an "Internal note" field or the chatter). Is it annoying for those who are used to using it? Sure. It requires a few more steps to get the same results. But nobody is prevented from working or indeed has to modify their workflow extensively.

The "Google Spreadsheet" integration will keep working in stable versions. Only when upgrading will the feature be removed, and existing sheets will keep working (since they use xmlrpc to fetch the data). I do not believe we need to actively warn customers that if/when they upgrade, this will no longer be available for the same reasons that I do not believe actively warning users of what is, in general, changing in future version before they plan on using it is useful: nobody will care or remember what they read 6 months prior when they upgrade their database.

I do agree we need to warn them of the feature change when they upgrade, therefore:

• the feature drop will be mentioned in the release notes (this was planned from the start)
• the feature drop will be announced in the upgrade message that will await administrators in Discuss when they test their upgrades (I've asked this to be done now, since indeed for some customers this can be quite a disruptive change)

I do not believe this needs more communication than that - changing versions can always come with feature changes and, less frequently, feature drops. Customers don't really care why - I don't think sending a lengthy email explaining why we are dropping support for a current feature in future Odoo versions will interest them, especially since they do not have any real way to act upon it besides changing their workflow, which is something they will do when they upgrade anyway (and then it makes sense to warn them indeed, as you suggest).

I appreciate this follow up very much, I actually did not receive the point before that for old versions, spreadsheet integration will keep working.

With that information, I also want to thank you for including the message in the upgrade, I think that will be very needed for customers but that should be sufficient for that use case.

As for the drive, integration agreed, that one is not business critical and not nearly as useful as the spreadsheet integration, for that I understand and see your point.

For the future, it would be great if really such a list of changes could be mainteined on your Odoo website, so people are not "forced" to go through git commits (which not everyone even knows can be done) or find our "randomly mentioned" by account managers of Odoo (as was my case).

But I am glad we could have this discussion, appreciate this last summary, that clears up a lot for me :-)

[bouvyd]

I appreciate this follow up very much, I actually did not receive the point before that for old versions, spreadsheet integration will keep working.

That's what I deduced from your message, and indeed it was not made obvious enough I think.

For the future, it would be great if really such a list of changes could be mainteined on your Odoo website, so people are not "forced" to go through git commits (which not everyone even knows can be done) or find our "randomly mentioned" by account managers of Odoo (as was my case).

There is a spectrum between total communication and total silence, and for sure the community would like the former whilst Odoo S.A. will tend towards less communication (because devs don't like communicating, because it has a tendency to slow things down, because we consider that design by committee is not efficient, etc.). The release notes are already improving with increase in versions, but we live in a competitive market: we will always choose to move forward with ideas and then, if need be, communicate more about it.

We tweet more, we make clearer commit/PR messages, etc. and also try to explain things more clearly if we missed the mark (like we did here). I still expect conversations like these to happen (they always will), but I'm glad to see that we can reach a situation where everybody gets something they wanted (moving fast/dropping features with high maintenance vs. utility ratio for us, clear communication with users for you), and I also personnally believe that the way we communicate today is better than what we did 2 or 5 years ago (and I hope it'll keep getting better without getting in the way of us moving forward with the product).

But I am glad we could have this discussion

Same here :)

Have a good day!

[bouvyd]

Hello @jakubsmolka

Well, I owe you an apology - we were a little fast in our analysis (it's an old module, knowledge may have been lost along the way) and indeed the deprecation will also impact google spreadsheet, limiting it's functionality. We're about to publish a blog post explaining the changes (because this time the impact on user flow is real and kind of annoying, even if it will still be possible to create and use spreadsheets; it'll just be less easy). The changes can be seen for series 13 to 15 separately: #99988, #99999, #100011.

In short: we can no longer duplicate the template and insert the formula for the user (nor their credentials, but that hadn't been the case for some time already). We'll guide the user to duplicate the template themselves, then they'll need to use the Odoo > Settings menu in the spreadsheet to authenticate, then they'll need to copy/paste the provided formula.

This is not ideal, and a last resort solution. I explained the reasoning behind not improving this further (we suggest using Documents instead, it'd require hosting an authentication service on our cloud with possibly an open redirect which would be a security nightmare which we don't want to do, etc.), and it held even in light of the UX degradation that will occur.

The blog post will be available at https://www.odoo.com/blog - I'll spread the news (internally so that our support agents, consultants and account managers can easily answer questions from partners/clients, and externally through the main Odoo twitter account if all goes well).

My apologies for realizing this late (i'd prefer we got it right the first time, obviously).

[jakubsmolka]

Hello @jakubsmolka

Well, I owe you an apology - we were a little fast in our analysis (it's an old module, knowledge may have been lost along the way) and indeed the deprecation will also impact google spreadsheet, limiting it's functionality. We're about to publish a blog post explaining the changes (because this time the impact on user flow is real and kind of annoying, even if it will still be possible to create and use spreadsheets; it'll just be less easy). The changes can be seen for series 13 to 15 separately: #99988, #99999, #100011.

In short: we can no longer duplicate the template and insert the formula for the user (nor their credentials, but that hadn't been the case for some time already). We'll guide the user to duplicate the template themselves, then they'll need to use the Odoo > Settings menu in the spreadsheet to authenticate, then they'll need to copy/paste the provided formula.

This is not ideal, and a last resort solution. I explained the reasoning behind not improving this further (we suggest using Documents instead, it'd require hosting an authentication service on our cloud with possibly an open redirect which would be a security nightmare which we don't want to do, etc.), and it held even in light of the UX degradation that will occur.

The blog post will be available at https://www.odoo.com/blog - I'll spread the news (internally so that our support agents, consultants and account managers can easily answer questions from partners/clients, and externally through the main Odoo twitter account if all goes well).

My apologies for realizing this late (i'd prefer we got it right the first time, obviously).

Hi Damien,

Thank you for the follow up. You call it "last resort solution", for me it seems as a perfectly fine workaround with the new limitations and without doing your own authentication service. The guide window inside odoo seems perfectly fine and the important part is that users of Odoo 13 to 15 will still be able to use the spreadsheet service - that's what matters from my point of view.

Regarding the documents - I have been checking the native Odoo spreadsheet functionality and dashboards in V16 (master branch) and I do have to admit those are some impressive improvements, definitely close enough to handle most of standard reporting, but for older versions, I know the odoo spreadsheet functionality is still quite basic or cumbersome to work with.

Just FYI - there is one use case that it cannot handle. For many companies and customers, connecting Odoo to a BI tool is really the only way to get a proper, advanced reporting, as Odoo itself in old versions cannot really handle computing reporting metrics from different models easily. The V16 dashboard is definitely closer to it, but still I see there some limitations in terms of data visualization (it simply cannot yet compare to proper BI tools, which is understandable). And the limit then is Odoo.sh. Since on Odoo.sh you cannot get access to postgres database, you usually had no better way of pulling data to the BI (in almost real time) other than going through google spreadsheets → and having e.g google studio be pulling the data through google spreadsheet. Since this way will be rermoved in v16, you might consider for the future finding a way how customers hosted on odoo.sh can plug their BI tolls into Odoo without having to buy expensive modules (like e.g this one https://apps.odoo.com/apps/modules/15.0/tableau_direct_connecter/)

And one last note - will you be informing at least odoo partners (preferably all customers but I know that can be impossible) when the blog post is released, so you ensure they were proactively informed before the deadline of 3rd October is passed?
You mention external communication through twitter account, but honestly - most of the world does not really use twitter. And I think at least to include it in newsletter/separate mail to Odoo partners should be feasible and it should not be a problem for the account managers to write 1 short email and send it to partners, especially since most information will come from you already.
Theoretically for v13-15 not much will changed as the option to create spreadsheets will still be there, but I think everyone will appreciate such a proactive communication from Odoo's side.

PS: If you would create a mailing list/newsletter for both customers and partners "Major changes in new versions", I would definitely subscribe ;-) :D

[bouvyd]

I'll send this to the #announcement mailing list, which is that kind of list (though we seldom use it for that, it is mostly used to convey CVEs announcements, etc.) and see if I can post in a few more places (no guarantee).

Blog is published here.

[ctringham]

Just to add to this discussion (and apologies if I am stating the obvious), communicating this more widely can help to expose both the limitations and the workarounds to scrutiny and help to arrive at an acceptable solution.

[JFKTiz]

Hi Damien,

I unterstand the security decision as well, but I would double down on what @jakubsmolka writes, even more so for Odoo Online users :
The Spreadsheets integration is the only way for Odoo Online users to get the data out of Odoo into BI Tools (say Google Data Studio, or even Power BI since PowerQuery now supports natively Google Sheets).
When this integration is gone, there'll be no way to export the data in real time in an external tool (without upgrading to Odoo.sh or a different version of Odoo).
Or am I missing something ?

For me it's a red flag, as Odoo can't compete with the leading BI softwares...