Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems with the syncronization #28

Open
IMT1111 opened this issue Aug 24, 2023 · 41 comments
Open

Problems with the syncronization #28

IMT1111 opened this issue Aug 24, 2023 · 41 comments

Comments

@IMT1111
Copy link

IMT1111 commented Aug 24, 2023

When I set to import to the calendar automatically, it deletes all the formulas and leaves just the values. If I protect the the columns with the formulas so the user that uses the Sheets2GCal extension can't edit the columns with the formulas, Sheets2GCal doesn't synchronize with calendar automatically, just manually. The manual import works even if those columns are protected but apparently no the automatic one. Thanks so much

@GCalToolkit
Copy link
Collaborator

I'll look into this today and let you know what I find ASAP. Thanks for letting me know!

@IMT1111
Copy link
Author

IMT1111 commented Aug 24, 2023 via email

@IMT1111
Copy link
Author

IMT1111 commented Aug 25, 2023 via email

@GCalToolkit
Copy link
Collaborator

Hi Israel, that's normal - to reduce the significant overhead of checking existing values cell by cell before updating Sheets2GCal overwrites the row. Google limits the time that App script is allowed to run between user interactions, and checking cell values is time-consuming. This means that updating cells which have changed would mean a massive reduction in the number of rows/events that the add-on could manage.

However, the formulas should not be affected as the automatic update uses exactly the same code as the manual update. Unfortunately, to test properly under the exact same conditions as yourself requires me to test with triggers... which limits the speed at which I can test.

Thanks you for your patience with this issue, and if you'd like to continue the conversation by email please contact me via my website or "info" at "GCalToolkit" dot com.

@GCalToolkit
Copy link
Collaborator

I'm sorry for the delay, but could you please clarify a few things to help me reproduce this? Are the formulas being deleted from new rows, from existing rows which are being updated, or from every row in the Sheet?

@IMT1111
Copy link
Author

IMT1111 commented Aug 26, 2023 via email

@GCalToolkit
Copy link
Collaborator

I'm currently trying to figure out why the manual update is working normally but the triggered version isn't even though they call the exact same function. It seems that for some reason Google Sheets is behaving differently when the user isn't present, but I'm still testing.

Also, are your formulas the same for the entire column? If so, have you already tried using ARRAYFORMULA instead of using formulas in each cell as explained here: https://www.bpwebs.com/apply-formula-to-entire-column-in-google-sheets ?

@IMT1111
Copy link
Author

IMT1111 commented Aug 26, 2023 via email

@IMT1111
Copy link
Author

IMT1111 commented Aug 27, 2023 via email

@IMT1111
Copy link
Author

IMT1111 commented Aug 27, 2023 via email

@GCalToolkit
Copy link
Collaborator

Thank you for your patience, I'm nearly ready to release an update that keeps all formulas intact during manual or automatic updates. I do have a few questions though about how you are using the formulas and the behaviour you would expect. The easiest way would be to either share an example Sheet with me containing the formulas you'll be using, or some screenshots.

For example, would you expect all cells containing a formula to ignore any changes to the calendar - even though subsequent Sheet > Calendar operations would then overwrite edits to the calendar? Are new rows created from new calendar events expected to prioritise the calendar data or (if present) a formula set for an entire column?

With over 70k users I have to be very careful when changing behaviour like this to ensure that it doesn't adversely affect other users who might use formulas in very different ways... so even though I have a version that works for me I still need to do extensive testing.

@IMT1111
Copy link
Author

IMT1111 commented Aug 30, 2023 via email

@GCalToolkit
Copy link
Collaborator

The changes I've made so far are all for Calendar => Sheet and basically ignore any edit to cells already containing a formula, which is something people have asked for in the past too.

Updating the Calendar from the Sheet doesn't affect formulas as it only updates the relevant cells, as you said.

However, rereading your posts I realise that you are expecting the Automatic Import to update the Calendar from the Sheet whereas it's the opposite: it imports updates from the Calendar to the Sheet (which was affecting formulas)

So I think what you really need is "Automatic Update" in addition to "Automatic Import". This is a work in progress because I need a way to handle errors with calls to the Calendar API when the user isn't present. It's been a problem for a while now as unlike Sheets, Google calendar didn't let you easily roll-back to an earlier version if there is a problem.

I'll first update so formulas are preserved on Syncing Calendar=>Sheet, then take another look at Automatic Update and let you know how long I think it might take.

@IMT1111
Copy link
Author

IMT1111 commented Aug 30, 2023 via email

@GCalToolkit
Copy link
Collaborator

You wouldn't believe the mess I've seen people get into when editing calendars, and not everything is recoverable using the Sheet - in particular changes made to repeat event series, for example. Recreating a calendar can also be a nightmare for users with many guests to thousands of events/Meets - sending thousands of new invitations to new versions of events is not a pleasant experience! Nevertheless, I'm hopeful I can find a solution, and I'll do some more work on it this week and let you know how I get on.

Also, I forgot to mention that it's currently impossible to automatically detect changes pulled into a Sheet as a result of a formula. Unfortunately the only way around this would be to update every event whether they have changed or not, or to store a shadow copy of every sheet to compare against... which might be fine for anyone with only a few hundred rows, but is very inefficient and will inevitably cause issues for large calendars.

@IMT1111
Copy link
Author

IMT1111 commented Aug 30, 2023 via email

@GCalToolkit
Copy link
Collaborator

GCalToolkit commented Sep 4, 2023

Hi Israel, I've just released a big update which adds Automatic Import/Export without affecting formulas. I managed to work around not being able to use the "Update" column by first Importing changes to Google Calendar, then checking for differences between the updated Sheet and the Calendar before updating the Calendar. I am a little concerned that this might take too long to run as a trigger (without the user present)... but I'll never know if I don't try, so the new version should be live very soon if it isn't already.

You'll know you are using the new version as the menus have also changed - "Update Calendar from Sheet" is now "Export Sheet to Calendar". Please test it out and let me know how it performs!

@IMT1111
Copy link
Author

IMT1111 commented Sep 5, 2023 via email

@GCalToolkit
Copy link
Collaborator

GCalToolkit commented Sep 5, 2023

You wrote "would it be possible when the automatic import/export to run first the Check Sheet vs Calendar". This is exactly what happens: the update boxes are not actually checked, but the events to be processed are those flagged by that option.

Deleted rows are the biggest problem since there is no way to detect these, and without the event ID the event can't be deleted from Google calendar anyway. The only solution I can see for this right now is a "hard export" which wipes the entire calendar and performs a full export... but that is extremely inefficient and would be a nightmare for events with guests/meets as guests would need to be reinvited for every event and the Meet IDs would change.

It's best for events to be deleted by manually checking the "Delete" column instead of deleting the row, although depending on the conditions for deleting your events I suppose it might be possible to use a formula for Delete column which sets it to TRUE if certain conditions are met - but that would open up another can of worms.

Alternatively, if your rows are somehow being automatically deleted I could add a mechanism to delete events with no title/start as long as they still have an event ID - that way you'd have to modify your row deletion to remove only titles and start date for events to be deleted.

@IMT1111
Copy link
Author

IMT1111 commented Sep 6, 2023 via email

@GCalToolkit
Copy link
Collaborator

GCalToolkit commented Sep 6, 2023

Unfortunately it's not possible to split the automatic import/export because add-ons can only have a maximum of one timed trigger, per user, per document.

I've already double checked that the new version doesn't affects formulas, so I think it's possible there's a problem with the ARRAYFORMULA either trying to overwrite non-formula cell data, or returning blank cells. It's very hard to say without being able to see your Sheet and the formula, but it might be worth checking the section entitled "Blank cells challenge in Google Sheets ARRAYFORMULA output" here https://blog.coupler.io/arrayformula-google-sheets/.

Alternatively, you could use the standard formula method for each cell to avoid this issue.

For event deletes: I'll add a check for invalid title and start date with a valid event ID, and if that condition is met delete the event and the event ID. It's probably better to leave the rest of the row data intact for the user to clean up at their own convenience in case they used other custom columns or need to know which events were deleted.

@GCalToolkit
Copy link
Collaborator

Update submitted: when using Automatic Import/Export events will be deleted if a row has an event ID but no Title and Start Date, the event ID and HTML link will be removed, and the row's "Delete" cell will be coloured red. The menu option "Check Sheet vs Calendar" will also detect these rows and mark them for deletion.

@IMT1111
Copy link
Author

IMT1111 commented Sep 7, 2023 via email

@IMT1111
Copy link
Author

IMT1111 commented Sep 7, 2023 via email

@GCalToolkit
Copy link
Collaborator

"if we split the automatic import/export, as long as we use only one of the options, either import or export per document we would still be having
only one trigger."

I could add an option for each Sheet's "Automatic Import/Export" to choose between Import, Export, or both (and even in which order). I'll look into this over the next few days and let you know how I get on.

Thanks again for your feedback on this, it's thanks to users like yourself that my software keeps improving!

@IMT1111
Copy link
Author

IMT1111 commented Sep 7, 2023 via email

@GCalToolkit
Copy link
Collaborator

When exporting manually the "Delete" checkbox must be checked (either manually or by using "Check Sheet vs Calendar") to delete a row. If you want to preserve row data and formulas you would also need to enable the option "Keep row data for deleted events" before you export.

When using Automatic Import/Export that event would have been detected by running "Check Sheet vs Calendar" and ONLY the event ID and HTML link would be deleted, just as if you had checked the option to "Keep Row Data...". What I can do is set the default to be "Keep Rows" even when deleting manually.

I'm making progress on an option to either "Import", "Export", "Import ->Export" or "Export -> "Import" so that should be ready soon, although it will take some testing before I update.

@GCalToolkit
Copy link
Collaborator

Testing went better than expected... the latest version let's you choose a sync option for each Sheet, and defaults to Import>Export for existing settings. It also defaults to keep all formulas intact unless you sync manually and specify not to keep the deleted rows. There are some other improvements too so let me know how you get on. The biggest problem now is likely to be Google's restrictions - there's no way around those so I do expect problems with large Sheets/Calendars with too many sheets syncing.

@IMT1111
Copy link
Author

IMT1111 commented Sep 13, 2023 via email

@IMT1111
Copy link
Author

IMT1111 commented Sep 14, 2023 via email

@GCalToolkit
Copy link
Collaborator

GCalToolkit commented Sep 14, 2023

When you have set up automatic syncing for a Sheet and refresh the page, close the sidebar etc. and open it again, does it still say the sheet is set to sync? Or does the list empty out only after the time for which syncing was set?

If you visit this page it should show all the triggers that you have set up (or which scripts like Sheets2GCal have set up for you):
https://script.google.com/home/triggers

Is a "Timed Trigger" listed there after you set it up in Sheets2GCal? And is there anything listed under "Executions" at https://script.google.com/home/executions ?

@IMT1111
Copy link
Author

IMT1111 commented Sep 14, 2023 via email

@GCalToolkit
Copy link
Collaborator

Thanks for that information - when you wrote " I can see some Timed Trigger not exactly in the moment I set it up" did this mean there is already a trigger listed at https://script.google.com/home/triggers?
If so, can you click the menu for that trigger and delete it, then try to set it up again in Sheets2GCal?

Also, if you create a new spreadsheet document with a new sheet connected to Google Calendar, does it suffer from the same problem?

@IMT1111
Copy link
Author

IMT1111 commented Sep 22, 2023 via email

@GCalToolkit
Copy link
Collaborator

GCalToolkit commented Sep 23, 2023

Hi Israel, I'm glad to hear that's all working - I'll fix the calendar name issue in the next update so it doesn't default to the original calendar name.

@IMT1111
Copy link
Author

IMT1111 commented Sep 23, 2023 via email

@GCalToolkit
Copy link
Collaborator

This update should be active in the next few hours - calendars will now always use their latest display names, and any changes to the calendar names in Google Calendar will be reflected in Sheets2GCal as soon as you reopen the sidebar. Read-only calendars are now marked with **

@IMT1111
Copy link
Author

IMT1111 commented Sep 24, 2023 via email

@GCalToolkit
Copy link
Collaborator

My pleasure - one question if I may... I've been contacted by a user who is having trouble using formulas to pull data from other sheets. Do you do this and do you have any problems with that?

@IMT1111
Copy link
Author

IMT1111 commented Sep 25, 2023 via email

@IMT1111
Copy link
Author

IMT1111 commented Sep 25, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants