Existing items being shown as "new" repeatedly

[yellowbarchetta]

As a consequence of using the email alerts for new items, I've been seeing the same items added continually each time the script runs. The GCal looks right at the end (i.e. it's not duplicating the entries), but it does seem to create new ones with new IDs each time the script runs - and I think as a consequence it's exceeding the number of emails allowed too!

I've added a couple of logging lines to the if loop around lines 177-190ish (lines may have moved) as follows:-

      if (addEventsToCalendar){
        Logger.log("Adding new event " + newEvent.extendedProperties.private["id"] + newEvent.summary + newEvent.start.date);
        newEvent = callWithBackoff(function(){
          return Calendar.Events.insert(newEvent, targetCalendarId);
        }, 2);
        if (newEvent != null && emailWhenAdded){
          try{
            GmailApp.sendEmail(email, "New Event \"" + newEvent.summary + "\" added", "New event added to calendar \"" + targetCalendarName + 
                                                                                              "\" at " + newEvent.start.toString());
            Logger.log("Email try success new, sent \"" + newEvent.summary);
          }
          catch(error){Logger.log("New email failure \"" + error)} 
        }
      }

The typical output is as follows:

[19-12-18 00:54:39:114 PST] Syncing 1 calendars to Work Calendar (LIVE)
[19-12-18 00:54:39:341 PST] Working on calendar: xxxxxxxxxxxxx@group.calendar.google.com
[19-12-18 00:54:40:120 PST] Fetched 221 existing events from Work Calendar (LIVE)
[19-12-18 00:54:41:189 PST] Parsed 246 events from ical sources
[19-12-18 00:54:41:190 PST] Processing 246 events
[19-12-18 00:54:41:459 PST] Timezone GMT Standard Time unsupported!
[19-12-18 00:54:41:459 PST] Using Timezone Europe/London!
[19-12-18 00:54:41:462 PST] Adding new event 74cf55f8-d665-44dd-9ee9-e40a3b1e1746Balliol Patrolundefined
[19-12-18 00:54:41:767 PST] New email failure "Exception: Service invoked too many times for one day: email.
[19-12-18 00:54:41:809 PST] Skipping unchanged Event 040000008200E00074C5B7101A82E008000000007002F1BF618ED101000000000000000010000000A93C934B10FD5B42BA6EFF2BEB646B42
[19-12-18 00:54:41:856 PST] Timezone GMT Standard Time unsupported!
[19-12-18 00:54:41:856 PST] Using Timezone Europe/London!
[19-12-18 00:54:41:859 PST] Adding new event 62deb299-e254-4823-bccc-57d7651c2c9bBalliol Patrolundefined
[19-12-18 00:54:42:323 PST] New email failure "Exception: Service invoked too many times for one day: email. 

The next hourly run reports the following, without any changes being made to the "Balliol Patrol" entries:-

[19-12-18 01:54:21:770 PST] Syncing 1 calendars to Work Calendar (LIVE)
[19-12-18 01:54:22:011 PST] Working on calendar: 6uogmsbirss078fiv2ebghi08o@group.calendar.google.com
[19-12-18 01:54:22:875 PST] Fetched 221 existing events from Work Calendar (LIVE)
[19-12-18 01:54:24:589 PST] Parsed 247 events from ical sources
[19-12-18 01:54:24:590 PST] Processing 247 events
[19-12-18 01:54:24:850 PST] Timezone GMT Standard Time unsupported!
[19-12-18 01:54:24:850 PST] Using Timezone Europe/London!
[19-12-18 01:54:24:854 PST] Adding new event 16244687-a759-412a-9549-0d4e4f9f355cBalliol Patrolundefined
[19-12-18 01:54:25:330 PST] New email failure "Exception: Service invoked too many times for one day: email.
[19-12-18 01:54:25:402 PST] Skipping unchanged Event 040000008200E00074C5B7101A82E008000000007002F1BF618ED101000000000000000010000000A93C934B10FD5B42BA6EFF2BEB646B42
[19-12-18 01:54:25:466 PST] Timezone GMT Standard Time unsupported!
[19-12-18 01:54:25:467 PST] Using Timezone Europe/London!
[19-12-18 01:54:25:470 PST] Adding new event e73273b1-acea-438e-94c9-af6d85413a24Balliol Patrolundefined
[19-12-18 01:54:25:768 PST] New email failure "Exception: Service invoked too many times for one day: email.

I've not been able to capture the specific dates of these entries (hence the "undefined" at the end), so I can't be certain which events in the ical file are causing this. If you can help me tie the events to a specific entry I can provide more info. One thing is true, that the event ids reported above are NOT found in the ical file, so I don't know if something is changing regularly at the ical generation end by Outlook somehow.

[jonas0b1011001]

it does seem to create new ones with new IDs each time the script runs

The IDs are not created by the script but from where you are getting the sourcefile from.

I think as a consequence it's exceeding the number of emails allowed too!

This definitely seems to be the case. I will work on your suggestion #87 as soon as i get the time, this should reduce the amount of emails sent quite a bit (good suggestion by the way!).

I've not been able to capture the specific dates of these entries (hence the "undefined" at the end)

newEvent.start.date is only set for all day events, 'normal' events use newEvent.start.dateTime.

the event ids reported above are NOT found in the ical file, so I don't know if something is changing regularly at the ical generation end by Outlook somehow.

As the log shows, the script is skipping the second event as it is recognized to be unchanged. So it is definitely working as expected on that event. I don't really know why Outlook would randomly change the id of event 1 and 3 but not on event 2. If possible please share the definitions of 1 and 3, maybe they help.

[yellowbarchetta]

Thanks for the pointer to the dateTime. I'll be able to follow through which appointments these are specifically and look at the ical file. I did assume the ids were iCal generated ones, so it does look like Outlook for some reason is recreating these on a regular basis (at least hourly, if not more). I can look at that separately.

[yellowbarchetta]

OK, there's definitely something odd happening in my ical output - so this isn't a bug for here. I've just run the process twice within a few minutes, making no changes to the source calendar at all. It's easy to see that entries created in the first pass are then given different IDs in the second pass, and that the first pass ID entries are then deleted in the second pass. So your code is handling this perfectly, it's just my ical source which is screwy somehow. It does seem to be limited to a number of specific entries, so I'll see if I can make any changes to them with any success. I seem to remember that these entries were made by me copying and pasting within Outlook calendar, but I can't for the life of me work out why that would make a difference!!

Here's the outputs for completeness, I'll close the bug report.

Pass 1

[19-12-18 10:22:59:366 GMT] Syncing 1 calendars to Work Calendar (LIVE)
[19-12-18 10:22:59:564 GMT] Working on calendar: xxxxxxxx@group.calendar.google.com
[19-12-18 10:23:00:420 GMT] Fetched 222 existing events from Work Calendar (LIVE)
[19-12-18 10:23:01:768 GMT] Parsed 247 events from ical sources
[19-12-18 10:23:01:769 GMT] Processing 247 events
[19-12-18 10:23:01:998 GMT] Timezone GMT Standard Time unsupported!
[19-12-18 10:23:01:999 GMT] Using Timezone Europe/London!
[19-12-18 10:23:02:002 GMT] Adding new event ad9edb42-cde5-41c8-99da-57b64105d48fBalliol Patrol2019-09-18T08:00:00
[19-12-18 10:23:02:322 GMT] New email failure "Exception: Service invoked too many times for one day: email.
[19-12-18 10:23:02:377 GMT] Skipping unchanged Event 040000008200E00074C5B7101A82E008000000007002F1BF618ED101000000000000000010000000A93C934B10FD5B42BA6EFF2BEB646B42
[19-12-18 10:23:02:431 GMT] Timezone GMT Standard Time unsupported!
[19-12-18 10:23:02:432 GMT] Using Timezone Europe/London!
[19-12-18 10:23:02:434 GMT] Adding new event 6eea0a9a-cc73-41d7-9f9d-dcc114653c58Balliol Patrol2019-09-18T12:00:00

Pass 2

[19-12-18 10:29:47:906 GMT] Syncing 1 calendars to Work Calendar (LIVE)
[19-12-18 10:29:48:184 GMT] Working on calendar: xxxxxxxxx@group.calendar.google.com
[19-12-18 10:29:49:020 GMT] Fetched 222 existing events from Work Calendar (LIVE)
[19-12-18 10:29:50:366 GMT] Parsed 247 events from ical sources
[19-12-18 10:29:50:367 GMT] Processing 247 events
[19-12-18 10:29:50:569 GMT] Timezone GMT Standard Time unsupported!
[19-12-18 10:29:50:570 GMT] Using Timezone Europe/London!
[19-12-18 10:29:50:573 GMT] Adding new event c7a3557e-d353-4cbc-a192-f369ef1682c6Balliol Patrol2019-09-18T08:00:00
[19-12-18 10:29:50:871 GMT] New email failure "Exception: Service invoked too many times for one day: email.
[19-12-18 10:29:50:951 GMT] Skipping unchanged Event 040000008200E00074C5B7101A82E008000000007002F1BF618ED101000000000000000010000000A93C934B10FD5B42BA6EFF2BEB646B42
[19-12-18 10:29:51:035 GMT] Timezone GMT Standard Time unsupported!
[19-12-18 10:29:51:036 GMT] Using Timezone Europe/London!
[19-12-18 10:29:51:039 GMT] Adding new event 40baa19e-9ef1-46d4-b63f-a8e5fbbdcfaaBalliol Patrol2019-09-18T12:00:00
[19-12-18 10:29:51:586 GMT] New email failure "Exception: Service invoked too many times for one day: email.

Deletions shown in pass 2

[19-12-18 10:31:23:486 GMT] Deleting old event ad9edb42-cde5-41c8-99da-57b64105d48f
[19-12-18 10:31:23:781 GMT] Deleting old event 6eea0a9a-cc73-41d7-9f9d-dcc114653c58

[yellowbarchetta]

Just to wrap this up, I've looked through the iCal file and retrieved it twice within a matter of minutes. I can confirm that for some (but not all!) appointments, the Outlook 365 operation is creating UIDs which change each time the file is fetched. But with others, the UID is not changeing.

Events with IDs in the format xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx are the ones which change, events with the IDs NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN do not.

I need to see if I can amend / delete the problem instances - this is definitely an outlook issue. This bug for another app referred to it https://theeventscalendar.com/support/forums/topic/duplicate-ical-import-still-a-problem/ with Outlook365 not following the standard.

It just occurs to me now that when I dropped in these appointments I did it via an excel upload of a file, so the nature of these appointments are different. So I'll probably need to change the source data and see if they disappear!

[derekantrican]

See if you can try reporting this to Outlook/Microsoft somehow otherwise I start a new job at Microsoft in a couple weeks and I'll see what I can do ;)

[yellowbarchetta]

I'll see what I can do!! Good luck with MS! The oddity is that from the front end client (i.e. Outlook) I can't see any difference between the two calendar entries - ones that were "uploaded" from a spreadsheet and ones that were manually entered by either myself or someone in my organisation. It seems it's an issue that others have encountered in similar fashions:- https://social.msdn.microsoft.com/Forums/en-US/e1714a15-1ef7-4868-a701-f53c5ceae1ad/ews-api-differences-in-icaluid-returned-when-appointments-are-created-by-office-365-account-vs?forum=exchangesvrdevelopment I wonder whether this article though has food for thought in terms of which unique ID you're using to match appointments:- https://stackoverflow.com/questions/36737596/why-differs-the-icaluid-in-length

…

On Wed, 18 Dec 2019 at 14:22, Derek Antrican ***@***.***> wrote: See if you can try reporting this to Outlook/Microsoft somehow otherwise I start a new job at Microsoft in a couple weeks and I'll see what I can do ;) — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#88?email_source=notifications&email_token=AOAEASMU5J54FKBBIEXHIRTQZIW3DA5CNFSM4J4H6L22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHGIOWA#issuecomment-567052120>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOAEASMXG26VSQYHEV65ZZTQZIW3DANCNFSM4J4H6L2Q> .