As a member, my calendar will automatically update if the event changes

[allella]

Quincy was targeting a feature to integrate with Google Calendar to update events when they are changed or cancelled.

In the Nov 7th meeting, we talked about not using Google Auth, or depending on Google services.

In that case, iCal and similar techniques are the way to add a calendar to various calender tools.

Fran asked about only using email authentication and event notifications, plus using iCal for the add-to-calendar feature. He mentioned Google Calendar users can now link to an iCal, through webcal:// or https://, using a feature which will use the link to keep the event in sync based on regular synchronization interval. This would be instead of a one-time importing of an iCal file or an "Add to Google" button (example) that uses a query string to save the event to Google but where never updates again. For this to work the iCal/webcal link would need to be persistent and updated in real-time with each request. See "Subscribe to an Online Calendar" section for an example.

[allella]

@Zeko369 @Ryuno-Ki does this look like a good characterization of what was suggested as a new MVP issue?

[Ryuno-Ki]

I'll try to put it in my own words.
Perhaps it helps getting a more complete picture.

What is the problem we want to solve?

If I RSVP to an event at my chapter, I want to be able to have that in my calendar.
There are two ways here:

1. Downloading an ICS (or iCal, as Apple called it) file and import that into my calendar (could be Google, Microsoft, Nextcloud, whatever).
2. Importing an webcal URL to my Calendar (webcal is a protocol that wraps ics).

The second option would allow for updates if the organiser change things (like moving date or venue or even cancelling the event).

The „add to Google button” prefills Google calendar with event details via Query Parameters. The user only needs to confirm the addition. The alternative discussed was more involved (adding for permission to access the calendar on the user's behalf. Making a decision which calendar to add an event to etc).

When downloading the file, I'd miss those updates. So I could take a trip just to learn, that the event isn't going to happen. We want to avoid that if possible.

[Ryuno-Ki]

I talked with @Zeko369 on Discord about it more in-tech.

Since Google (or another provider) would crawl the ICS file periodically, it needs permission to do so.
One option would be to achieve that via auth token, that gets passed along side a ICS URL by chapter.
In fact, there could be several of those URLs:

• /calendars/user?auth
• /calendars/venue?auth
• /calendars/chapter?auth

(the exact structure is open for discussion).

Since anybody with that URL will have access to that subscription it is important to stress, that this URL must only be pasted into the calendar (e.g. Google calendar, see link in original post) and NOT SHARED.
Additionally, it is important to blacklist put a token on a denylist if it got leaked by some means.

Lastly, I could imagine keeping an auth trail akin to https://github.com/settings/security-log to see, who accessed the link (with timestamp and User Agent?). This way, suspicious behaviour could be detected. It shouldn't be limited to auth tokens generated in this context.

[allella]

So, is the idea for MVP to use a feed-based link that has all of a member's event subscriptions, or have the member copy and paste a link to their calendar for each specific event?

The authorization tokens above sound more applicable to a specific person's calendar feed and not as much for a single event link. True?

Fran, was the link you used for your university for a specific event, or did you have a link with a feed of all events

[allella]

Also, if the idea is to have a feed link with calendar data for multiple events, we'd want to decide if we're talking about a link with all events which are happening for a specific chapter, or only events the member registered to attend.

[Zeko369]

One event => not updatable, simple to add (one click or download .ics)
Whole calendar => updatable, generate url (generate token per say), copy paste link into gcal/icloud/outlook/whateverCal.inc

We can have whatever customisation you want on the /user feed, but you can also subscribe to specific venue/chapter (more for information stuff / ie that Hackgreenville thingy)

[allella]

Thanks. I was thinking your initial example was a replacement for the "Add to Google Calendar" button, but a link for a single event where that single event was kept up to date. Never heard of that and I didn't realize you were talking about feeds.

Technically, you could have a feed URL specific to an event, so that would be possible, but probably not an ideal implementation for the feed provider or consumer.

[Zeko369]

Pretty sure google calendar only supports adding calendars dynamically, so I don't really know how we would implement updatable events.

I could be wrong though.

[allella]

I'm not saying it's a good idea, but I was that you could create a webcal / feed with just one event. People would need to add each feed link and the calendar would have a lot more feeds to keep in sync, but seems like it's doable since it would just be a feed with n=1.

[Zeko369]

Image:

Gcal image

This is my gcal sidebar, I don't really think adding a calendar for each event is a good idea haha

[Zeko369]

Also if you only have one event it would still show in your "user" calendar

[allella]

Yeah, I agree. Though, in a different tool / interface / use case it would be feasible and may actually be a neat protocol. I've never used it but I wonder if that would be similar to having a https://github.com/pubsubhubbub/ on a single thing you want to track.

[Ryuno-Ki]

For reference: PubSubHubBub is now known as WebSub in some parts of the web.

[Zeko369]

I have a prototype of this working with ICAL emails and same IDs for events, will PR soon

[allella]

More notes from Fran about the direction of this feature were shared in the June 26, 2021 meeting.

• The iCal RSVP feature is not fully implemented, but it is based on the iCal specification, which is commonly supported.
• iCal will allow users of Google, Outlook, and other calendar systems to add, and even update, events onto their calendar. Google and Outlook will automatically add and update the events to their users' calendar based on an RSVP email that contain a proper iCal file. Fran explained that this iCal approach avoids Chapter administrators needing setup Google OAuth, plus it expands support beyond just Google users.
• Users of offline calendars and other non-Google, non-Outlook calendars can still download / click the iCal file within an email to import an event to their calendar of choice. However, any changes to the event would need to be re-imported.

[ojeytonwilliams]

Hey @Zeko369 do you have a branch with the prototype that we can take a look at? I know you've got a lot on, so I'd be happy to finish of whatever you've put together so far.

[allella]

@QuincyLarson @ojeytonwilliams

I've setup a couple fake events to test integrations with Google, Outlook, iPhone Calendar, and Yahoo.

I'm going to wait until tomorrow to update the event and see if anything auto-updates.

If you want to register and add to your own calendars as extra tests, then the links are below.

https://www.meetup.com/Hack-Greenville/events/282100731/
https://www.eventbrite.com/e/mystery-eventbrite-test-tickets-211201267677

Eventbrite has an Add to Calendar link on the event page and it's not necessary to register to use that feature. The "Add to Calendar" link in the Eventbrite confirmation email seems to have the same effect, even without registering.

[ojeytonwilliams]

Cheers, I've signed up for both. Interestingly meetup sent an ics file, but eventbright just sent their own ticket.

[allella]

Thanks. I don't think either of them are going to auto-update, even with the iCal file and Add to Calendar links, but we'll see.

[allella]

@ojeytonwilliams @QuincyLarson

TLDR; there is no automatic updating of the calendars with Meetup and Eventbrite. They are actually worse at sending emails and iCals on updates than I was expecting.

@ojeytonwilliams if you want to confirm your experience, then check your calendar. The modified events changed to a different date / time and have "Modified" in the event title.

Detailed Findings

• I ran tests with Meetup and Eventbrite on Google, Outlook, and Yahoo calendars. I started testing iPad but my wife deleted the event thinking I was done with testing.
• Google Calendar does not show pending, low opacity event place holders on the calendar for Meetup and Eventbrite just from the confirmation email. It will show pending events on the calendar for more traditional invites originating from Google Calendar, Yahoo Calendar, and Outlook Calendar.
• Both Meetup and Eventbrite emails render custom email snippets above the emails in Gmail based on the content of the initial emails.
  • Meetup's email has a custom snippet with an Add to Calendar link in Google / Gmail inbox and an .ics file as an attachment.
  • Meetup's "event has changed" email does not render a custom snippet in Google / Gmail.
  • Outlook and Yahoo inboxes don't seem to do an custom snippets for emails, but Outlook will show an "Add to Calendar" when an iCal file is present
  • Eventbrite's email has a custom snippet with a link to "Modify Reservation", but this just links back to Eventbrite, and Eventbrite does not include iCal files in the initial email or update emails.
• Eventbrite doesn't show an add to calendar on the confirmation page after RSVP. It only shows it on the event page and the confirmation email.
• Meetup sends an email titled "The date and time were changed for {event name}, but there is no magic update to existing calendar entries and the email doesn't even have an updated iCal file. It links of to the event page and you can manually "Add to Calendar", which creates a new calendar entry and doesn't update or delete any existing entries for the same event.
• Evenbrite doesn't automatically send an email about changes to the event. Also, any existing events are not magically updated in Google, Outlook, or Yahoo.
• Meetup shows add to calendar upon RSVP confirmation, on the event page, and sends an iCal file on the first confirmation email, but not on emails stating the event has been changed.
• I also manually imported an iCal file into Thunderbird's calendar and it didn't update
• I have screenshots of many of these things and downloaded the iCal files

How it May be Possible to Update a Calendar and Reasons Meetup and Eventbrite Likely Don't Try

• I'll go back and read Fran's accessment of the spec, but the only thing I've seen that remotely seems to update calendar invites is when you send an "Invite" between calendar systems. I tested Google, Outlook, and Yahoo calendars.
• When an "Invite" is sent the calendars will show them as pending, usually a white background to show it's not been accepted.
• Google Calendar updates the pending invites from Google, Outlook, and Yahoo, as soon as an email update arrives EVEN without opening the emails.
• When update emails are sent for invites they update the calendars automatically even when the invite came from another platform (ex. Google changes when receiving updated invites from Outlook or Yahoo).
• So, while it may be possible to emulate what the calendar systems are doing, there's a catch. Those seem to use an "Invite" and it's potentially very confusing to the end user if they receive an Invite for an event they just registered for on a website.
• Likewise, if someone were to get an "Invite" for their calendar, then they may incorrectly assume that changing their RSVP to "No" on their calendar would cancel their RSVP, which wouldn't do anything on a Chapter instance unless we were running the same sort of calendar system used by the big tech companies.

[ojeytonwilliams]

It looks like Meetup is the least bad in that they send something. We can easily implement that.

I dug into the spec a little and it seems that the key property is METHOD which is defined here.

REQUEST
Make a request for an event. This is an explicit invitation to one or more "Attendees". Event Requests are also used to update or change an existing event. Clients that cannot handle REQUEST may degrade the event to view it as an PUBLISH.

I'm guessing the magic happens (if it happens at all) if you send a REQUEST iCalendar with the same ids at the first invite.

As for "Invites", I think the magic is happening behind the scenes because Google Calendar updates even if you don't email the attendees. Presumably because it's all handled by Google.

That said, they also email you a new .ics file which is exactly the same as the original invite, but with any changes to the event (date changes, etc) and updates to LAST-MODIFIED and SEQUENCE, so that calendar software can figure out which .ics is most recent. I tested this by importing the invite .ics and then importing the update .ics into Google Calendar and it behaved exactly as you'd expect: the original invite appeared and was replaced by the update.

At the very least we could mirror that, using something like https://www.npmjs.com/package/ics and tracking SEQUENCE in the db.

[allella]

@ojeytonwilliams yes, I noticed the REQUEST in the iCal files from Google, Outlook, and Yahoo calendar invites. I have copies of all the files from Meetup, Eventbrite, and the calendar-to-calendar iCal files that did the magic updates.

The takeaway for me is that magic updating isn't likely an MVP.

Even if it's easy to implement the REQUEST / invite iCal files, there are potential issues with sending a member an "invite" for something they just RSVPed for an instance of Chapter. I think the default will ask the recipient "Yes" / "No" for if they are coming and it's easy to imagine that someone receives that and makes false assuptions.

For example, if someone on a waitlist received an invite with a "Yes" / "No" they might assume a "Yes" reply means they are confirming when the reality is they may still be on the waitlist.

Similarly, a member who RSVP'ed on the Chapter site may change their reply to "No" in their calendar and assume that cancels their reservation on the website, which wouldn't be the case unless we did something super slick and hosted our own service to receive responses to those calendar invites / REQUEST iCal files.

The PUBLISH method is used Meetup iCal files, which is why it doesn't attempt to update the calendar on subsequent iCals for the same event.

BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Meetup//RemoteApi//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
...

It seems the iCal from Eventbrite doesn't even declare a method, so that's acting similar to METHOD:PUBLISH

BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
URL:https://www.eventbrite.com/e/modified-mystery-eventbrite-test-tickets-211201267677
DTSTART:20211223T093000Z
DTEND:20211223T100000Z
SUMMARY:Modified - Mystery Eventbrite Test
DESCRIPTION:For details, go here:\nhttps://www.eventbrite.com/e/modified-mystery-eventbrite-test-tickets-211201267677
LOCATION:Test Greenville - 101 N Main St. - Suite 302 - Greenville, SC 29601
END:VEVENT
END:VCALENDAR

In practice, Evenbrite relies more on the "Add to Google", "Add to Outlook", "Add to Yahoo" links in their emails and doesn't actually send any iCals. There's a link to download an iCal in the email and you can trigger an iCal download after an RSVP,

https://clicks.eventbrite.com/f/a/jXPz1tS0dgH82rrHrGukMQ~~/AAQxAQA~/RgRjc9F-P0SoaHR0cHM6Ly93d3cuZXZlbnRicml0ZS5jb20vY2FsZW5kYXIuaWNzP3V0bV9jYW1wYWlnbj1vcmRlcl9jb25maXJtJnV0bV9tZWRpdW09ZW1haWwmcmVmPWVlbWFpbG9yZGNvbmYmZWlkPTIxMTIwMTI2NzY3NyZ1dG1fc291cmNlPWV2ZW50YnJpdGUmdXRtX3Rlcm09aWNhbCZjYWxlbmRhcj1pY2FsVwNzcGNCCmGM_p2SYQVv40pSFGppbUBmaXhpbnRvc2NyYXAuY29tWAQAAAAA
but the iCal is secondary to the "Add to xxxxxxx" links, which use custom URLs like below.

Add to Google
https://clicks.eventbrite.com/f/a/K_3DMC3CT0lZ3-qrukHtNw~~/AAQxAQA~/RgRjc9F-P0SvaHR0cHM6Ly93d3cuZXZlbnRicml0ZS5jb20vY2FsZW5kYXIuaWNzP3V0bV9jYW1wYWlnbj1vcmRlcl9jb25maXJtJnV0bV9tZWRpdW09ZW1haWwmcmVmPWVlbWFpbG9yZGNvbmYmZWlkPTIxMTIwMTI2NzY3NyZ1dG1fc291cmNlPWV2ZW50YnJpdGUmdXRtX3Rlcm09Z29vZ2xlY2FsJmNhbGVuZGFyPWdvb2dsZVcDc3BjQgphjP6dkmEFb-NKUhRqaW1AZml4aW50b3NjcmFwLmNvbVgEAAAAAA~~

Add to Outlook
https://clicks.eventbrite.com/f/a/lexS4-F4yORZ2sXbCR0H_w~~/AAQxAQA~/RgRjc9F-P0SuaHR0cHM6Ly93d3cuZXZlbnRicml0ZS5jb20vY2FsZW5kYXIuaWNzP3V0bV9jYW1wYWlnbj1vcmRlcl9jb25maXJtJnV0bV9tZWRpdW09ZW1haWwmcmVmPWVlbWFpbG9yZGNvbmYmZWlkPTIxMTIwMTI2NzY3NyZ1dG1fc291cmNlPWV2ZW50YnJpdGUmdXRtX3Rlcm09b3V0bG9vayZjYWxlbmRhcj1vdXRsb29rVwNzcGNCCmGM_p2SYQVv40pSFGppbUBmaXhpbnRvc2NyYXAuY29tWAQAAAAA

Add to Yahoo
https://clicks.eventbrite.com/f/a/u_Q_8jQfqAGoF1GER5pFBw~~/AAQxAQA~/RgRjc9F-P0StaHR0cHM6Ly93d3cuZXZlbnRicml0ZS5jb20vY2FsZW5kYXIuaWNzP3V0bV9jYW1wYWlnbj1vcmRlcl9jb25maXJtJnV0bV9tZWRpdW09ZW1haWwmcmVmPWVlbWFpbG9yZGNvbmYmZWlkPTIxMTIwMTI2NzY3NyZ1dG1fc291cmNlPWV2ZW50YnJpdGUmdXRtX3Rlcm09eWFob29jYWwmY2FsZW5kYXI9eWFob29XA3NwY0IKYYz-nZJhBW_jSlIUamltQGZpeGludG9zY3JhcC5jb21YBAAAAAA~

[ojeytonwilliams]

The takeaway for me is that magic updating isn't likely an MVP.

I agree. If we can get the magic working, great, but I think you're right - it's probably too much for the MVP.

something super slick and hosted our own service to receive responses to those calendar invites / REQUEST iCal files.

This sounds tough. It sounds cool, but I'm not sure where to start! Definitely something that will need a bit of thought and prototyping before we can proceed.

[ojeytonwilliams]

What I thought about this recently:

What we currently do

At this time, all of Chapters interactions with users are via the app itself or email.

Why this is an issue

It creates additional friction for users, since they will have to update their calendars manually. Potentially resulting in missed events and frustration.

What we can do

Make use of the Google calendar api to update user's calendars for them.

Potential issues

We'd need to investigate the api, integrate with Google auth, fallback to email for users that don't want to use Google's services and make sure that we store refresh tokens securely.

In addition, it's possible that we may need to verify with Google, depending on precisely which scopes we need. Calendar is probably fine, but if we need to send emails through Google, that might require verification.

So, in conclusion this is potentially a little involved, but it would be great to get it in the MVP. I recommend we tackle this last, though, simply because it will take a while to get right and we can beta test the email-only version while we do that.

[allella]

As with Quincy's Google calendar invite for today's meeting, it's necessary to send an .ics file with
METHOD:REQUEST
to have any chance of the .ics files within an email actually updating the calendar.

Google manages to make this work, and it may still be a decent standard, but there are questions about if it's the best approach for Chapter.

[allella]

Quincy is most interested in exploring the Google Calendar API as there are questions on if the iCal / webcal method is reliable.

[allella]

@ojeytonwilliams

As mentioned in the last comment, we determined Quincy and others aren't convinced the .ics file protocol for auto-updating calendar events is the way to go.

It may work okay, but this also assumes we're able to send a .ics file to an email that uses the METHOD:REQUEST and does not include a "Will you attend?" (Yes, No, Maybe).

As mentioned above, we could easily confuse a user if they are clicking (Yes, No, Maybe) in their local calendar, but setting a false assumption that their local calendar response will update their RSVP in Chapter.

.ics would be less of an issue if it works and we can send an "invite" without the (Yes, No, Maybe) options, which may not be possible. This may also be why Meetup and Eventbrite don't send an invite through .ics because it could easily confuse people. In particular, if there's a wait list for an event and you send an "invite" to a local calendar, then someone may assume that's a confirmation when it was just a reminder of the event details.

I'm not sure if a Google Calendar API approach will avoid these potential pitfalls, but checking that out seems to be the next step after today's meeting.

[ojeytonwilliams]

I'm not sure if a Google Calendar API approach will avoid these potential pitfalls, but checking that out seems to be the next step after today's meeting.

@allella Neither am I, to be honest, but testing out the Google Calendar API is very high priority right now. We'll know soon enough if it's workable and if it addresses these concerns. If not, we can do some thinking about how to avoid them.

[gikf]

Is there a more specific expected/ideal workflow for this?

Looking at the documentation, generally two pieces are needed to be able to edit events - calendar id and event id. Event id can be defined when creating event. For calendar id there's couple options:

• Require user to provide calendar id of the existing calendar.
• Use the primary calendar of user.
• Create new calendar for user. This option would require slightly higher scope (permission).

[ojeytonwilliams]

I've only just started looking into this, but it seems that basically everything of interest requires a sensitive scope and must be audited by Google before the app can be published. Even if we want to create events on calendars that we own, we'd still need to access a sensitive scope: /auth/calendar.events.owned

The exception is /auth/calendar.app.created which lets you

Make secondary Google Calendars and see, create, change and delete events on them

In principle this is enough, but I've not tested it yet.

However, Google's guide recommends using the https://www.googleapis.com/auth/calendar scope - which gives you full access, so I'm not sure if secondary calendars will work. We'll see.

Anyways, to your comment:

For calendar id there's couple options:

• Require user to provide calendar id of the existing calendar.
• Use the primary calendar of user.
• Create new calendar for user. This option would require slightly higher scope (permission).

@gikf I could be wrong, but I believe the user in question would be a chapter admin. The calendar should be one they have access to. i.e. when a Chapter is created someone will need to create a Calendar for it and then admins would be granted access. We can store that calendar id in the db and use it when creating events. Ideally all of this would be on a secondary calendar, so that we don't need any sensitive scopes.

All of this comes with the caveat that I'm still learning the Google Calendar API, but this is my current understanding.

[gikf]

If calendar would be created externally https://www.googleapis.com/auth/calendar.events scope might be enough - https://developers.google.com/calendar/api/v3/reference/events/insert, but that's also sensitive scope.

After event is created on the Chapter calendar, how that would be connected with the user taking part in the event? Do they add manually add Chapter calendar, or would be adding individual events?

[ojeytonwilliams]

If calendar would be created externally https://www.googleapis.com/auth/calendar.events scope might be enough - https://developers.google.com/calendar/api/v3/reference/events/insert, but that's also sensitive scope.

It would be sensitive, so assuming the secondary calendar approach fails, we'll need sensitive scopes. However, I'd very much prefer to avoid using https://www.googleapis.com/auth/calendar if possible and, yeah, https://www.googleapis.com/auth/calendar.events might be enough.

After event is created on the Chapter calendar, how that would be connected with the user taking part in the event? Do they add manually add Chapter calendar, or would be adding individual events?

My understanding is that the Chapter admin would create an event where the attendees are all the users that RSVP'd to that event. They'd still have to confirm/reject the invite, but once they've confirmed it should be added to their calendar. At that point it's Google's responsibility to keep them updated with any changes to the event.

https://developers.google.com/calendar/api/concepts/sharing#inviting_attendees_to_events explains this (and a few other things)

[ojeytonwilliams]

Related: #1004 in that we want to make sure we and Google collectively only send one reminder. Also, I think people would expect unsubscribing to prevent them getting any notifications, including ones from Google Calendar.

[ojeytonwilliams]

@allella since the Google calendar integration seems to be working, I think we have enough functionality for the MVP. As we get user feedback, I'll add more issues, but I think this issue is resolved.