cache webcal calendars on server #10059

georgehrke · 2018-06-29T17:12:10Z

ToDos:

adapt refresh cycle to refresh rate of subscription
- this could actually be tricky. the interval set in the constructor is called before run, but we only know the corresponding subscription when we know the arguments for run
- we can obviously keep track of the last actual update ourself and just run the job once an hour, but I'd prefer if IJobList could just handle that.
update subscription in database when remote returns 301
unit tests

rullzer · 2018-07-02T07:41:22Z

@georgehrke I'd say for now just write a small abstraction class yourself to solve the extra check issue.
Then if it turns out to work corectly etc we can always merge it back to the server part

MorrisJobke · 2018-07-20T09:58:16Z

Most likely nothing for 14 -> moving to 15.

apps/dav/lib/CalDAV/WebcalCaching/Plugin.php

apps/dav/lib/BackgroundJob/RefreshWebcalJob.php

georgehrke · 2018-10-31T11:29:47Z

apps/dav/lib/Migration/Version1006Date20180628111625.php

+			if ($calendarObjectsTable->hasIndex('calobjects_index')) {
+				$calendarObjectsTable->dropIndex('calobjects_index');
+			}
+			$calendarObjectsTable->addUniqueIndex(['calendarid', 'calendartype', 'uri'], 'calobjects_index');


@nickvergessen Will building new indices be an issue? Should it be outsourced to a migration step?

I think for this it is fine. Should be relatively fast.

However, a step first to see if there are duplicated might be needed.
As else it will just explode

It removes the old index consisting of calendarid, uri and creates a new index calendarid, calendartype, uri. And calendartype is the all new column that was introduced by the same migration and set to 0 for all existing entries. So there should be no duplicates.

Ah right. :) Carry on then

@rullzer I was generally not sure what's the cleanest / best approach here.
We have calendars and subscriptions in different tables: calendars respectively calendarsubscriptions. This was no issue so far since we didn't store objects related to subscriptions.

So basically there are two options

Have duplicate tables of calendarchanges, calendarobjects and calendarobjects_props for subscriptions

Introduce calendartype that indicates whether calendarid refers to the calendars or the subscriptions table.

I'm fine with both ways - even if the second one would be cleaner you need to update all the existing queries, right?

Yes, i guess the big question here is whether introducing the new column calendartype is reducing performance of day-to-day queries.

When it is used in queries just add an index to it and it should be fine.

georgehrke · 2018-10-31T11:35:40Z

@tcitworld Can you please test? Are there any other remarks? :)

apps/dav/lib/BackgroundJob/RefreshWebcalJob.php

georgehrke · 2018-11-07T10:20:20Z

The acceptance test results are unrelated.

The phan thing seems to be a faulty php-doc in Sabre/Dav.
https://github.com/nextcloud/3rdparty/blob/214c4155f587f5178d792fe4a839044bdc9982f1/sabre/vobject/lib/Splitter/ICalendar.php#L87

georgehrke · 2018-11-07T10:21:12Z

Should I just use @phan-suppress-next-line?

georgehrke · 2018-11-07T10:27:17Z

Just wanted to send a pull-request, but it's already fixed upstream: https://github.com/georgehrke/vobject/blob/master/lib/Splitter/ICalendar.php#L86

blizzz · 2018-11-07T10:29:29Z

@georgehrke is the upstream released and reasonable to get this in? Otherwise, let's suppress it with a note/issue to remove the suppression once we pull again.

georgehrke · 2018-11-07T10:31:15Z

The fix was not released yet, adding @phan-suppress-next-line ...

georgehrke · 2018-11-07T10:52:20Z

@MorrisJobke How to update phan?
We are running phan 0.11.1, but I need 0.12.9 to be able to use @phan-suppress-next-line

blizzz · 2018-11-07T11:03:40Z

@georgehrke try to push https://github.com/nextcloud/server/blob/master/.drone.yml#L104

kesselb · 2018-11-07T11:03:52Z

apps/dav/lib/CalDAV/CalDavBackend.php

+	/**
+	 * @param $subscriptionId
+	 */
+	public function getSubscriptionById($subscriptionId) {


When int $subscriptionId works for bigint colums i would use it here.

Signed-off-by: Georg Ehrke <developer@georgehrke.com>

MorrisJobke · 2018-11-07T15:01:21Z

Please update the 3rdparty repo due to
nextcloud/3rdparty#185

Signed-off-by: Georg Ehrke <developer@georgehrke.com>

georgehrke · 2018-11-07T16:06:13Z

Failing tests seem unrelated to me

rullzer

Code looks good to me. Lets do this.

ChristophWurst · 2018-11-12T11:39:56Z

FYI: This caused a regression: #12410

BernieO · 2019-08-22T04:53:00Z

I accidentally stumpled accross this when debugging an abnormality with my script calcardbackup and noticed that this is not in the changelog for Nextcloud 15

This commit adds events with the same calendarid belonging to different calendars ("regular" calendars and "subscribed" calendars). To determine to which calender the events belong there is a new database field calendartype added.

All this are quite big changes which are definitely worth to be mentioned in the changelog. Could this new featur please be added to the changelog?

georgehrke added the 2. developing Work in progress label Jun 29, 2018

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch 4 times, most recently from 9d83b9f to 2cba151 Compare June 30, 2018 12:38

georgehrke added this to the Nextcloud 14 milestone Jun 30, 2018

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch 3 times, most recently from 88c8f76 to 864e40f Compare June 30, 2018 20:58

georgehrke mentioned this pull request Jul 1, 2018

Allow (internal) sharing of subscribed webcal calendars #1600

Open

MorrisJobke modified the milestones: Nextcloud 14, Nextcloud 15 Jul 20, 2018

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch 4 times, most recently from b072922 to c0868f8 Compare October 26, 2018 12:25

tcitworld reviewed Oct 26, 2018

View reviewed changes

apps/dav/lib/CalDAV/WebcalCaching/Plugin.php Show resolved Hide resolved

apps/dav/lib/BackgroundJob/RefreshWebcalJob.php Show resolved Hide resolved

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch 4 times, most recently from fb7c1f4 to 1b17e2f Compare October 31, 2018 11:28

georgehrke requested a review from nickvergessen October 31, 2018 11:28

georgehrke commented Oct 31, 2018

View reviewed changes

georgehrke added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Oct 31, 2018

georgehrke requested a review from rullzer October 31, 2018 12:20

tcitworld previously approved these changes Oct 31, 2018

View reviewed changes

apps/dav/lib/BackgroundJob/RefreshWebcalJob.php Show resolved Hide resolved

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch from 31ac4b6 to cc66f65 Compare November 7, 2018 11:06

kesselb reviewed Nov 7, 2018

View reviewed changes

MorrisJobke mentioned this pull request Nov 7, 2018

15.0.0 beta 1 #12310

Merged

29 tasks

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch 3 times, most recently from 730c48e to 8967e85 Compare November 7, 2018 12:08

cache webcal calendars on server

30d13bb

Signed-off-by: Georg Ehrke <developer@georgehrke.com>

georgehrke mentioned this pull request Nov 7, 2018

manually patch PHP-Doc of VObject Splitter nextcloud/3rdparty#185

Merged

update 3rdparty to fix phan error message

712b79e

Signed-off-by: Georg Ehrke <developer@georgehrke.com>

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch from 8967e85 to 712b79e Compare November 7, 2018 15:11

MorrisJobke added the enhancement label Nov 7, 2018

rullzer approved these changes Nov 8, 2018

View reviewed changes

MorrisJobke merged commit 0fb74af into master Nov 8, 2018

MorrisJobke deleted the feature/1497/cache_webcal_calendars branch November 8, 2018 09:55

ChristophWurst mentioned this pull request Nov 12, 2018

Unknown column 'calendartype' (table calendarobjects) #12410

Closed

nickvergessen mentioned this pull request Nov 14, 2018

Index identifier calendarid_calendartype_synctoken is too long #12446

Closed

BernieO mentioned this pull request Aug 22, 2019

remove orphaned events also from deleted webcal calendars #16833

Closed

tcitworld mentioned this pull request Aug 23, 2019

Remove orphaned calendar data from deleted subscriptions #16835

Merged

redtux mentioned this pull request Sep 14, 2019

Expose subscribed webcal calendars via CalDAV #1497

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cache webcal calendars on server #10059

cache webcal calendars on server #10059

georgehrke commented Jun 29, 2018 •

edited

Loading

rullzer commented Jul 2, 2018

MorrisJobke commented Jul 20, 2018

georgehrke Oct 31, 2018

rullzer Nov 1, 2018

georgehrke Nov 1, 2018

rullzer Nov 1, 2018

georgehrke Nov 1, 2018

MorrisJobke Nov 2, 2018

georgehrke Nov 2, 2018

nickvergessen Nov 7, 2018

georgehrke commented Oct 31, 2018

georgehrke commented Nov 7, 2018 •

edited

Loading

georgehrke commented Nov 7, 2018

georgehrke commented Nov 7, 2018

blizzz commented Nov 7, 2018

georgehrke commented Nov 7, 2018

georgehrke commented Nov 7, 2018

blizzz commented Nov 7, 2018

kesselb Nov 7, 2018

MorrisJobke commented Nov 7, 2018

georgehrke commented Nov 7, 2018

rullzer left a comment

ChristophWurst commented Nov 12, 2018

BernieO commented Aug 22, 2019

cache webcal calendars on server #10059

cache webcal calendars on server #10059

Conversation

georgehrke commented Jun 29, 2018 • edited Loading

rullzer commented Jul 2, 2018

MorrisJobke commented Jul 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgehrke commented Oct 31, 2018

georgehrke commented Nov 7, 2018 • edited Loading

georgehrke commented Nov 7, 2018

georgehrke commented Nov 7, 2018

blizzz commented Nov 7, 2018

georgehrke commented Nov 7, 2018

georgehrke commented Nov 7, 2018

blizzz commented Nov 7, 2018

Choose a reason for hiding this comment

MorrisJobke commented Nov 7, 2018

georgehrke commented Nov 7, 2018

rullzer left a comment

Choose a reason for hiding this comment

ChristophWurst commented Nov 12, 2018

BernieO commented Aug 22, 2019

georgehrke commented Jun 29, 2018 •

edited

Loading

georgehrke commented Nov 7, 2018 •

edited

Loading