New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[stable11] Add DAV repair step to fix calendar data #2807

Merged
merged 1 commit into from Dec 22, 2016

Conversation

Projects
None yet
7 participants
@rullzer
Member

rullzer commented Dec 21, 2016

Backport of #2273

Add DAV repair step to fix calendar data
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
@mention-bot

This comment has been minimized.

Show comment
Hide comment
@mention-bot

mention-bot Dec 21, 2016

@rullzer, thanks for your PR! By analyzing the history of the files in this pull request, we identified @nickvergessen and @LukasReschke to be potential reviewers.

mention-bot commented Dec 21, 2016

@rullzer, thanks for your PR! By analyzing the history of the files in this pull request, we identified @nickvergessen and @LukasReschke to be potential reviewers.

@MorrisJobke

Tested and works 👍

@LukasReschke LukasReschke merged commit 40d8025 into stable11 Dec 22, 2016

4 checks passed

codecov/patch 94.44% of diff hit (target 57.20%)
Details
codecov/project 57.22% (+0.02%) compared to cd4b15b
Details
continuous-integration/drone/pr the build was successful
Details
continuous-integration/drone/push the build was successful
Details

@LukasReschke LukasReschke deleted the stable11-2273 branch Dec 22, 2016

@michauk

This comment has been minimized.

Show comment
Hide comment
@michauk

michauk Feb 7, 2017

Hi,
I think the live repair of incorrect cal entries isn't correct (complete?).

I switched from OC 9.1.3 to NC 10.0.x, no problem
Then I upgraded from 10.x to 11.0, calendar broken with this "Unsupported VALUE parameter for DTEND property" in the logs. After some googling, I found all these github issues 2741, this one, 2273 and so on.

I've done again the upgrade (restored to oc 9 and so on) several times, to be sure (I wasn't sure if I had to have the calendar plugin reactivated while in transition NC 10, before upgrading to 11.0.1). Same problem, every time.

I mass-edited some cal entries with this bad DTEND property. Now the logs (logging menu in NC adminitration) don't show any error (all debug levels enabled) but I have even more calendars loading endlessly. I don't think I have bad entries because of the mass-editing. But in any case, the upgrade from 10 to 11 doesn't correct the entries that make NC 11 load endlessly.

Can't move to NC with this bug. I switched back to OC 9.1.3
I can help find the incorrect entries if needed. My different calendars are many years old, a looooong time ago in google calendars, moved to owncloud (7 if I'm not wrong, some years ago) and upgraded little by little to 9.1.3. My cal entries have been created with different tools (thunderbird, some android apps, directly in the OC web interface...), I might have some entries NC 11.0.1 doesn't like any more.

Regards,

JAcques M.

michauk commented Feb 7, 2017

Hi,
I think the live repair of incorrect cal entries isn't correct (complete?).

I switched from OC 9.1.3 to NC 10.0.x, no problem
Then I upgraded from 10.x to 11.0, calendar broken with this "Unsupported VALUE parameter for DTEND property" in the logs. After some googling, I found all these github issues 2741, this one, 2273 and so on.

I've done again the upgrade (restored to oc 9 and so on) several times, to be sure (I wasn't sure if I had to have the calendar plugin reactivated while in transition NC 10, before upgrading to 11.0.1). Same problem, every time.

I mass-edited some cal entries with this bad DTEND property. Now the logs (logging menu in NC adminitration) don't show any error (all debug levels enabled) but I have even more calendars loading endlessly. I don't think I have bad entries because of the mass-editing. But in any case, the upgrade from 10 to 11 doesn't correct the entries that make NC 11 load endlessly.

Can't move to NC with this bug. I switched back to OC 9.1.3
I can help find the incorrect entries if needed. My different calendars are many years old, a looooong time ago in google calendars, moved to owncloud (7 if I'm not wrong, some years ago) and upgraded little by little to 9.1.3. My cal entries have been created with different tools (thunderbird, some android apps, directly in the OC web interface...), I might have some entries NC 11.0.1 doesn't like any more.

Regards,

JAcques M.

@jokakilla

This comment has been minimized.

Show comment
Hide comment
@jokakilla

jokakilla Feb 9, 2017

Hi,
I had some trouble with the calendar parsing too.
After upgrading from 10.0.3 to 11.0.1 the calendar was endlessly loading.
The logs showed something like this:

Sabre\VObject\InvalidDataException: Unsupported VALUE parameter for DTEND property. You supplied ""

/var/www/html/owncloud/3rdparty/sabre/vobject/lib/Parser/MimeDir.php - line 453: Sabre\VObject\Document->createProperty('DTEND', NULL, Array)
/var/www/html/owncloud/3rdparty/sabre/vobject/lib/Parser/MimeDir.php - line 234: Sabre\VObject\Parser\MimeDir->readProperty('DTEND;TZID=Euro...')

I've exported the calendar entries from the DB and checked the format. All entries seem to have a consistent DTEND format.

DTEND;TZID=Europe/Berlin;VALUE=:20140517T180000

To me it looks like the readProperty method has an issue parsing the line.
I hope someone can dig deeper into this issue. I don't have an experimental installation to do it and needed to go back to 10.0.3.

jokakilla commented Feb 9, 2017

Hi,
I had some trouble with the calendar parsing too.
After upgrading from 10.0.3 to 11.0.1 the calendar was endlessly loading.
The logs showed something like this:

Sabre\VObject\InvalidDataException: Unsupported VALUE parameter for DTEND property. You supplied ""

/var/www/html/owncloud/3rdparty/sabre/vobject/lib/Parser/MimeDir.php - line 453: Sabre\VObject\Document->createProperty('DTEND', NULL, Array)
/var/www/html/owncloud/3rdparty/sabre/vobject/lib/Parser/MimeDir.php - line 234: Sabre\VObject\Parser\MimeDir->readProperty('DTEND;TZID=Euro...')

I've exported the calendar entries from the DB and checked the format. All entries seem to have a consistent DTEND format.

DTEND;TZID=Europe/Berlin;VALUE=:20140517T180000

To me it looks like the readProperty method has an issue parsing the line.
I hope someone can dig deeper into this issue. I don't have an experimental installation to do it and needed to go back to 10.0.3.

@jokakilla

This comment has been minimized.

Show comment
Hide comment
@jokakilla

jokakilla Feb 9, 2017

Ok after having a look at pull request #2807 it seems like the repair step would convert

DTEND;TZID=Europe/Berlin;VALUE=:20140517T180000

into

DTEND;TZID=Europe/Berlin:20140517T180000

But how is the repair triggered?

jokakilla commented Feb 9, 2017

Ok after having a look at pull request #2807 it seems like the repair step would convert

DTEND;TZID=Europe/Berlin;VALUE=:20140517T180000

into

DTEND;TZID=Europe/Berlin:20140517T180000

But how is the repair triggered?

@nickvergessen

This comment has been minimized.

Show comment
Hide comment
@nickvergessen
Member

nickvergessen commented Feb 10, 2017

@jokakilla

This comment has been minimized.

Show comment
Hide comment
@jokakilla

jokakilla Feb 10, 2017

Second try upgrading today. It's working now after manually repairing the calendar entries.

  • Upgrade
  • Export oc_calendarobjects table as CSV
  • Replace "Berlin;VALUE=:" by "Berlin:" (suggested solution of my last comment)
  • Flushed database and reimported from CSV
  • All fine

Just wonder why the repair job of NC didn't do that automatically.

jokakilla commented Feb 10, 2017

Second try upgrading today. It's working now after manually repairing the calendar entries.

  • Upgrade
  • Export oc_calendarobjects table as CSV
  • Replace "Berlin;VALUE=:" by "Berlin:" (suggested solution of my last comment)
  • Flushed database and reimported from CSV
  • All fine

Just wonder why the repair job of NC didn't do that automatically.

@michauk

This comment has been minimized.

Show comment
Hide comment
@michauk

michauk Feb 13, 2017

Thanks to @jokakilla , I managed to upgrade to nextcloud 10 then 11. But some problems should still be considered by the team, I suppose:

You can correct your faulty calendar entries with something like :
UPDATE oc_calendarobjects SET calendardata = REPLACE(calendardata, "DTEND;TZID=Europe/Paris;VALUE=:", "DTEND;TZID=Europe/Paris:");
Someone might use a correct regular expression to replace Europe/Paris by anything like "some/city"
Be careful, test it on a specific calendar row if needed (add AND id=xxx), spot an faulty entry with something like :
SELECT id, calendardata FROM oc_calendarobjects WHERE calendardata LIKE '%DTEND;TZID=Europe/Paris;VALUE:%';
It fixed approx 250 entries in my case O_O

But, new problem, as I couldn't migrate from OC to NC, some days ago, I upgraded my OC 9.1.3 to 9.1.4.
The OC->NC migration doc says you can migrate from OC9.1.x to NC10.x but it's not entirely true, the latest 9.1.4 is not correctly handled. I decided to test this suggestion : https://help.nextcloud.com/t/owncloud-9-1-4-2-migration-to-nextcloud-not-working/8630 ; in a few words : fake the version so the upgrade command doesn't complain.

Then I could migrate to NC10 and 11 right after, disabled maintenance mode and my calendars aren't loading forever anymore. whoaaaa.
I tried some sync, both ways, from my android, thunderbird, nextcloud ;used my carddav/caldav app on android. Everything seems OK.

I hope all this will help.

Regards,

DO A CODE AND DB BACKUP BEFORE PLAYING WITH IT ALL.

michauk commented Feb 13, 2017

Thanks to @jokakilla , I managed to upgrade to nextcloud 10 then 11. But some problems should still be considered by the team, I suppose:

You can correct your faulty calendar entries with something like :
UPDATE oc_calendarobjects SET calendardata = REPLACE(calendardata, "DTEND;TZID=Europe/Paris;VALUE=:", "DTEND;TZID=Europe/Paris:");
Someone might use a correct regular expression to replace Europe/Paris by anything like "some/city"
Be careful, test it on a specific calendar row if needed (add AND id=xxx), spot an faulty entry with something like :
SELECT id, calendardata FROM oc_calendarobjects WHERE calendardata LIKE '%DTEND;TZID=Europe/Paris;VALUE:%';
It fixed approx 250 entries in my case O_O

But, new problem, as I couldn't migrate from OC to NC, some days ago, I upgraded my OC 9.1.3 to 9.1.4.
The OC->NC migration doc says you can migrate from OC9.1.x to NC10.x but it's not entirely true, the latest 9.1.4 is not correctly handled. I decided to test this suggestion : https://help.nextcloud.com/t/owncloud-9-1-4-2-migration-to-nextcloud-not-working/8630 ; in a few words : fake the version so the upgrade command doesn't complain.

Then I could migrate to NC10 and 11 right after, disabled maintenance mode and my calendars aren't loading forever anymore. whoaaaa.
I tried some sync, both ways, from my android, thunderbird, nextcloud ;used my carddav/caldav app on android. Everything seems OK.

I hope all this will help.

Regards,

DO A CODE AND DB BACKUP BEFORE PLAYING WITH IT ALL.

@rullzer

This comment has been minimized.

Show comment
Hide comment
@rullzer

rullzer Mar 9, 2017

Member

@jokakilla @michauk sorry for the late reply (but I'm glad you figured it out). Since we wanted to be a bit smarted when migrating and making upgrade faster the fix is not run on upgrade. Since if you have a large large enough instance that would make it all painfully slow.

So if cron would have been triggered a few times it should have launched the repair job.

Anyway hope all is fixed already!

Member

rullzer commented Mar 9, 2017

@jokakilla @michauk sorry for the late reply (but I'm glad you figured it out). Since we wanted to be a bit smarted when migrating and making upgrade faster the fix is not run on upgrade. Since if you have a large large enough instance that would make it all painfully slow.

So if cron would have been triggered a few times it should have launched the repair job.

Anyway hope all is fixed already!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment