Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duplicate event for recurring meeting instance exception #129

Closed
jasonheffner opened this issue Apr 20, 2017 · 30 comments
Closed

Duplicate event for recurring meeting instance exception #129

jasonheffner opened this issue Apr 20, 2017 · 30 comments

Comments

@jasonheffner
Copy link

jasonheffner commented Apr 20, 2017

  • PHP Version: 5.6.30
  • PHP date.timezone: America/New_York

Description of the Issue:

Duplicate meetings for recurring meetings. This happens on meetings that have single occurrences that have different times.

Steps to Reproduce:

Example iCal from Zimbra. On 20/04/2017 there should be 2x 1 hour meetings at 9:00 and 15:30 14:00. Instead 4x 3x meetings are returned. Two One is a duplicate.

@u01jmg3 u01jmg3 changed the title duplicate event for recurring meeting instance exception Duplicate event for recurring meeting instance exception Apr 20, 2017
@u01jmg3 u01jmg3 self-assigned this Apr 20, 2017
@u01jmg3
Copy link
Owner

u01jmg3 commented Apr 20, 2017

  • Looking at your iCal and focussing on 20-04-2017, the parser returns 3 events.
  • These 3 events are defined separately in your iCal so the parser is correct with its output.

demo


  • In total your iCal contains 7 events but here are the 3 that have events occurring on 20-04-2017:
BEGIN:VEVENT
UID:e12d2536-d35b-4917-9573-5670003d9802
RRULE:FREQ=WEEKLY;UNTIL=20170601T035959Z;INTERVAL=2;BYDAY=TH
SUMMARY:CREATE Bi-weekly mtgs.
DESCRIPTION:\n
X-ALT-DESC;FMTTYPE=text/html:<html><body id='htmlmode'></body></html>
LOCATION:Shields (J Franklin)\,003C\,UP
DTSTART;TZID="America/New_York":20170420T090000
DTEND;TZID="America/New_York":20170420T100000
STATUS:CONFIRMED
CLASS:PUBLIC
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
TRANSP:OPAQUE
LAST-MODIFIED:20170315T201152Z
DTSTAMP:20170315T201152Z
SEQUENCE:0
BEGIN:VALARM
ACTION:DISPLAY
TRIGGER;RELATED=START:-PT15M
DESCRIPTION:Reminder
END:VALARM
END:VEVENT

BEGIN:VEVENT
UID:e9968032-846d-4c26-8839-e2c49bbbf9d3
SUMMARY:Student Forecasting/ Advising Project Meeting
LOCATION:shieldsjfranklin-003c-up
DTSTART;TZID="America/New_York":20170420T140000
DTEND;TZID="America/New_York":20170420T150000
STATUS:CONFIRMED
CLASS:PUBLIC
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
TRANSP:OPAQUE
RECURRENCE-ID;TZID="America/New_York":20170420T153000
LAST-MODIFIED:20170414T175947Z
DTSTAMP:20170414T175947Z
SEQUENCE:2
BEGIN:VALARM
ACTION:DISPLAY
TRIGGER;RELATED=START:-PT15M
DESCRIPTION:Reminder
END:VALARM
END:VEVENT

BEGIN:VEVENT
UID:e9968032-846d-4c26-8839-e2c49bbbf9d3
RRULE:FREQ=WEEKLY;UNTIL=20170519T035959Z;INTERVAL=1;BYDAY=TH
SUMMARY:Student Forecasting/ Advising Project Meeting
LOCATION:shieldsjfranklin-003c-up
DTSTART;TZID="America/New_York":20170406T153000
DTEND;TZID="America/New_York":20170406T163000
STATUS:CONFIRMED
CLASS:PUBLIC
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
TRANSP:OPAQUE
LAST-MODIFIED:20170208T154316Z
DTSTAMP:20170208T154316Z
SEQUENCE:1
EXDATE;TZID="America/New_York":20170427T153000
BEGIN:VALARM
ACTION:DISPLAY
TRIGGER;RELATED=START:-PT15M
DESCRIPTION:Reminder
END:VALARM
END:VEVENT

@jasonheffner
Copy link
Author

jasonheffner commented Apr 20, 2017

The one event is a single occurrence of a repeating event. In your translation of the iCal you'll notice the EXDATE tag. That should replace the recurring event above that. The 14:00 15:30 meeting should not show up. In my linked gist version it's denoted as RECURRENCE-ID.

@u01jmg3
Copy link
Owner

u01jmg3 commented Apr 21, 2017

Gotcha.

There is an issue of time zones being re-applied when populating alteredRecurrenceInstances() which controls RECURRENCE-ID info.

However, when fixing this, it is the 15:30 event which is removed not the 14:00?

@jasonheffner
Copy link
Author

jasonheffner commented Apr 21, 2017

That's correct, I had it backward. The 15:30 meeting should not be displayed. There should only be two meetings for that day then.

What is the fix for the timezones then?

@u01jmg3 u01jmg3 added this to the v2.x.x milestone Apr 21, 2017
@jasonheffner
Copy link
Author

I believe this also may be happening with EXDATE

@u01jmg3
Copy link
Owner

u01jmg3 commented Apr 25, 2017

Thanks - yes it all goes through iCalDateToUnixTimestamp() which needs some tweaking.

@jasonheffner
Copy link
Author

jasonheffner commented Apr 26, 2017

I attempted a patch to try and return the correct timestamp. This worked on almost all my test files except one, where I believe the recurring event uses a different TZ format. I believe that the recurring events also may need to be parsed by timestamp with the correct conversions possibly with the same function.

ical.patch.txt

Here was my PHP code to display the entries

date_default_timezone_set('America/New_York');
use ICal\ICal;
$ical = new ICal('ical.ics', array(
    'defaultSpan'           => 2,     // Default value
    'defaultWeekStart'      => 'MO',  // Default value
    'skipRecurrence'        => false, // Default value
    'useTimeZoneWithRRules' => false, // Default value
));
$events = $ical->eventsFromRange($dtStartDay, $dtEndDay);

foreach ($events as $event) {
    $dtStart = new \DateTime('now', new \DateTimeZone('America/New_York'));
    $dtEnd = new \DateTime('now', new \DateTimeZone('America/New_York'));
    $dtStart->setTimestamp((int) $ical->iCalDateToUnixTimestamp($event->dtstart));
    $dtEnd->setTimestamp((int) $ical->iCalDateToUnixTimestamp($event->dtend));
    echo sprintf('  <li>%s - %s : %s</li>' . PHP_EOL, $dtStart->format('r'), $dtEnd->format('r'), $event->summary);
}

I believe I might have taken a different approach and not sure if this helps.

@u01jmg3
Copy link
Owner

u01jmg3 commented Apr 30, 2017

Thanks for your efforts - your patch actually crashes my SourceTree but opening in an editor I get the gist of your change. Still working on an effective resolution. Hope to have something soon.

@jasonheffner
Copy link
Author

The patch I did has been working for a few weeks now, well. The gist of the change is setting the TimeZone first then the time such that the conversions are done properly, inside the datetime object, as compared trying to figure out the time difference afterward and adjusting. You also have to explicitly set the TZ for the TimeDate object as I observed it seems to like to inherit this setting from elsewhere; unfortunately, I don't use TimeDate often enough to know best practice.

@u01jmg3
Copy link
Owner

u01jmg3 commented May 16, 2017

Wow 16 days has flown by - apologies, I have been away. Will look at this again shortly

@jasonheffner
Copy link
Author

Here is what I've been running.

jasonheffner@db0049a#diff-50681920b8994c51dcfbb9a44850348f

I have one meeting every other week that seems to break with recurrence I still need to look into.

u01jmg3 added a commit that referenced this issue May 27, 2017
…ly the event time zone if explicitly requested
@u01jmg3
Copy link
Owner

u01jmg3 commented May 27, 2017

Finally made time to look at this. Using your diff I have tailored your code to provide a slightly different fix. The reason being I am dubious whether the time zone should even be applied when iCalDateToUnixTimestamp() is called because really its function should only be to convert a datetime to its UNIX timestamp equivalent.

What I was finding before with your example was that the time zone was being re-applied to the 15:30 event when iCalDateToUnixTimestamp() was used. This meant it became 11:30 (-4 hours). This in turn meant the recurrence ID logic that was in place couldn't match correctly and thus you had a duplicate event because the parser did not know the 15:30 event (it saw as 11:30) should be replaced by the 14:00 one.

See what you think.


I have one meeting every other week that seems to break with recurrence I still need to look into.

  • Is it still present after using the most recent fix (06275bb)?
  • Can you provide the iCal for me to debug?

@jasonheffner
Copy link
Author

jasonheffner commented May 31, 2017

Sorry for the delay after the holiday. I took a look at the code and tested it on a Month's worth of past days in April. It's closer but I do believe that you need to set the TimeZone each time for the TimeDate object based on the entry in the ical file. Currently, the code is working "most of the time" but I found a few days where it breaks since the TZ format changes and for some complex rules.

It's working "relative" to your current TZ. For example, if you are working with everything in the same TZ as your default TZ set it would work fine, but if you are working with dates outside your current TZ, ie DST, then the timestamps returned are incorrect and skewed. If you look at the "timestamp" returned by iCalDateToUnixTimestamp() being your "test" for correctness it is setting the wrong date/time because of the TZ. I believe this is also why I still have some problems with some of the complex recurrences and exclusions for some of our meetings.

@u01jmg3
Copy link
Owner

u01jmg3 commented May 31, 2017

Thanks - could you provide the test data which you are using?

@jasonheffner
Copy link
Author

I shared some unedited versions with you to help troubleshoot.

@u01jmg3
Copy link
Owner

u01jmg3 commented May 31, 2017

Thanks - this will help a lot. Can you point me to a few problematic events and what is wrong with them and what you would expect them to be?

Perhaps a README in the repo with this info would give me a good starting point.

@jasonheffner
Copy link
Author

jasonheffner commented May 31, 2017

Updated with a readme .. hopefully, illustrates the wrong timestamp set

u01jmg3 referenced this issue Jun 1, 2017
Enhance Event object to provide access to the ICal object and its functions
@u01jmg3
Copy link
Owner

u01jmg3 commented Jun 1, 2017

  • Been through your examples and made numerous changes in a separate branch to correct the issues you found

  • Gist of it is, you can now get the UNIX timestamp you are looking for when iterating through the events using:

    $ical->iCalDateToUnixTimestamp($event->dtstart_array[3], $forceTimeZone = true);
  • I have also corrected the logic for matching recurrence events by making the comparison aware of time zone data - goodbye dupes

@jasonheffner
Copy link
Author

jasonheffner commented Jun 2, 2017

It's looking good .. the dupes are also gone in the branch which is an extra bonus. I was wondering why the need for the extra config parameter of $forceTimeZone, shouldn't this be done in all cases to have the correct timestamp delivered?

If you wanted to force TZ wouldn't that be for overriding the one supplied in the iCal file already, meaning the option would be the opposite case? More trying to understand what you are going for. Perhaps it's just a difference in what we think the default handling should be. You are pulling the TZ out of iCal so thought you would want to use it by default.

I tested in a few other dates and I noticed no dupes so it's working great.

@u01jmg3
Copy link
Owner

u01jmg3 commented Jun 3, 2017

Thanks for taking the time to QA

I was wondering why the need for the extra config parameter for "forceTimeZone", shouldn't this be done in all cases to have the correct timestamp delivered?

This is more me being cautious as it has been tricky to nail down time zone logic in the parser with various other issues being raised. Once things are stable I will look to remove but for now I feel happier having it.

If you wanted to force TZ wouldn't that be for overriding the one supplied in the iCal file already, meaning the option would be the opposite case?

You're right - this could become confusing but as above I hope to phase out the parameter once the dust has settled and things become stable. I could rename the variable?

Once you are happy I'll raise a PR and if you have time, could you review and then I will merge into master?

@jasonheffner
Copy link
Author

Makes sense.. and it's easy to switch the default value later. I think the variable name works and I can set it to true for our parsing needs. It corrects all the parsing errors I've seen in the last month. This is the most complete ical parser I found for PHP. I've tried several, all with different problems.

I like how you extended the class to include the TZ. That's the part I hadn't looked closely enough on how you were handling TZ within recurrences; I guess you weren't. I wasn't sure how best to implement that either and like your solution.

@u01jmg3
Copy link
Owner

u01jmg3 commented Jun 3, 2017

This is the most complete ical parser I found for PHP.

Appreciate the compliment.


Cast your eye over PR #142 and see what you think - made some final changes to refactor and improve the code I had already written but essentially it achieves the same thing and addresses the issues you raised.

You may see some code that looks repetitive and not as neat as it could be - at this point I don't want to introduce any breaking changes to the parser but eventually I will bump to v3.0.0 and take the hit.

@jasonheffner
Copy link
Author

jasonheffner commented Jun 5, 2017

I went through the changes and only a few comments. I see what you did to the function so altered my code to match instead using the datetime object

date_default_timezone_set('America/New_York');

use ICal\ICal;
$ical = new ICal('data.txt', array(
    'defaultSpan'           => 2,     // Default value
    'defaultWeekStart'      => 'MO',  // Default value
    'skipRecurrence'        => false, // Default value
    'useTimeZoneWithRRules' => false, // Default value
));

$events = $ical->eventsFromRange($dtStartDay, $dtEndDay);
$ESTTZ = new DateTimeZone('America/New_York');

foreach ($events as $event) {
    $dtStart = $ical->iCalDateToDateTime($event->dtstart, $forceTimeZone = true);
    $dtEnd = $ical->iCalDateToDateTime($event->dtend, $forceTimeZone = true);
    // Set TZ for display purposes
    $dtStart->setTimezone($ESTTZ);
    $dtEnd->setTimezone($ESTTZ);
}

The setTimezone line #638 is really only needed for displaying the DateTime object in your preferred TZ at the end of the routine, as the times were entered into their corresponding TZs further up. I don't think you would need forceUtc and I would instead set it to the default TZ for consistency. You can see in the code above I set the TZ on both objects returned. I had to do this since they were occasionally, not always, coming back in UTC, hence not consistent; if the DateTime returned was always in my default TZ then I wouldn't have to do this for instance.

I saw quite a bit of repetitive code but never fault anyone for it. I'm a sysadmin so in most cases, it's more important for code to work correctly. I'm figuring at some point you'll go back and clean it up after the bugs and features are addressed.

To note: I tried a lot of the PHP iCal parsers and this is the closest to correctly parsing all the ical features. Each one had it's own issues.. sabre/vobject was the most disappointing as I looked into that one first since GLPI uses it. The sorted eventsFromRange() function was exactly what I needed as well. I appreciate the work.

@u01jmg3
Copy link
Owner

u01jmg3 commented Jun 5, 2017

The setTimezone line #638 is really only needed for displaying the DateTime object in your preferred TZ at the end of the routine, as the times were entered into their corresponding TZs further up. I don't think you would need forceUtc and I would instead set it to the default TZ for consistency.

This was added because when fixing the issue with the duplicates in your iCal, I found that the RECURRENCE-ID dates could be given in different time zones which is not ideal when comparing. I therefore included a $forceUtc flag in order to be able to make a comparison across a consistent time zone (UTC) and thus match when an event has been overridden by another in a recurrence set. Hopefully that makes sense.

You can see in the code above I set the TZ on both objects returned. I had to do this since they were occasionally, not always, coming back in UTC, hence not consistent; if the DateTime returned was always in my default TZ then I wouldn't have to do this for instance.

I could add this as a new feature but as above I would have to keep the $forceUtc functionality.

The sorted eventsFromRange() function was exactly what I needed as well. I appreciate the work.

👍

@jasonheffner
Copy link
Author

jasonheffner commented Jun 5, 2017

Ohh yeah, that makes sense! I was going to suggest "forceUtc or use the default TZ" at first, but now wondering if there could be a use for using the originating TZ from the iCal; so the current pull works for me. I'll just need to set the TZ after the DateTime is returned; perhaps documentation about setting your display TZ after calling that function? That could confuse folks who aren't familiar with DateTime objects if they aren't consistently returned in the same TZ. I'm assuming that function is preferred over using a Unix timestamp since you changed your example.

@u01jmg3
Copy link
Owner

u01jmg3 commented Jun 5, 2017

perhaps documentation about setting your display TZ after calling that function? That could confuse folks who aren't familiar with DateTime objects if they aren't consistently returned in the same TZ.

  • Changes made to the README - see PR.
  • Would using $ical->iCalDateToDateTime($event->dtstart_array[3], $forceTimeZone = true) prevent you having to apply the time zone?

I'm assuming that function is preferred over using a Unix timestamp since you changed your example.

  • Yes - there are several limitations when using a Unix timestamp (e.g. https://bugs.php.net/bug.php?id=66836) but these are removed when returning a DateTime object.
  • For backwards compatibility until I move to v3.0.0, iCalDateToUnixTimestamp() will still be available but I would recommend moving to DateTime objects in the meantime.

@jasonheffner
Copy link
Author

jasonheffner commented Jun 6, 2017

Would using $ical->iCalDateToDateTime($event->dtstart_array[3], $forceTimeZone = true) prevent you having to apply the time zone?

Unfortunately not, I still get some dates returned in the UTC TZ. I don't mind setting the TZ though and can switch to using iCalDateToDateTime().

@u01jmg3
Copy link
Owner

u01jmg3 commented Jun 6, 2017

  • Okay - will merge in the PR and close off this issue.
  • If at any point you think the parser requires extra functionality to support your needs just raise a new issue and I will assist.
  • Thank you for all your help.

u01jmg3 added a commit that referenced this issue Jun 6, 2017
@u01jmg3 u01jmg3 removed their assignment Jun 6, 2017
@u01jmg3 u01jmg3 closed this as completed Jun 6, 2017
@u01jmg3
Copy link
Owner

u01jmg3 commented Jun 6, 2017

See #142

@jasonheffner
Copy link
Author

Ditto! Thanks for your support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants