Roadmap for v0.8

[N-Coder]

This is a meta-issue to collect all the things we want to have in v0.8 (see also the milestone) and how we want to get there. I'll try to keep this list updated when things change over time.

Update 11.07.2022 with things sorted according to what will be included in a v0.8 alpha release, what we hopefully manage to do before an actual release (or maybe just let the users do all the testing?), and what will be deferred for the next version.

Major Changes included in / missing for v0.8 alpha

• remove arrow, transition to attrs and add Timespan (PR Remove arrow, transition to attrs and add Timespan #222, including the old ToDo list)
• migration to new data conversion implementation (PR [WIP] new implementation of parsing and serialization #240, PR improved ComponentMeta system and Timezone conversion #249)
• new build system (PR New build system #243, Move build system from poetry to hatch #354)
• proper timezone handling (PR improved ComponentMeta system and Timezone conversion #249, Add VTIMEZONE / tzinfo Serialization and better TZID Parsing #129, Time zone conversion support #161)
• faster ContentLine parser implementation (PR Contentline parser #247, V0.7 is much slower to parse large calendar feed then V0.5 #244)
• line folding (ics does not support 5545 3.1 (long line folding) #215, Long DESCRIPTION has embedded newline #210)
• parsing / serialization hooks and customizations (PRs improved ComponentMeta system and Timezone conversion #249, [WIP] Converter System clean-up #355)
• auto-formatting, more automatic checks and fixes, also including the documentation (PR Introduce auto-formatters #351, Docstring formatting #344)
• the ics_vtimezones project should have a new release in sync with our 0.8 release

Missing Tests and Open ToDos for v0.8 release

• improved documentation (Improve the documentation of ics #220, Improve docstrings in classes & functions for API #225, PR Restructure documentation #223)
• finally working all_day event support (make_all_day() issues #155, all-day event is still off by one day #173, Off by one issues in all-day events #92, ...)
• real world test data ([POC] Create a corpus of various ics files for more robust tests #236)
• migration of old test suite
• full test coverage for new code
• tests preventing regressions on old bugs (PR improved ComponentMeta system and Timezone conversion #249)
  Timezone information of time fields is ignored #181, Timestamp seconds are ignored #182, Timezone information is dropped when converting internally to UTC #188, ...
• test compatibility with other tools (Outlook, Apple Calendar, Google, Thunderbird, Evolution)
  esp. with regard to line-folding, attribute preservation and interpretation

Wishlist for v0.9

• recurring events (Support recuring events #14, Add priority, due date and rrule to Event class  #168)
• varying levels of strictness (validate vs. try hard to interpolate the right data)
• webcal, jcal, vcards
• parity with all RFC 5545 / 7986 properties (Support of SEQUENCE property #115, Consider adding fields from RFC-7986 #307)
• "compatibility" serialization for different targets / RFC versions (i.e. Google, Outlook, RFC5545 vs RFC7986, X-WR-CALNAME vs NAME)

[waynew]

Has this been abandoned or is there still progress happening to this release?

[N-Coder]

It's definitely not abandoned, I'm still hoping to make this release happen.
The biggest blocker is probably the timezone handling, which proved to be more complicated than I initially thought.
I guess my preferred approach would now be to have as little complications in that area as possible (i.e. not serialize anything if not explicitly given all the data) and maybe deliver the missing parts in a later release.
Of course only speaking to myself in that thread and not getting any feedback (or even help) from anyone else isn't exactly motivating.
Other than that, my main motivation for working on this was to get all my private meetings and events properly managed, but that issue was pretty much solved in another, less enjoyable way a little over a year ago, while work also got a lot more stressfull.
Maybe once life comes back to normal in a hopefully not too distant future, I (and hopefully also others - maybe you?) will again have more time and motivation to work on this.

[make-github-pseudonymous-again]

@N-Coder Thanks for the effort. I am looking forward to this release too. I must admit I have not put any effort in it. Barely tried. I hope at some point I can help you finish the job. Just @ me in the near or distant future.

[N-Coder]

Wow, it looks like there are already some people using this code (and it's probably more than a year since I promised an alpha release "soonish"), so here's a small status update. Most of the big points are by now done, but there are still two points I'd like to complete before doing an alpha release, which are:

• varying levels of strictness, as pretty much everything is currently more geared towards validation than lenient parsing of files from the wild. This will probably need some way of converting previously fatal errors into ignorable warnings at the right points, but I'd also like something like a hook-mechanism, so that you can fill in a sensible alternative or fix things yourself at the exact point where something fails to parse.
• parsing / serialization hooks and customizations, as im not 100% happy with how the Converter and ComponentMeta system currently works. It's highly flexible and most things get derived automatically from the attr.s fields, but it's also somewhat hard to see how the different parts achieving that interact with each other and how customizations should be made. So maybe there are still some changes required there to make it also easily understandable and customizeable in addition to being highly automated.

After that is done, I guess we'd be good to go for a v0.8.alpha release. 🎉
I'd then tackle the documentation and testing issues on the way from alpha to beta, also using anyone who is daring enough to use the bleeding-edge version as tester. 😁

[make-github-pseudonymous-again]

Hi @N-Coder. Thanks for all the hard work!

* I'd also like something like a hook-mechanism, so that you can fill in a sensible alternative or fix things yourself at the exact point where something fails to parse

* parsing / serialization hooks and customizations

Have you considered having a generator (asynchronous?) at the lowest level. For parsing for example you could yield each event and let the consumer decide what to do with errors for instance. Maybe this is not the best abstraction. Just leaving it there in case you have not thought about it.

PS: Also have not looked at the codebase in a while so no idea of how it works.

[S1SYPHOS]

I just started using your library and it works like a charm - thanks so much! I'm generating ics files from parsing a PDF, containing the weekly assignments of all public prosecutors in our district - since the software generating the PDFs won't do it 🤦

Cheers!

[stefanv]

@N-Coder We would love to see a release for this on PyPi (an alpha or beta release would do), since one of our packages depends on it. Is there anything we can do to help, any specific issues you'd like someone to work on?

[N-Coder]

The blocker is still finding a better architecture for easily handling slightly weird inputs and customizing th (de-)serialization mentioned here, now also with the added complication of supporting different output modes mentioned here. The current approach with using singleton Converters and an immutable lists of conversion functions is pretty unwieldy and hard to extend. Using (possibly multiple) Converter instances (with a global default one) could make providing different variants and storing temporary values in a multi-threaded environment a lot easier, also allowing us to drop the intransparent state dict that is currently thrown around everywhere. Autogenerating one method of the respective converter class/instance for each field instead of using a list of methods would probably make extension via monkey-patching or inheritance easier. So I have a rough idea there, but I'd still need to try out a few things to fix the details (e.g. how we could lazily create / initialize this hierachy/-ies of converters) and find the time to actually do the refactoring. If you want to take a deeper look on the current implementation, feedback on how things could be improved would be highly welcome, anyways!

Regarding the handling of broken inputs, I tried an approach similar to the suggestion from here. Parsing is done in a line-wise fashion, so it's relatively easy to add error-handlers when some individual line fails to parse. The problem is with creating compound objects consisting out of multiple parsed lines and especially during their finalization / validation, e.g. what to do if you realize that an object is missing a required property or has contradicting properties (e.g. an event ending before it starts). Handling each of these kinds of problems individually is tedious, but turning off validation entirely is also problematic as that could lead to issues later on (e.g. with repeating such a broken event).

[N-Coder]

Oh, and regarding an alpha release, this should be possible with two gotchas:

• We are breaking the serialization from v0.7 in a very bad way, i.e. previously serialziation was done via str(calendar), but that behaviour proved to be problematic (see [WIP] new implementation of parsing and serialization #240). Now serialization should be done via an explicit method call, while __str__ only returns a short description, so anybody simply (accidentally) updating the package without changing their code will loose data without their code actually producing exceptions. Even though we are still pre-1.0 software and alpha releases are afaik not automatically updated to, we should have some more graceful way of handling this, e.g. having __str__ always throw an exception in v0.8 unless you set some __future__ flag that will only become default in the next release thereafter.
• We are still in the middle of a very big refactoring, big parts of the test suite haven't been ported yet, there are big future changes coming to the Converter system, and the library is currently still very picking about slightly weird inputs, so we should make sure nobody thinks this alpha release is any more stable than checking out the GitHub master branch.

Maybe @C4ptainCrunch wants to test the release once we fixed those two points?

[stefanv]

• We are breaking the serialization from v0.7 in a very bad way, i.e. previously serialziation was done via str(calendar), but that behaviour proved to be problematic (see [WIP] new implementation of parsing and serialization #240). Now serialization should be done via an explicit method call, while __str__ only returns a short description, so anybody simply (accidentally) updating the package without changing their code will loose data without their code actually producing exceptions. Even though we are still pre-1.0 software and alpha releases are afaik not automatically updated to, we should have some more graceful way of handling this, e.g. having __str__ always throw an exception in v0.8 unless you set some __future__ flag that will only become default in the next release thereafter.

I would not have __str__ throw an exception; you're likely to get in trouble with interactive interpreters etc. Why don't you have __str__ call the new serializer for at least one release while raising a FutureWarning or similar? Then, at least, people will have a chance to fix their code.

• We are still in the middle of a very big refactoring, big parts of the test suite haven't been ported yet, there are big future changes coming to the Converter system, and the library is currently still very picking about slightly weird inputs, so we should make sure nobody thinks this alpha release is any more stable than checking out the GitHub master branch.

In that case, perhaps best to hold off on such an alpha release until you can complete more of the work. I'm just worried that the work seems quite daunting, and that it may take a long time to resolve. For now, perhaps consider pairing down the list of issues that absolutely must be resolved before release (or making issues where none exist), then we can start knocking them out one by one.

[allenporter]

I'd like to work on helping accelerating 0.8 release. There are a few items around testing which seem reasonable for a newcomer to take on?

I saw there were some attempts at building up an .ics repo before and having testing based on that, and a few specific regressions listed above but they are from 2020. Is adding testing still a relevant priority?

[N-Coder]

Testing is definitely a good thing to work on! When starting my big refactoring, I basically deleted the old test suite, so moving over the relevant parts and fixing things where needed would be a good thing. In addition, writing more new tests (maybe targeting specific functionally) would also be very useful to further increase the coverage. Further enlarging the test corpus in #236 would also be very good to test our library with the weird ics files from the wild. Currently, there's probably gonna be a lot of errors with these as I'm still working on making the parser more forgiving, but still building up the corpus will be very helpful later-on. I guess these three points from my list at the top are the best candidates for contributions.

...nonetheless here's some comments on the remaining "testing" ToDos. The make-all-day issues are probably resolved by now, but that still needs some thorough testing (or at least collection of the different tests we already had around). Testing the interoperability with other tools is probably best done shortly before we make the 0.8 release. I already included the regression tests for the linked issues and I guess it's not too much use going through even older issues to add more of these tests (instead of directly testing functionality), so that point is probably more for future issues that should be converted to regression tests whenever possible.

[N-Coder]

Note: The converter system changes are on their way and I updated to ToDo list at the top to reflect the current status and what really needs to be done. The only thing we might wanna discuss is how much testing we want / need for an actual v0.8 release (see the second block of ToDos). We could be strict and only make an alpha release without that, or we can "involve" our downstream users a little more and do the full release sooner, maybe incorporating reported fixes and further tests (and better docs) in a v0.8.1 release afterwards.

[jarrodmillman]

A release soon would be much appreciated even if it is just an alpha, beta, or rc prerelease! Just having a new package on PyPI in any form would be wonderful. (I also think it is reasonable to just release 0.8 soon with a 0.8.1 afterwards.) Thanks for all your work on this!

FYI: I am trying to make a release of yaml2ics, which currently depends on ics-py/ics-py main:
https://github.com/scientific-python/yaml2ics/blob/42638740cf3d1c6d52222e9934313579411ebecf/pyproject.toml#L16

But I am not able to upload to PyPI, since it doesn't allow direct dependencies.

[C4ptainCrunch]

Ok, let's do that. I'll try to release something like 0.8dev0, it will be a good way to test the release GitHub action, should not impact end users that don't use pre-releases and can help early adopters :)

Edit: @jarrodmillman : releasing 0.8.0dev0 on PyPI is currently blocked by #274 (comment) as soon as i can fix it it will be released !

[C4ptainCrunch]

@jarrodmillman 0.8.0.dev0 is available on PyPI 🎉

[jarrodmillman]

I will test it later today! Thanks!!

[jarrodmillman]

It worked great! Thanks!!

[JulienPalard]

I tested it too, it also works great here! Thanks for all the work! ♥

[S1SYPHOS]

Same here ;)

[honzajavorek]

I use 0.8.0.dev0 for months and it seems to work just fine :) At least for what I need. I value that this new version doesn't depend on ancient arrow anymore. Thanks!

[stefanv]

Perhaps it's time to call it for 0.8? The dev0 release quite well, and I'm worried about the release process stalling.

[N-Coder]

Perhaps it's time to call it for 0.8? The dev0 release quite well, and I'm worried about the release process stalling.

Good point. Doesn't seem like I will find the time to fix the remaining points any time soon. And if things still seem to work without the testsuite or updated documentation, maybe an incomplete release is better than no release.

[C4ptainCrunch]

And if things still seem to work without the testsuite or updated documentation, maybe an incomplete release is better than no release.

And we can still release 0.8.1, .2, .3 if we make incremental improvements to the doc or tests :)

Should i start the release process then ?

Side quest: what do we do with this ticket and the github milestones ?

[N-Coder]

Should i start the release process then ?

👍🏼 from my side

Side quest: what do we do with this ticket and the github milestones ?

We can move (i.e. rename/reassign) the milestone on to Version 0.9 or so (as these are mostly bigger points and the 0.9 is currently empty it seems). Once the release is through, I'll close this issue and create a new issue with all the open points for Version 0.9 so that we have a clean overview there.

[N-Coder]

One thing that we should be aware of before releasing are the changed behaviours for __str__ and __iter__ mentioned in #318, which we only announced to change in v0.9. As the code currently is, we are already changing to the new (i.e. standard-compliant) behaviour of __str__ with this version, but that might lead to data losses (which is why we introduced a warning in v0.7.1 more than a year ago). The warning for Calendar.__iter__ from #318 also now seems to be missing in 0.8 while it is present in 0.7.

[jarrodmillman]

@C4ptainCrunch It would be great if you could release main as 0.8.0! I am happy to test a release candidate. If there is anything you need help with, please let me know. Thanks for all your work on the project!

[C4ptainCrunch]

@N-Coder : unfortunately, i don't really have time to review the depreciation warnings, but i might find some time to cut a release though. Do you think it's OK to release it "as is" (as opposed to not releasing anything :D) ?

Edit: in //, i'll work on #395 because as our release cadence is really low and Python 3.12 is going to be out soon, it would be a shame to not support it in 0.8

[jarrodmillman]

@C4ptainCrunch @N-Coder I made a PR #401 that is a one-line fix for #395. Once #401 is merged, I think #395 will be ready to merge.

It would be great to get 0.8 released as-is after those two PRs are merged.

[C4ptainCrunch]

Thank you @jarrodmillman for the PR !
I tried to write some release notes, here is what i have at the moment.

I still have to improve at least those 2 lines:

• Support for parsing and serializing timezones
• Calendar constructor / parse methods

But i have trouble remembering what those are exactly 🙈.

As well as detailing the bugfixes

• Fix all-day issues
• Fix timezone issues
• Fix SEQUENCE bug

If someone has inspiration, feel free to post here or op a PR.

After that, i guess we can do a quick check of the documentation and then start the release process !

Edit: the notes are also missing those:

• faster ContentLine parser implementation (PR Contentline parser #247, V0.7 is much slower to parse large calendar feed then V0.5 #244)
• Basing event equality on uid #313

[stefanv]

@C4ptainCrunch @N-Coder Please let us know what you need in order to get a release out.

[C4ptainCrunch]

@stefanv The main blocking point i see is the release notes i mentioned above (help welcome !). After that i guess we could scream YOLO and publish the release, it has waited long enough.