Import from Khal calendar #27

diego-rapoport · 2022-08-03T14:17:59Z

Is it possible make an import from Khal Calendar events as well? If there's anyway I can help with that, I would be glad.

anufrievroman · 2022-08-04T00:49:27Z

Remind me, how khal stores the events? Is it .ics or some native format as well? If it is .ics, then I am already working in this direction, if native file format, could you drop an example file with a few events?

diego-rapoport · 2022-08-04T11:46:06Z

As I can see it's just .ics files.

anufrievroman · 2022-08-04T13:26:59Z

In this case, I think the import should become possible in the next version. Currently, I am working on the support for the .ics files, and users will be able to see events from external ics files or import events from ics files into native format.

By the way, when you say import, do you mean like reading events from khal's file once and appending them to your calcure events, or more like continuous syncing with khal's file?

diego-rapoport · 2022-08-04T17:04:42Z

I'm still experimenting calcure so I don't know if I'll make a total migration right now. In this case I would be doing a continuous syncing, even if that means run some code from caculre each time I need it to update.
But khal is integrated with vdirsyncer, which continuous syncs my gmail's agenda. So a full migration to calcure would require something like that in order for me to use in my daily basis. Don't know if something like that is on your plan for calcure.

ceuk · 2023-01-02T01:12:59Z

Just an FYI guys - Khal uses an SQLite db to store its entries. It's usually stored somewhere like $XDG_DATA_HOME/khal/khal.db

It's common to use Khal in conjunction with something like vdirsyncer where you would configure Khal to parse ics files in a local directory, which I think is where OP might be getting confused.

Here's a sample query to get all events

SELECT item, recs_loc.href, dtstart, dtend, ref, etag, dtype, events.calendar 
FROM recs_loc
JOIN events
ON recs_loc.href = events.href
AND recs_loc.calendar = events.calendar
ORDER BY dtstart;

HTH

Tonus1 · 2023-03-01T23:02:55Z

Hello
Khal user here.
I think what the OP wants is to have calcure read from multiple ics files.

vdirsyncer gets individual files from remote repo
khal parse all the ics fileS
It would be great if calcure could use those or a mirror of those.

Pro : vdirsyncer runs once, it is faster than with single ics file
Con : calcure works with single file, might be necessary to mirror/copy back and forth the files before processing with either script

anufrievroman · 2023-03-01T23:35:34Z

Hello @Tonus1
Yes, that basically is working. For the design reasons, calcure wants the single file (because we also accept URLs and multiple files from multiple places), but one can just set up another record in vdirsyncer config to have both filesystem and singlefile. It's a bit of a duplication, but it's the easiest solution.

@diego-rapoport does that work for you? I mean I am afraid I won't write a real parcer from khal's sql database, so sync can be achieved via .ics files only. In the future, I am thinking to make an importer that turns items in .ics files into native calcure ones.

Tonus1 · 2023-03-02T02:03:10Z

Thank you for your quick reply Roman. I just read some issues that explain very well the state of developpement. A bit sorry to have written too fast ! I've got about a dozen calendars and it's not a trivial job to get another synchro up. And it uses a bit more bandwith I would like. Anyway, I will stick around here, the UI worths it ! I submited the build script for Slackware-15.0 at slackbuilds.org and I think it will be avaible next week. Hoping more folks will get involved and give some help. Regards, Tonus

anufrievroman · 2023-03-02T02:37:54Z

Actually, now as I think about it, it shouldn't be too difficult to support folders instead of single files. I'll just check if it's not ending with .csv (like we check for http now) and then assume it's a folder and parse everything from it. I guess it would simplify life for many people.

anufrievroman · 2023-03-02T08:09:45Z

So, in bd8e9ae I implemented reading from ics folders. Now you can put either a folder or file or url in your config. Just pushed version 2.8 to Pypi please pip install --upgrade calcure and test!
Especially, if someone could confirm if it works on Windows, it would be great.

Tonus1 · 2023-03-02T13:09:21Z

There's indeed something better now !
I achieved to get 3 folders to be parsed.
Got some errors about ValueError: Begin must be before end, crashing on 0 length events.

Traceback (most recent call last):
  File "/usr/bin/calcure", line 33, in <module>
    sys.exit(load_entry_point('calcure==2.8.0', 'console_scripts', 'calcure')())
  File "/usr/lib64/python3.9/site-packages/calcure/__main__.py", line 1040, in cli
    curses.wrapper(main)
  File "/usr/lib64/python3.9/curses/__init__.py", line 94, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/lib64/python3.9/site-packages/calcure/__main__.py", line 928, in main
    user_ics_events = event_loader_ics.load()
  File "/usr/lib64/python3.9/site-packages/calcure/loaders.py", line 368, in load
    cal = ics.Calendar(ics_file)
  File "/usr/lib64/python3.9/site-packages/ics/icalendar.py", line 69, in __init__
    self._populate(containers[0])  # Use first calendar
  File "/usr/lib64/python3.9/site-packages/ics/component.py", line 59, in _populate
    parser(self, lines)  # Send a list or empty list
  File "/usr/lib64/python3.9/site-packages/ics/parsers/icalendar_parser.py", line 71, in parse_vevent
    calendar.events = set(map(event_factory, lines))
  File "/usr/lib64/python3.9/site-packages/ics/parsers/icalendar_parser.py", line 69, in event_factory
    return Event._from_container(x, tz=calendar._timezones)
  File "/usr/lib64/python3.9/site-packages/ics/component.py", line 31, in _from_container
    k._populate(container)
  File "/usr/lib64/python3.9/site-packages/ics/component.py", line 62, in _populate
    parser(self, lines[0])  # Send the element
  File "/usr/lib64/python3.9/site-packages/ics/parsers/event_parser.py", line 26, in parse_dtstart
    event.begin = iso_to_arrow(line, tz_dict)
  File "/usr/lib64/python3.9/site-packages/ics/event.py", line 157, in begin
    raise ValueError('Begin must be before end')
ValueError: Begin must be before end

Got rid of most of this (my script still fails to find some in one of the huge folder).

Strange thing, another crash occurs even after I removed/substitute the offending file...

Traceback (most recent call last):
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 613, in _call
    result = self._recursive_call(ruleinfo)
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 641, in _recursive_call
    return self._invoke_rule(ruleinfo, self.memokey)
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 690, in _invoke_rule
    ruleinfo.impl(self)
  File "/usr/lib64/python3.9/site-packages/ics/grammar/contentline.py", line 162, in _ALPHADIGIT_MINUS_PLUS_
    self._pattern('[a-zA-Z0-9-]+')
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 749, in _pattern
    self._error(pattern, exclass=FailedPattern)
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 549, in _error
    raise self._make_exception(item, exclass=exclass)
tatsu.exceptions.FailedPattern: (1:1) expecting /[a-zA-Z0-9-]+/ :
./perso/dca88360-b1c8-407f-bbfd-b5312ac694e4.icsBEGIN:VCALENDAR VERSION:2.0
^
ALPHADIGIT_MINUS_PLUS
contentline
start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/calcure", line 33, in <module>
    sys.exit(load_entry_point('calcure==2.8.0', 'console_scripts', 'calcure')())
  File "/usr/lib64/python3.9/site-packages/calcure/__main__.py", line 1040, in cli
    curses.wrapper(main)
  File "/usr/lib64/python3.9/curses/__init__.py", line 94, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/lib64/python3.9/site-packages/calcure/__main__.py", line 928, in main
    user_ics_events = event_loader_ics.load()
  File "/usr/lib64/python3.9/site-packages/calcure/loaders.py", line 368, in load
    cal = ics.Calendar(ics_file)
  File "/usr/lib64/python3.9/site-packages/ics/icalendar.py", line 64, in __init__
    containers = calendar_string_to_containers(imports)
  File "/usr/lib64/python3.9/site-packages/ics/grammar/parse.py", line 227, in calendar_string_to_containers
    return string_to_container(string)
  File "/usr/lib64/python3.9/site-packages/ics/grammar/parse.py", line 204, in string_to_container
    return lines_to_container(txt.split("\n"), linewise)  # linewise
  File "/usr/lib64/python3.9/site-packages/ics/grammar/parse.py", line 197, in lines_to_container
    return parse(tokenize_line(unfold_lines(lines, with_linenr=True)))  # linewise
  File "/usr/lib64/python3.9/site-packages/ics/grammar/parse.py", line 187, in parse
    for line in tokenized_lines:
  File "/usr/lib64/python3.9/site-packages/ics/grammar/parse.py", line 180, in tokenize_line
    yield ContentLine.parse(line[1], line[0])
  File "/usr/lib64/python3.9/site-packages/ics/grammar/parse.py", line 82, in parse
    ast = GRAMMAR.parse(line)
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 239, in parse
    raise self._furthest_exception
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 232, in parse
    result = rule()
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 73, in wrapper
    return self._call(ruleinfo)
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 613, in _call
    result = self._recursive_call(ruleinfo)
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 641, in _recursive_call
    return self._invoke_rule(ruleinfo, self.memokey)
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 690, in _invoke_rule
    ruleinfo.impl(self)
  File "/usr/lib64/python3.9/site-packages/ics/grammar/contentline.py", line 85, in _start_
    self._contentline_()
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 73, in wrapper
    return self._call(ruleinfo)
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 613, in _call
    result = self._recursive_call(ruleinfo)
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 641, in _recursive_call
    return self._invoke_rule(ruleinfo, self.memokey)
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 690, in _invoke_rule
    ruleinfo.impl(self)
  File "/usr/lib64/python3.9/site-packages/ics/grammar/contentline.py", line 104, in _contentline_
    self._ALPHADIGIT_MINUS_PLUS_()
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 73, in wrapper
    return self._call(ruleinfo)
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 624, in _call
    self._error('Expecting <%s>' % ruleinfo.name)
  File "/usr/lib64/python3.9/site-packages/tatsu/contexts.py", line 549, in _error
    raise self._make_exception(item, exclass=exclass)
tatsu.exceptions.FailedParse: (1:1) Expecting <ALPHADIGIT_MINUS_PLUS> :
./perso/dca88360-b1c8-407f-bbfd-b5312ac694e4.icsBEGIN:VCALENDAR VERSION:2.0
^
ALPHADIGIT_MINUS_PLUS
contentline
start

File is

BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//ddaysoftware.com//NONSGML DDay.iCal 1.0//EN
BEGIN:VTIMEZONE
TZID:Romance Standard Time
BEGIN:STANDARD
DTSTART:19701025T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYHOUR=3;BYMINUTE=0;BYMONTH=10
TZNAME:Paris\, Madrid
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:19700329T020000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYHOUR=2;BYMINUTE=0;BYMONTH=3
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
CLASS:PUBLIC
DTEND;VALUE=DATE:20220828
DTSTAMP:20221012T062556Z
DTSTART;VALUE=DATE:20220827
PRIORITY:5
SEQUENCE:0
SUMMARY:Déménagement
TRANSP:TRANSPARENT
UID:dca88360-b1c8-407f-bbfd-b5312ac694e4
X-MICROSOFT-CDO-BUSYSTATUS:FREE
END:VEVENT
END:VCALENDAR

anufrievroman · 2023-03-02T22:36:26Z

Interesting, could you also check info.log file (in your config folder)? It should specify what has gone wrong. I feel like it might be a specific .ics file, and errors are coming from py-ics library. To be honest I doubt I can do much about it, except excepting these errors and just skipping bad files, but I'll take a look.
Overall, just by looking at the errors and the file, I don't really see what''s wrong.

anufrievroman · 2023-03-02T23:11:33Z

So, in version 2.8.1 I've added #710ba1fd5cbf46bb8ebe47e84b2966480885fbaa a general exception for now, so that the program does not crash. Now it should read all files except problematic ones, and in your info.log file you should be able to see which once caused the problem.

If we knew which line in the file causes the problem, I could still skip this line before trying to parse the file with py-ics (like I do with other problematic lines), but like I said, I don't really see what's wrong here. Anyone else?

Tonus1 · 2023-03-03T01:29:27Z

Odd : got the same error even if I got rid of the problematic file. And still get the ValueError: Begin must be before end with the other folder.
I will investigate that further, I might not be so accurate when it's late in the night...
Btw, you did not change the naming of the folders in your last release build ;)

anufrievroman · 2023-03-03T01:32:04Z

Btw, you did not change the naming of the folders in your last release build ;)

Sorry, more specifically?

Tonus1 · 2023-03-03T01:35:29Z

When I unpack the tarball, I had to modify my install script because the folder indicates 2.8 and not 2.8.1
No big deal, just wanted to warn you if it wasn't intended

anufrievroman · 2023-03-03T01:59:21Z

Ah, sorry, I mean I pushed the update 2.8.1 to pypi but here on GitHub I tend to publish releases only for major changes. So, the new version can be installed only from pip.

Tonus1 · 2023-03-04T02:05:23Z

Hi
I istalled from PyPi and it's far better.
I noticed some glitches :

a faulty folder stops ics files discovery (for the folders from same parent directory ?)
missing a per file warning
does not find the 'holiday' module even if found at build time
I have a folder that generates duplicate entries (white ones, seems last ones ? are those colors documented ?)

anufrievroman · 2023-03-04T09:00:17Z

Hmm, that's strange. I'd say:

for the holiday module, no clue, I didn't change anything about it. I'll try to reproduce.
for the folders, so you have folders inside the folder that you specified in the config? Actually, even in this case it should work. I'll try to create my own ics folder to see...
It should give per file warnings in log file. But I didn't test, so there might be some issues...
I am not sure if I understand about generation of duplicate items, but the colors for ics calendars (paths in config) work like that. You specify:

color_ics_calendars = 2,3,5,4,7,5

and so on, up to 10 colors. If you specified only a few (3 or 4 by default), only the first few calendars have different colors and the remaining assume the value of the last one. Is that what you mean?

EDIT: Yes, the whole parsing breaks if one fie is broken. I fixed that now. I'll push the new version a bit later.
EDIT: Done, please try version 2.8.2

Tonus1 · 2023-03-04T13:41:10Z

Hi
2.8.2 installed
I'm ok up to 9 folders, but I only have green, yellow, red and white despite the color_ics_calendars = 2,1,3,3,9,6,5,5,2
I will now try with the 2 faulty (and huge) folders.
Next I will try to find out the holidays bug. Btw is there something to configure outside the config.ini for that ?

Thanks a lot for your help

Tonus1 · 2023-03-04T14:25:16Z

First find, the events on multiple days show as single day. This one in only visible on the first day :

BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//ddaysoftware.com//NONSGML DDay.iCal 1.0//EN
BEGIN:VTIMEZONE
TZID:Romance Standard Time
BEGIN:STANDARD
DTSTART:19701025T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYHOUR=3;BYMINUTE=0;BYMONTH=10
TZNAME:Paris\, Madrid
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:19700329T020000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYHOUR=2;BYMINUTE=0;BYMONTH=3
TZNAME:Paris\, Madrid (heure d’été)
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
CLASS:PUBLIC
DTEND;VALUE=DATE:20230522
DTSTAMP:20221012T062558Z
DTSTART;VALUE=DATE:20230518
PRIORITY:5
SEQUENCE:0
SUMMARY:Ascension
TRANSP:TRANSPARENT
UID:e53660ce-271e-446b-9aa0-684e5732aedc
X-MICROSOFT-CDO-BUSYSTATUS:FREE
BEGIN:VALARM
ACTION:DISPLAY
DESCRIPTION:This is an event reminder
TRIGGER:-PT2H
END:VALARM
END:VEVENT
END:VCALENDAR

I continue to search.

Tonus1 · 2023-03-04T16:55:13Z

Found that I was missing a few deps for holidays : I'll soon update the build scripts on slackbuilds.org.

anufrievroman · 2023-03-04T23:22:31Z

Hi 2.8.2 installed I'm ok up to 9 folders, but I only have green, yellow, red and white despite the color_ics_calendars = 2,1,3,3,9,6,5,5,2 I will now try with the 2 faulty (and huge) folders.

Also, note that this setting must be inside the Colors block of config.ini

Next I will try to find out the holidays bug. Btw is there something to configure outside the config.ini for that ?

No, only config.ini. But just to make sure that it's not some parameter causing trouble, try removing config.ini and starting the calcure. By default it should show US holidays. If still no luck, check in which "environments" you are. Sometimes people install general python libs into one environment but then run calcure from another. In general, I advise uninstalling everything related from pip, or AUR, or any other method, and the clean install:

pip install calcure
By the way, does info.log mentioning something about holidays?

anufrievroman · 2023-03-04T23:24:39Z

First find, the events on multiple days show as single day. This one in only visible on the first day :

Unfortunately, recurring events are not supported yet, because last time I checked py-ics does not support it yet

Tonus1 · 2023-03-05T02:30:17Z

Hi 2.8.2 installed I'm ok up to 9 folders, but I only have green, yellow, red and white despite the color_ics_calendars = 2,1,3,3,9,6,5,5,2 I will now try with the 2 faulty (and huge) folders.

Also, note that this setting must be inside the Colors block of config.ini

That was the culprit ! I was replacing all reccuring event and had archived everything too old when I saw your message.
Surprise, the parsing of my 2 directories was ok as soon as I had change the place of the color_ics_calendars =

Everything is ok now ! thank you very much !

Will you release the 2.8.2 on github or shall I stick packaging the Pypi version ?

Best regards

anufrievroman · 2023-03-05T03:16:21Z

Everything is ok now ! thank you very much !

Will you release the 2.8.2 on github or shall I stick packaging the Pypi version ?

Great! Okay, I'll release on GitHub but later today. Right now I am literally on snowboard :D

Tonus1 · 2023-03-05T11:23:12Z

@diego-rapoport Since we now can use same folder as khal I believe this can be closed now :)

anufrievroman · 2023-03-05T12:51:12Z

Okay, so I released 2.8.2, and this perhaps indeed concludes discussion reading Khal.
If you have any other questions or problems, feel free to open a new issue or continue the #4 thread.
Thank you everyone for your discussion and contribution!

anufrievroman added the Feature request New feature or request label Aug 4, 2022

anufrievroman changed the title ~~Feature Request: Khal Calendar~~ Import from Khal calendar Aug 4, 2022

anufrievroman closed this as completed Mar 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Import from Khal calendar #27

Import from Khal calendar #27

diego-rapoport commented Aug 3, 2022

anufrievroman commented Aug 4, 2022

diego-rapoport commented Aug 4, 2022

anufrievroman commented Aug 4, 2022

diego-rapoport commented Aug 4, 2022

ceuk commented Jan 2, 2023

Tonus1 commented Mar 1, 2023

anufrievroman commented Mar 1, 2023

Tonus1 commented Mar 2, 2023 via email

anufrievroman commented Mar 2, 2023 •

edited

Loading

anufrievroman commented Mar 2, 2023

Tonus1 commented Mar 2, 2023

anufrievroman commented Mar 2, 2023 •

edited

Loading

anufrievroman commented Mar 2, 2023 •

edited

Loading

Tonus1 commented Mar 3, 2023

anufrievroman commented Mar 3, 2023

Tonus1 commented Mar 3, 2023

anufrievroman commented Mar 3, 2023

Tonus1 commented Mar 4, 2023

anufrievroman commented Mar 4, 2023 •

edited

Loading

Tonus1 commented Mar 4, 2023

Tonus1 commented Mar 4, 2023

Tonus1 commented Mar 4, 2023

anufrievroman commented Mar 4, 2023

anufrievroman commented Mar 4, 2023

Tonus1 commented Mar 5, 2023

anufrievroman commented Mar 5, 2023

Tonus1 commented Mar 5, 2023

anufrievroman commented Mar 5, 2023

Import from Khal calendar #27

Import from Khal calendar #27

Comments

diego-rapoport commented Aug 3, 2022

anufrievroman commented Aug 4, 2022

diego-rapoport commented Aug 4, 2022

anufrievroman commented Aug 4, 2022

diego-rapoport commented Aug 4, 2022

ceuk commented Jan 2, 2023

Tonus1 commented Mar 1, 2023

anufrievroman commented Mar 1, 2023

Tonus1 commented Mar 2, 2023 via email

anufrievroman commented Mar 2, 2023 • edited Loading

anufrievroman commented Mar 2, 2023

Tonus1 commented Mar 2, 2023

anufrievroman commented Mar 2, 2023 • edited Loading

anufrievroman commented Mar 2, 2023 • edited Loading

Tonus1 commented Mar 3, 2023

anufrievroman commented Mar 3, 2023

Tonus1 commented Mar 3, 2023

anufrievroman commented Mar 3, 2023

Tonus1 commented Mar 4, 2023

anufrievroman commented Mar 4, 2023 • edited Loading

Tonus1 commented Mar 4, 2023

Tonus1 commented Mar 4, 2023

Tonus1 commented Mar 4, 2023

anufrievroman commented Mar 4, 2023

anufrievroman commented Mar 4, 2023

Tonus1 commented Mar 5, 2023

anufrievroman commented Mar 5, 2023

Tonus1 commented Mar 5, 2023

anufrievroman commented Mar 5, 2023

anufrievroman commented Mar 2, 2023 •

edited

Loading

anufrievroman commented Mar 2, 2023 •

edited

Loading

anufrievroman commented Mar 2, 2023 •

edited

Loading

anufrievroman commented Mar 4, 2023 •

edited

Loading