Sync problem?

[Sblop]

Hi, Thanks for the updated docker image. This is great!
However I have one small problem. It seels only discovery runs, eventhough I have set the flag to true.

Docker composer:

vdirsyncer:
 # image: pschlieker/vdirsyncer-docker
    image: bleala/vdirsyncer:latest
    container_name: vdirsyncer
    volumes:
      - "/share/Docker/vdirsyncer:/vdirsyncer" 
      - "/share/Docker/vdirsyncer/config:/vdirsyncer/config"
      
      - "/etc/localtime:/etc/localtime:ro"  
    environment: 
      - CRON_TIME=0 */1 * * *
      - AUTODISCOVER=true
      - AUTOSYNC=true
    restart: unless-stopped
    networks:
      docker_frontend:

In the log it confirms the correct piece of the script is running:

Starting Logging...

Autodiscover and Autosync are enabled


crond: crond (busybox 1.33.1) started, log level 8


crond: USER root pid  92 cmd run-parts /etc/periodic/15min


crond: USER root pid 200 cmd run-parts /etc/periodic/15min


crond: USER root pid 296 cmd run-parts /etc/periodic/15min


crond: USER root pid 402 cmd run-parts /etc/periodic/15min


crond: USER root pid 403 cmd run-parts /etc/periodic/hourly


crond: USER root pid 404 cmd yes | vdirsyncer discover && vdirsyncer metasync && vdirsyncer sync

It runs the discovery, however not the meta or the sync?

After the discovery has run (I will not post the log as this contains personal names etc.) this happens.

error: 1 out of 11 tasks failed.


crond: USER root pid 512 cmd run-parts /etc/periodic/15min


crond: USER root pid 616 cmd run-parts /etc/periodic/15min


crond: USER root pid 716 cmd run-parts /etc/periodic/15min


crond: USER root pid 820 cmd run-parts /etc/periodic/15min


crond: USER root pid 821 cmd run-parts /etc/periodic/hourly


crond: USER root pid 822 cmd yes | vdirsyncer discover && vdirsyncer metasync && vdirsyncer sync

Same problem over again...

The error is due to a sync problem with sogo, so I would not expect this to be a problem?

It doesnt seem like this is correct behaviour?

[Bleala]

Hello there!

As i wrote in the README.md, AUTODISCOVER is quite a sketchy feature.

I tried it (like 5 minutes ago) with AUTODISCOVER and AUTOSYNC enabled and everything worked fine. I'm using Apple, Google, Baikal, SOGo (Mailcow) and local storage. But normally i'm not using AUTODISCOVER, i run the command manually once, if i'm adding a new pair, because if you don't change your config, AUTODISCOVER is useless and is only producing overload.

This seems like a problem with Vdirsyncer itself, if the discovery fails and you get an error like in your logs error: 1 out of 11 tasks failed. So this is not a problem because of the container.
And if the discovery fails, the sync also won't happen, because discovery, metasync and sync are executed afterwards!

So there is most likely some kind of a problem with your configuration.

Could you share your configuration with me? Without passwords, usernames and maybe private domains, just delete them from the config.

And where does the problem occur? With a Carddav or Caldav pair/storage?

[Sblop]

Well, in that case, I will remove autodiscover. The error happens with sogo. It claims my work calendar is not existing. When I clear everything and resync, all is fine, then next time it happens again. I will post my config below. I have sh#t load of errors with google, but accepts these.
My setup is like this: Nextcloud is my main contact and calendar storeage which works okay. I sync this to local storeage so I can backup everything smoothly. (Cal + Card)dav. Additionally I have a number of ical calendars (kids school, work etc) i sync to my drive, oneway, so I can add to this.
From local sync contacts + cal to SOGO. Contacts works flawlessly, calendar seems buggy, but mostly works.
From local sync all calendars to google. I dont like to use google, but wife uses this, so she can see and change everything.

Below config is laidout like this:
Carddav
pairs
storeage
Caldav
pairs
storeage dynamic
storeage ical

[general]
# A folder where vdirsyncer can store some metadata about each pair.
status_path = "/vdirsyncer/status/"

# CARDDAV PAIRS##########################

[pair card_local_next]
a = "card_local"
b = "card_next"
collections = [["card_from_next", "contacts", "contacts"]]
metadata = ["displayname"]
conflict_resolution = "a wins"

[pair card_local_sogo]
a = "card_local"
b = "card_sogo"
#collections = ["from a", "from b"]
collections = [["sogo_card", "contacts", "personal"]]
#collections = "from a"
metadata = ["displayname"]
conflict_resolution = "a wins"

# CARDDAV STOREAGE##########################




[storage card_next]
type = "carddav"
#url = "https://next.<CENSORED>/remote.php/dav/carddav"
url = "https://next.<CENSORED>/remote.php/dav"
username = "<CENSORED>"
password = "<CENSORED>"


[storage card_local]
type = "filesystem"
path = "/vdirsyncer/data/contacts/"
fileext = ".vcf"


[storage card_sogo]
type = "carddav"
url = "https://mail.<CENSORED>/SOGo/dav/<CENSORED>/Contacts/personal"
username = "<CENSORED>"
password = "<CENSORED>"



# CALDAV PAIRS #####################################
[pair cal_next_local]
a = "cal_local"
b = "cal_next"
collections = ["personal", "Arbejde", "Frida", "Karla", "Bertil", "Holidays", "Thilda_arbejde", "Thilda"]
#collections = ["from a", "from b"]
metadata = ["displayname", "color"]
conflict_resolution = "a wins"

[pair cal_local_sogo]
a = "cal_local"
b = "cal_sogo"
collections = ["personal", "Arbejde", "Frida", "Karla", "Bertil", "Holidays", "Thilda_arbejde", "Thilda"]
metadata = ["displayname", "color"]
conflict_resolution = "a wins"

[pair cal_local_gcal]
a = "cal_local"
b = "cal_google"
collections = [["Personal", "personal", "<CENSORED>"], ["frida", "Frida", "<CENSORED>@group.calendar.google.com"], ["karla", "Karla", "<CENSORED>@group.calendar.google.com"], ["thilda", "Thilda", "<CENSORED>@gmail.com"], ["bertil", "Bertil", "<CENSORED>@group.calendar.google.com"]]
metadata = ["displayname", "color"]
conflict_resolution = "b wins"











# CALDAV PAIRS FROM READONLY#####################################

[pair cal_holidays_local]
a = "ical_holidays"
b = "cal_holidays"
collections = null
partial_sync = "ignore"
conflict_resolution = "b wins"

[pair cal_frida_local]
a = "ical_frida"
b = "cal_frida"
collections = null
partial_sync = "ignore"
conflict_resolution = "b wins"

[pair cal_karla_local]
a = "ical_karla"
b = "cal_karla"
collections = null
partial_sync = "ignore"
conflict_resolution = "b wins"

[pair cal_bertil_local]
a = "ical_bertil"
b = "cal_bertil"
collections = null
partial_sync = "ignore"
conflict_resolution = "b wins"

[pair cal_danfoss_local]
a = "ical_danfoss"
b = "cal_danfoss"
collections = null
partial_sync = "ignore"
conflict_resolution = "b wins"


[pair cal_thildaarb_local]
a = "ical_thildaarb"
b = "cal_thildaarb"
collections = null
partial_sync = "ignore"
conflict_resolution = "b wins"


# CALDAV STOREAGE #####################################

[storage cal_google]
type = "google_calendar"
token_file = "/vdirsyncer/google_token.txt"
client_id = "<CENSORED>"
client_secret = "<CENSORED>"
#start_date = null
#end_date = null
item_types = ["VEVENT"]
#item_types = []


[storage cal_next]
type = "caldav"
url = "https://next.<CENSORED>/remote.php/dav/caldav"
item_types = ["VEVENT"]
username = "<CENSORED>"
password = "<CENSORED>"

[storage cal_local]
type = "filesystem"
path = "/vdirsyncer/data/calendar/"
fileext = ".ics"
#encoding = "utf-8"
#post_hook = null

[storage cal_holidays]
type = "filesystem"
path = "/vdirsyncer/data/calendar/Holidays"
fileext = ".ics"

[storage cal_frida]
type = "filesystem"
path = "/vdirsyncer/data/calendar/Frida"
fileext = ".ics"

[storage cal_karla]
type = "filesystem"
path = "/vdirsyncer/data/calendar/Karla"
fileext = ".ics"

[storage cal_bertil]
type = "filesystem"
path = "/vdirsyncer/data/calendar/Bertil"
fileext = ".ics"

[storage cal_danfoss]
type = "filesystem"
#smider danfoss events ind i min arbejdskalender
path = "/vdirsyncer/data/calendar/Arbejde" 
fileext = ".ics"

#[storage cal_thilda]
#type = "filesystem"
#path = "/vdirsyncer/data/calendar/Thilda"
#fileext = ".ics"

[storage cal_thildaarb]
type = "filesystem"
path = "/vdirsyncer/data/calendar/Thilda_arbejde"
fileext = ".ics"

[storage cal_sogo]
type = "caldav"
item_types = ["VEVENT"]
url = "https://mail.<CENSORED>/SOGo/dav/<CENSORED>/Calendar/personal"
username = "<CENSORED>"
password = "<CENSORED>"

[storage ical_holidays]
type = "http"
read_only = true
url = "https://calendar.google.com/calendar/ical/da.danish%23holiday%40group.v.calendar.google.com/public/basic.ics"

[storage ical_frida]
type = "http"
read_only = true
url = "<CENSORED>"

[storage ical_karla]
type = "http"
read_only = true
url = "<CENSORED>"

[storage ical_bertil]
type = "http"
read_only = true
url = "<CENSORED>"


[storage ical_danfoss]
type = "http"
read_only = true
url = "<CENSORED>"

[storage ical_thildaarb]
type = "http"
read_only = true
url = "<CENSORED>"

[Bleala]

I'm maybe having an idea, why SOGo is not working as expected:

You are only having one caldav storage for sogo:

[storage cal_sogo]
type = "caldav"
item_types = ["VEVENT"]
url = "https://mail.<CENSORED>/SOGo/dav/<CENSORED>/Calendar/personal"
username = "<CENSORED>"
password = "<CENSORED>"

This will only sync your personal calendar in sogo.
If you have for example a second calendar, then you also need to add a second storage for that calender, because it has a different url!

This would be my SOGo caldav storages:

[storage mailcow_caldav_personal]
type = "caldav"
url = "https://<url>/SOGo/dav/<account>/Calendar/personal/"
username = "<login>"
password = "<password>"

[storage mailcow_caldav_family]
type = "caldav"
url = "https://<url>/SOGo/dav/<account>/Calendar/45-61237100-1B7-56230300/"
username = "<login>"
password = "<password>"

As you can see the family calendar has a differnt url, you cannot sync everything in your personal calender here.

That's why i recommend this way of collection configuration (as an example how mailcow and baikal sync these two calendars):

[pair mailcow_baikal_caldav_personal]
a = "mailcow_caldav_personal"
b = "baikal_caldav"
collections = [["personal", "personal", "default"]]
conflict_resolution = "b wins"
metadata = ["color", "displayname"]

[pair mailcow_baikal_caldav_family]
a = "mailcow_caldav_family"
b = "baikal_caldav"
collections = [["family", "45-61237100-1B7-56230300", "family"]]
conflict_resolution = "b wins"
metadata = ["color", "displayname"]

You see the mailcow/SOGo storage changed, also the collection. For Baikal only the collection changed, storage (and url) are the same (explanation at the end).

So for example to sync your calender Arbejde:
Create a calender in SOGo called like that, then click on the 3 dots besides the calendar name and click on Copy link to this calendar. You will see you will also get a random number, like i got for my family calendar.

Next create a new (second) caldav storage with different name, where the url is your new one with the random numbers. And then add a second pair for SOGo like you can see from my configuration.
Using this way collections = [["family", "45-61237100-1B7-56230300", "family"]] for collections is the best way in my opinion, this works perfectly for me. So do not sync all at once from two different storage, create a pair and storage for each calendar separately.

I hope this is not too confusing for you and that you understand what i am trying to say.

The only storages where you can use a single url for multiple calendars are:

• Apple
• Google
• Baikal
• Local

This is my experience so far.

And because of your google problem, i would also know a solution, tried for a few hours to find a working solution, but first of all we should solve your SOGo problem :)

PS: I recommend a Card-/Caldav Server like Baikal, Radicale etc. i think this makes it easier in general to sync through multiple companies/services.

[Sblop]

Thanks for your long answer. I will look into changing the structure of my sogo instance.

Back to my original question that is regarding sync, that is failing. You run the following command in your script:
vdirsyncer discover && vdirsyncer metasync && vdirsyncer sync
Why would this care, if anything is failing? Wouldn’t it just automatically go to next item, +metasync, after the discovery, and sync after this? Or dont I understand the logic correctly? As per my understanding the "&&" doesnt care?

Regarding your "PS". What would make sync different, using eg baikal, compared to local storeage?

[Bleala]

Something like this vdirsyncer discover && vdirsyncer metasync && vdirsyncer sync is a single command, containung multiple arguments/commands. So if an error is thrown, the command stops and the rest does not get executed.

If you would run everything separatly it would not be a problem.

Maybe i will change that in the future, that the cronjob will execute a scirpt containing multiple commands, but think syncing is not the best options if there are failures in the config anyway.

Regarding Baikal:
It would not make it different, but in my opinion have a specific dav server as master is a better option as syncing away from a local storage.
Maybe it does not matter, but there is a reason why programs like Baikal, Radical etc exist. And with Baikal you can also use a Mysql/MariaDB database (using in docker container), so if there is ever a big issue, i am able to use a normal database too.

Should i give you an example on how to solve your google problem too?

[Sblop]

Can you solve my google problems???? Then I will be greatefull. It throws a lot of error 403.
Copying (uploading) item ae19b1ba-25a5-4a39-89e8-fc203b8242df to cal_google<CENSORED>@gmail.com error: Unknown error occurred for cal_local_gcal/Personal: 403 Client Error: Forbidden for url: https://apidata.googleusercontent.com/caldav/v2/<CENSORED>@gmail.com/events/ae19b1ba-25a5-4a39-89e8-fc203b8242df.ics error: Use -vdebugto see the full traceback. Copying (uploading) item affb4f13-f804-4836-8865-4a481d50998d to cal_google<CENSORED>@gmail.com error: Unknown error occurred for cal_local_gcal/Personal: 403 Client Error: Forbidden for url: https://apidata.googleusercontent.com/caldav/v2/<CENSORED>@gmail.com/events/affb4f13-f804-4836-8865-4a481d50998d.ics error: Use-vdebugto see the full traceback. Copying (uploading) item b2158e34-bc7f-4019-822a-a209b184cd96 to cal_google/<CENSORED> error: Unknown error occurred for cal_local_gcal/Personal: 403 Client Error: Forbidden for url: https://apidata.googleusercontent.com/caldav/v2/<CENSORED>@gmail.com/events/b2158e34-bc7f-4019-822a-a209b184cd96.ics error: Use-vdebug to see the full traceback.

When I use -vdebug I get logs like this:

debug: Sending request... debug: 403 debug: {'Expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'Date': 'Fri, 27 Aug 2021 09:51:47 GMT', 'Pragma': 'no-cache', 'Content-Type': 'text/xml; charset=UTF-8', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Vary': 'Origin, X-Origin, Referer', 'Content-Encoding': 'gzip', 'Server': 'ESF', 'X-XSS-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'Transfer-Encoding': 'chunked'} debug: b'<?xml version="1.0" encoding="UTF-8"?>\n<D:error xmlns:D="DAV:"/>\n' error: Unknown error occurred for cal_local_gcal/Personal: 403 Client Error: Forbidden for url: https://apidata.googleusercontent.com/caldav/v2/<CENSORED>/events/affb4f13-f804-4836-8865-4a481d50998d.ics error: Use -vdebug to see the full traceback. debug: File "/usr/lib/python3.9/site-packages/vdirsyncer/cli/utils.py", line 75, in handle_cli_error debug: raise e debug: File "/usr/lib/python3.9/site-packages/vdirsyncer/sync/__init__.py", line 154, in sync debug: action.run(a_info, b_info, conflict_resolution, partial_sync) debug: File "/usr/lib/python3.9/site-packages/vdirsyncer/sync/__init__.py", line 177, in run debug: self._run_impl(a, b) debug: File "/usr/lib/python3.9/site-packages/vdirsyncer/sync/__init__.py", line 207, in _run_impl debug: href, etag = self.dest.storage.upload(self.item) debug: File "/usr/lib/python3.9/site-packages/vdirsyncer/storage/base.py", line 13, in inner debug: return f(self, *args, **kwargs) debug: File "/usr/lib/python3.9/site-packages/vdirsyncer/storage/base.py", line 13, in inner debug: return f(self, *args, **kwargs) debug: File "/usr/lib/python3.9/site-packages/vdirsyncer/storage/base.py", line 13, in inner debug: return f(self, *args, **kwargs) debug: File "/usr/lib/python3.9/site-packages/vdirsyncer/storage/dav.py", line 548, in upload debug: return self._put(href, item, None) debug: File "/usr/lib/python3.9/site-packages/vdirsyncer/storage/dav.py", line 516, in _put debug: response = self.session.request( debug: File "/usr/lib/python3.9/site-packages/vdirsyncer/storage/dav.py", line 397, in request debug: return http.request(method, url, session=self._session, **more) debug: File "/usr/lib/python3.9/site-packages/vdirsyncer/http.py", line 174, in request debug: r.raise_for_status() debug: File "/usr/lib/python3.9/site-packages/requests/models.py", line 953, in raise_for_status debug: raise HTTPError(http_error_msg, response=self) Copying (uploading) item b2158e34-bc7f-4019-822a-a209b184cd96 to cal_google/<CENSORED> debug: Normalized URL from 'b2158e34-bc7f-4019-822a-a209b184cd96.ics' to '/caldav/v2/<CENSORED>/events/b2158e34-bc7f-4019-822a-a209b184cd96.ics' debug: ==================== debug: PUT https://apidata.googleusercontent.com/caldav/v2/<CENSORED>@gmail.com/events/b2158e34-bc7f-4019-822a-a209b184cd96.ics debug: {'User-Agent': <CENSORED>.apps.googleusercontent.com', 'Content-Type': 'text/calendar', 'If-None-Match': '*'} debug: b'BEGIN:VCALENDAR\nPRODID:-//Google Inc//Google Calendar 70.9054//EN\nVERSION:2.0\nCALSCALE:GREGORIAN\nX-WR-CALNAME:<CENSORED>\nX-WR-TIMEZONE:Europe/Copenhagen\nBEGIN:VTIMEZONE\nTZID:Europe/Copenhagen\nX-LIC-LOCATION:Europe/Copenhagen\nBEGIN:DAYLIGHT\nTZOFFSETFROM:+0100\nTZOFFSETTO:+0200\nTZNAME:CEST\nDTSTART:19700329T020000\nRRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU\nEND:DAYLIGHT\nBEGIN:STANDARD\nTZOFFSETFROM:+0200\nTZOFFSETTO:+0100\nTZNAME:CET\nDTSTART:19701025T030000\nRRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU\nEND:STANDARD\nEND:VTIMEZONE\nBEGIN:VEVENT\nDTSTART;VALUE=DATE:20191219\nDTEND;VALUE=DATE:20191220\nDTSTAMP:20210622T084405Z\nORGANIZER;CN=<CENSORED>\nUID:b2158e34-bc7f-4019-822a-a209b184cd96\nATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;CN=thilda\n <CENSORED>;X-NUM-GUESTS=0:mailto:<CENSORED>\nATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;CN=sb\n <CENSORED>;X-NUM-GUESTS=0:mailto:<CENSORED>@gmail.com\nATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;CN=<CENSORED>;X-NUM-GUESTS=0:mailto:<CENSORED>\nCREATED:20210609T075001Z\nDESCRIPTION:\nLAST-MODIFIED:20210622T084405Z\nLOCATION:\nSEQUENCE:0\nSTATUS:CONFIRMED\nSUMMARY:M\xc3\xb8de p\xc3\xa5 b\xc3\xb8rne psyk Kl.09.00\nTRANSP:TRANSPARENT\nX-MICROSOFT-CDO-OWNERAPPTID:-1021599213\nBEGIN:VALARM\nACTION:DISPLAY\nDESCRIPTION:This is an event reminder\nTRIGGER:-P0DT15H0M0S\nEND:VALARM\nEND:VEVENT\nEND:VCALENDAR\n' debug: Sending request... ^C^CException ignored in: <module 'threading' from '/usr/lib/python3.9/threading.py'> Traceback (most recent call last): File "/usr/lib/python3.9/threading.py", line 1428, in _shutdown lock.acquire() KeyboardInterrupt: debug: 403 debug: {'Expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'Date': 'Fri, 27 Aug 2021 09:51:48 GMT', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Content-Type': 'text/xml; charset=UTF-8', 'Vary': 'Origin, X-Origin, Referer', 'Content-Encoding': 'gzip', 'Server': 'ESF', 'X-XSS-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'Transfer-Encoding': 'chunked'}

Unfortunately, I do not make much sense of this.
I believe it might be because I have done a lot of syncing the last few months, and the UUIDs are "known" by google. When I try to sync an event that already has been saved it causes problems, but this is just a theory. I used gcaltoolkit to clean up my calendar after everthing was saved twice. - It works sort of. It syncs, but throws a lot of errors in the process. I also tried to "empty" nextcloud, sogo and local storeage, so it took only google events, however this also causes problems. Then I reversed to a backup to ensure all my data is saved. :-)

[Bleala]

I had exactly the same problem, so maybe my solution can help you too :)

Took me literally a whole day to find out with a lot of trial and error^^

So let's start!
(Try this out first, to see if it works)

First of all go to https://calendar.google.com/, then click on the wheel right above, settings, on the left click on one of your calendar (doesn't matter which one) and click on export calendar.
Then a download should start with an .ics file.

Now this is the point where Baikal comes in. I did it this way, so maybe (just for trying) set up a Baikal server, that we have the same structure.

Here is a valid docker-compose.yml

version: "3.9"

########################### NETWORKS
networks:                                 
  baikal:
    internal: true
  default:
    driver: bridge

########################### SECRETS
secrets:
  mariadb_database:
    file: $SECRETSDIR/mariadb_database
  mariadb_password:
    file: $SECRETSDIR/mariadb_password
  mariadb_root_password:
    file: $SECRETSDIR/mariadb_root_password
  mariadb_user:
    file: $SECRETSDIR/mariadb_user

########################### SERVICES
services:
  # Baikal - a Cal and CardDAV server, based on sabre/dav, that includes an administrative interface for easy management.
  # https://hub.docker.com/r/ckulka/baikal
  app:
    image: ckulka/baikal:0.8.0-nginx
    container_name: baikal
    restart: unless-stopped
    networks:
      - default
      - baikal
    ports:
      - target: 80
        published: <choose_your_port>
        protocol: tcp
        mode: host
    env_file:
      - .env 
    volumes:
      - $DOCKERDIR/baikal/config:/var/www/baikal/config
      - $DOCKERDIR/baikal/data:/var/www/baikal/Specific
    depends_on:
      - db


  db:
    # MariaDB - MySQL Database
    # https://hub.docker.com/_/mariadb
    container_name: mariadb_baikal
    image: mariadb:10.6.4
    restart: always
    security_opt:
      - no-new-privileges:true
    networks:
      - baikal
    volumes:
      - $DOCKERDIR/mariadb/config/my.cnf:/etc/mysql/conf.d/my.cnf:ro
      - $DOCKERDIR/mariadb/data:/var/lib/mysql
    environment:
      - PUID=$PUID
      - PGID=$PGID
      - TZ=$TZ
    env_file:
      - .env 
    secrets:
      - mariadb_database
      - mariadb_password
      - mariadb_root_password
      - mariadb_user

and a .env file, place it inside the same folder as the docker-compose.yml

COMPOSE_PROJECT_NAME=baikal
PUID=<ID_of_your_user> --> you can use 0 (root) for testing
PGID=<Group_ID_of_your_user> --> you can use 0 (root) for testing
TZ="Europe/Vienna"

### Directories ###
DOCKERDIR="<path/to/folder>"
SECRETSDIR="<path/to/folder>/secrets"


### MariaDB ###
MYSQL_DATABASE_FILE=/run/secrets/mariadb_database
MYSQL_PASSWORD_FILE=/run/secrets/mariadb_password
MYSQL_ROOT_PASSWORD_FILE=/run/secrets/mariadb_root_password
MYSQL_USER_FILE=/run/secrets/mariadb_user

If you wanna use the docker secrets place them inside the secrets folder and run chmod -R 600 secrets/ otherwise just delete the _FILE from the mariadb environment variables and place the value there.

Start everything, go to the website and do the setup. When the installer asks for a database, choose Mysql (or was it MariaDB?) fill in the credentials from the mariadb container and everything should work.

Log into the Baikal admin interface and then you can create a user. You can't sync with the admin user, so just create one for yourself.

Now in the user interface click on calendars besides your user and then Add Calendar. Name it test or whatever.

After you have done that go to your SOGo, click on calendars, add a new calendar (name ist test again) and then click on the three dots and import calendar. Here you upload the .ics file you downloaded from your google calendar. Wait till it's done.

Last step is to go back to Google Calendar, add a new calendar, go to the settings of this calendar, scroll down a little bit until you find the Calendar-ID. (Looks like that: o6u62a7grj3ehmjl24iltqk2p0@group.calendar.google.com)

---

So know the "funny" part:

Set up the test configuration inside you Vdirsyncer config.
(Best way would be just to rename you config file to maybe config_backup and create a new config to test this.)

One for google:

[storage google_caldav]
type = "google_calendar"
token_file = "/vdirsyncer/google_calendar"
client_id = "<your-client-id>"
client_secret = "<your-client-secret"

One for Baikal:

[storage baikal_caldav]
type = "caldav"
url = "<your-url>/dav.php/calendars/<your-username>/"
username = "<your-username>"
password = "<your-password>"

And the last one for SOGo:
(Get the link from the test calendar, click in SOGo ui on calendar, on the three dots besides the test calendar and click on Link to this calendar (looks like that: https:///SOGo/dav//Calendar/47-6128CD00-3-67407780/

[storage mailcow_caldav_test]
type = "caldav"
url = "<link-from-sogo>"
username = "<your-email>"
password = "<your-password>"

Now set up the first pair:

[pair mailcow_baikal_caldav_test]
a = "mailcow_caldav_test"
b = "baikal_caldav"
collections = [["test", "<HERE-GOES-THE-NUMBER-FROM-SOGO(Like 47-6128CD00-3-67407780)>", "test"]]
conflict_resolution = "a wins"
metadata = ["color", "displayname"]

You see the pair contains mailcow and baikal, the collection which only contains collection name, the test calendar from baikal and the number from the sogo calendar.
Conflict resolution is a wins here (so mailcow).

Now run vidirsyncer discover. Maybe you have to authenticate with your google account again. And afterwards vdirsyncer sync and vdirsyncer metasync.

So now mailcow/sogo and baikal should have sync their calendars (Containing the original one from google).

After that change the conflict_resolution to "b wins", so that Baikal is the boss :)

The last step is to finally include the google sync. Create a new sync pair like that:

[pair google_baikal_caldav_test]
a = "google_caldav"
b = "baikal_caldav"
collections = [["test", "<your-google-calendar-ig (like o6u62a7grj3ehmjl24iltqk2p0@group.calendar.google.com)>", "test"]]
conflict_resolution = "b wins"
metadata = ["color", "displayname"]

Now run vdirsyncer discover again and if everything's ok, finally run vdirsyncer sync and vdirsyncer metasync. It should sync everything from Baikal to the new Google calendar.

The full test config for Vdirsyncer should look like this:

### General ###
[general]
status_path = "/vdirsyncer/status/"

### Storage ###
[storage google_caldav]
type = "google_calendar"
token_file = "/vdirsyncer/google_calendar"
client_id = "<your-client-id>"
client_secret = "<your-client-secret"

[storage baikal_caldav]
type = "caldav"
url = "<your-url>/dav.php/calendars/<your-username>/"
username = "<your-username>"
password = "<your-password>"

[storage mailcow_caldav_test]
type = "caldav"
url = "<link-from-sogo>"
username = "<your-email>"
password = "<your-password>"

### Pairs ###
[pair mailcow_baikal_caldav_test]
a = "mailcow_caldav_test"
b = "baikal_caldav"
collections = [["test", "<HERE-GOES-THE-NUMBER-FROM-SOGO(Like 47-6128CD00-3-67407780)>", "test"]]
conflict_resolution = "a wins"
metadata = ["color", "displayname"]

[pair google_baikal_caldav_test]
a = "google_caldav"
b = "baikal_caldav"
collections = [["test", "<your-google-calendar-id (like o6u62a7grj3ehmjl24iltqk2p0@group.calendar.google.com)>", "test"]]
conflict_resolution = "b wins"
metadata = ["color", "displayname"]

And the log after the sync should look like this (tested everything for you to get the proof it is working :) )

Syncing google_baikal_caldav_test/test
Copying (uploading) item 3A25DF56-DABE-4C1B-AAFA-0918863C2E13 to google_caldav/e7tu59oq7nierdmgbn37m05tb4@group.calendar.google.com
Copying (uploading) item cgojep31cgrjcb9pcph6ab9k6ss68b9o6ks3ab9j68s32d1nccsj0e32c4@google.com to google_caldav/e7tu59oq7nierdmgbn37m05tb4@group.calendar.google.com

I know it is quite a lot, but this way worked for me!

If this does also work for you, then you have to export every google calendar, delete it, create a new one (do not create any events!) and to the full manual again.

Attention: It is not possible to sync your personal google calendar, the one which is called like the e-mail address you used for your google account. If you want a personal calendar, just create a new one, call it personal and turn off the one which is called like your e-mail address!

I hope this is helping you too, tell me if not :)

[Sblop]

Thanks for your write-up. - I will go through this during my weekend and revert.

[Bleala]

@Sblop
Hey there!

Have you been able to solve your google problems? :)

Greetings

[Sblop]

@Bleala
Hi again,
Hope you have had a nice weekend.
I haven’t had the time to redo my setup as per your instructions, however I was trying a bit to get stuff to work.
First:
1: Sogo. Problem was caused by metadatasync in some of my calendars. In nextcloud I was able to select any color, but in SOGO you only have like 16 different colours. This caused a metadata sync error. The calendar sync error was solved by making a whole lot of storage for SOGO one for each pair. - I still do not fully understand why this was necessary as it was working fine for like 3 calendars, but not for 7.... Anyway... now that works.

For Google cal sync I have been reading your bit several times. If I sum up everything, you export your calendar, start with a clean sync with maybe one event and that works. - This lead me back to my suspicion that my problem is UUID related. I would love if the repair function had a function like --force-change-all-uuid. This would remove all old events and only add the new UUID, however unfortunately this is not implemented and i havent been able to find any tool that can do this online.... Perhaps I need to fine tune my python skills and write a script that can assist.

I have been trying to export all events, delete nextcloud, local storage and sogo, and import again from Google, but the total event number of events is not matching up, with a lack of like 1000 events, and I would be sad to lose that much history. Also, I would like that google is NOT in control, but I am from my local storage. It does however seem that google doesn't like that...

Maybe it is an implementation error in local storage that causes this error, so I plan to try direct sync between Nextcloud and google, just to try if this solves the problem. Baical might be the solution, however by now I have soo many storages for Google, next, sogo and local, and it is hard to remember everything. I will implement baical if nothing else works, but for now I will continue fiddling around.

Edit: You write it is not possible to sync to primary account (username@google.com). I do have issues, but this mostly works. I have the same problem across primary and created calendars alike.

[Bleala]

@Sblop
Good that your SOGo works now!

Strange, that your personal Google calendar works, mine did not work, no matter what i did, but Google has kinda strange behavior, don't know why.
As i said, maybe baikal helps you, would be nice if you tell me if so.

And if you write a Python script that could help with --force-change-all-uuid let me know, i would like to have something like that implemented in Vdirsyncer or, if they don't want to, in the container.

Greetings and have a nice day :)