Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Contact syncing not working on Mac OS #70

Closed
zakkak opened this issue May 27, 2019 · 39 comments

Comments

@zakkak
Copy link
Contributor

commented May 27, 2019

Contacts appear on console when using -D but not in the Contacts.app

Possibly related to #54 #65

@tasn

This comment has been minimized.

Copy link
Member

commented May 28, 2019

Could maybe also be #66. I hope I can finally fix #66 in the next few days which will hopefully fix this whole array of issues. Extremely annoying, I know, and I'm sorry. It's just that Radicale doesn't support vcard 4.0 and I therefore need to implement it myself.

@zagatta-sonah

This comment has been minimized.

Copy link

commented Jul 7, 2019

I have the same problem.

I can download the VCF from the local webserver but the contacts are not showing up in the Contacts app

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 8, 2019

Any interesting information when running etesync-dav with -D?

@zagatta-sonah

This comment has been minimized.

Copy link

commented Jul 11, 2019

I cannot interpret if it is interesting, but here it is.
I (hopefully) removed all unique identifiers, since this is public:

etesync.log

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 15, 2019

I don't see anything relevant there. It works for me though. :|
What version of etesync-dav are you using? MacOS version?

@zakkak

This comment has been minimized.

Copy link
Contributor Author

commented Jul 16, 2019

Hi @tasn

When running with -D I can see my contacts printed on the terminal

$ mac-etesync-dav --version
EteSync DAV version:  0.6.0
Radicale Storage EteSync version:  0.8.0
Radicale version:  2.1.11

MacOS Mojave v10.14.5

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 19, 2019

I fixed #65 and a few other things which may be related. Could you please try again with v0.7.1?

@zakkak

This comment has been minimized.

Copy link
Contributor Author

commented Jul 23, 2019

I am trying to setup carddav with the latest version and I am getting "Unable to verify account name or password".

When I get the above etesync-dav -D prints nothing

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 23, 2019

Unable to verify account name at what stage? What are you trying and what gives you the error?

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 23, 2019

Where are you getting the above though? In Contacts.app? So before anything is running?
So let's get again as many details as possible, first of all what do you get with etesync-dav --version?
Second, what are the exact settings you used in Contacts.app?
Does calendar work or you haven't tried?

It looks like it doesn't even reach etesync-dav, so the problem may be before. Have you correctly set up SSL as instructed in: https://github.com/etesync/etesync-dav/blob/master/README.md#macos-mojave ?

@zakkak

This comment has been minimized.

Copy link
Contributor Author

commented Jul 25, 2019

The issue turned out to be that etesync-dav was failing to load the correct configuration. Passing the configuration through command line fixes the above issue.

etesync-dav -C ~/Library/Application\ Support/etesync-dav/radicale.conf

Note that I am running etesync-dav from /Applications/etesync-dav.app/Contents/MacOS.

Despite allowing me to add the account I still can't see my contacts.
I use the following configuration:
image

For completeness and future reference I respond to your questions below.

Unable to verify account name at what stage?

This is in the Internet Accounts -> New CardDav account dialog.

Where are you getting the above though? In Contacts.app?

No, in the System Preferences when adding a new CardDav account

What are you trying and what gives you the error?

I am trying to setup my MacOS to sync contacts with etesync.

So before anything is running?

Not sure what the question is here

So let's get again as many details as possible, first of all what do you get with etesync-dav --version?

$ ./etesync-dav --version
EteSync DAV version:  0.8.1
Radicale version:  2.1.11

Second, what are the exact settings you used in Contacts.app?

image

Does calendar work or you haven't tried?

I didn't try, but removing the account and trying to add it again fails with the same message

It looks like it doesn't even reach etesync-dav, so the problem may be before. Have you correctly set up SSL as instructed in: https://github.com/etesync/etesync-dav/blob/master/README.md#macos-mojave ?

Yes, that was done for the previous version.

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 26, 2019

You shouldn't be using the configuration file, etesync-dav has finally broken free from that. Are you using the etesync server on etesync.com? Or self-hosting? If the latter, please make sure ETESYNC_URL is correctly set when running.

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 26, 2019

Also, please take a look at https://github.com/etesync/etesync-dav/blob/master/macos-instructions.md
You should be using "advanced" (according to those instructions).

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 26, 2019

Actually, I think I know what it may be. Because we no longer use the configuration file it's not picking up the ssl configuration.
Try running etesync-dav like this:
etesync-dav --ssl --certificate ~/Library/Application Support/etesync-dav/etesync.crt --key ~/Library/Application Support/etesync-dav/etesync.key

Assuming you used the same paths from the guide. Do things now work?

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 26, 2019

I think I fixed everything with 0.9.1, please try that.

Edit: I fixed the SSL issue, so calendar will now work again. While it looks like Contacts are syncing correctly, I also can't seem to figure out how to get them to work again. I wonder when this regression happened.

@zakkak

This comment has been minimized.

Copy link
Contributor Author

commented Jul 26, 2019

Are you using the etesync server on etesync.com?

Yes

Also, please take a look at https://github.com/etesync/etesync-dav/blob/master/macos-instructions.md
You should be using "advanced" (according to those instructions).

For CardDav it says "Account Type: Manual"

image

Actually, I think I know what it may be. Because we no longer use the configuration file it's not picking up the ssl configuration.

Passing the configuration file as an argument though made it work (without the need for etesync-dav --ssl --certificate ~/Library/Application Support/etesync-dav/etesync.crt --key ~/Library/Application Support/etesync-dav/etesync.key)

I think I fixed everything with 0.9.1, please try that.

Just downloaded and installed 0.9.1

$ ./etesync-dav --version
Traceback (most recent call last):
  File "etesync-dav", line 12, in <module>
ImportError: cannot import name 'SSL_KEY_FILE' from 'etesync_dav.config' (/var/folders/s6/bpyt31h91h79xltdlc1v32600000gq/T/_MEIvELx4s/etesync_dav/config.pyc)
[67061] Failed to execute script etesync-dav
$ ./etesync-dav --ssl --certificate ~/Library/Application\ Support/etesync-dav/etesync.crt --key ~/Library/Application\ Support/etesync-dav/etesync.key
Traceback (most recent call last):
  File "etesync-dav", line 12, in <module>
ImportError: cannot import name 'SSL_KEY_FILE' from 'etesync_dav.config' (/var/folders/s6/bpyt31h91h79xltdlc1v32600000gq/T/_MEIOexPu1/etesync_dav/config.pyc)
[67117] Failed to execute script etesync-dav
@tasn

This comment has been minimized.

Copy link
Member

commented Jul 26, 2019

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 27, 2019

A couple of comments:

  1. About my comment above regarding the "manual" - please ignore that. I was wrong, as you said. :)
  2. I fixed things in 0.9.1, you no longer need to pass --ssl --certificate and --key, just run normally.

As for the issue. It seems like the binary I uploaded was malformed. I now uploaded a new binary that fixes this issue. Sorry about the noise.

Regarding the original issue (contact sync): I think there may still be a problem. I need to figure out this regression but you are right, it doesn't seem to be syncing for me either (only contacts and only on the mac).

@zakkak

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2019

Could maybe also be #66. I hope I can finally fix #66 in the next few days which will hopefully fix this whole array of issues. Extremely annoying, I know, and I'm sorry. It's just that Radicale doesn't support vcard 4.0 and I therefore need to implement it myself.

It looks like the problem is exactly the opposite. Contants.app only supports vcard 3.0 (allthough I was able to successfully import a vcard 4.0 manually).

Note that I tried creating a contact in the Contacts.app and it was successfully synced with my android device (in the debug logs it appears it was added as a vcard 3.0). The other way around though is not working!

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 30, 2019

Are you sure it doesn't support 4.0? Got a link to support that? I couldn't find anything online.
What you are describing still sounds like #66 to me. Radicale is reporting it's a 3.0 collection so Contacts.app is limiting you to only 3.0. I'm not sure, I really need to figure this one out.

@zakkak

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2019

Are you sure it doesn't support 4.0? Got a link to support that?

I base my claim on the fact that when exporting a contact it is exported in v3.0 format.
There is also this old link https://alessandrorossini.org/the-sad-story-of-the-vcard-format-and-its-lack-of-interoperability/
and an Apple "vCard4-as-vCard3" mention in https://www.davx5.com/faq/cant-manage-groups-on-device/

Radicale is reporting it's a 3.0 collection so Contacts.app is limiting you to only 3.0

I am not sure this is the case. Even if I export a contact manually (no etesync-dav involvement) I get a v3.0 vcard.

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 30, 2019

I don't know what vCard4-as-vCard3 actually means in practice, as I haven't been able to find information about it. Anyhow, v3 and v4 are fairly compatible, so I just need to figure out what's there that's breaking things. I wonder if there's a way to get debug output from Contacts.app.

@zakkak

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2019

I wonder if there's a way to get debug output from Contacts.app.

Yes, that would be great.
A google search gave me this defaults write com.apple.AddressBook ABShowDebugMenu -bool true, which enables a Debug menu in the app, however I don't see anything useful yet.

@zakkak

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2019

From Contacts.app preferences

image

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 30, 2019

This picture makes me so sad. :(

@zakkak

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2019

I have downloaded the vcf file from the webui and imported it to Contacts. The contacts were added (but twice each :/) and contacts containing unicode (Greek) characters where broken.

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 30, 2019

If memory serves, vCard v3.0 needs you to set the charset while v4 assumes utf-8, which would explain what you're seeing. I think at this point the only solution is to make etesync-dav serve only version 3.0 vCards (convert the 4.0 ones), and accept both. This will also fix #66 in a sense.

@zakkak

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2019

Looks like your memory is correct

v3 https://tools.ietf.org/html/rfc2425#section-5.5

5.3. Required parameters

Required parameters: charset

The "charset" parameter is as defined in [RFC-2046] for other body
parts. It is used to identify the default character set used within
the body part.

v4 https://tools.ietf.org/html/rfc6350

3.1. Charset

The charset (see [RFC3536] for internationalization terminology) for
vCard is UTF-8 as defined in [RFC3629]. There is no way to override
this. It is invalid to specify a value other than "UTF-8" in the
"charset" MIME parameter (see Section 10.1).

@tasn

This comment has been minimized.

Copy link
Member

commented Jul 30, 2019

I figured. My last suggestion will fix this issue as well. It shouldn't be too hard to do I think, at least for the common cases.

@zakkak

This comment has been minimized.

Copy link
Contributor Author

commented Jul 30, 2019

🤞

@tasn

This comment has been minimized.

Copy link
Member

commented Aug 27, 2019

Just to confirm again what we already figured out before: Contacts.app just ignores version 4.0 vCards. I'm really considering at this point, to just move all of EteSync to vCard 3.0 as trying to use 4.0 has been nothing but pain, or at the very least, as said above, create a conversion layer in etesync-dav. I'll start with a very hacky attempt and let's see how it goes.

tasn added a commit that referenced this issue Aug 27, 2019
Transform vCard 4.0 contacts to 3.0 when serving
This is a workaround for both #70 and #66. Essentially the problem is
that both Radicale and macOS only support vcard 3.0 and EteSync uses
version 4.0 internally.
Luckily, version 3.0 and 4.0 are quite similar, so we can hackily
transform from 4.0 to 3.0 when serving contacts so they at least show
and work on macOS.

We can in the future actually implement correct transformations, though
at least for now, this works and solves the issues people were facing on
Macs.
@tasn

This comment has been minimized.

Copy link
Member

commented Aug 27, 2019

OK, I implemented a fairly horrible workaround that should hopefully fix issue. We just transform vCard 4.0 to 3.0 when serving them in etesync-dav. This change doesn't make me very happy, but it at least "fixes" this issue. In the future we can implement a better, more complete, transformation.
What's even more annoying is that it used to work, and it just seems to be a regression with either macOS or potentially our code. I really don't know, but it looks like the mac is at fault.

@tasn

This comment has been minimized.

Copy link
Member

commented Aug 27, 2019

Fixed in v0.11.0

@zakkak

This comment has been minimized.

Copy link
Contributor Author

commented Aug 28, 2019

Thanks @tasn despite being "horrible" as you say, it works and that's good enough for now :)

@tasn

This comment has been minimized.

Copy link
Member

commented Aug 28, 2019

@zakkak, well, it's in because it's "horrible" and "works". If it was horrible and broken I wouldn't have pushed it. ;)

J/K, thanks for the feedback, yeah, I think it'll do the trick for now. I still don't understand why it stopped working, but I guess they changed something in Mojave. Who knows.

@lasscap

This comment has been minimized.

Copy link

commented Sep 6, 2019

It workes for my now, too. But unfortunately only for one of my two address books. It is the one which I created later.

The other one is not even shown.

@tasn

This comment has been minimized.

Copy link
Member

commented Sep 9, 2019

Not even shown where? In Contacts.app or https://localhost:37358 when you click on your username?

@lasscap

This comment has been minimized.

Copy link

commented Sep 10, 2019

Not shown in Contacts.app. In http://localhost:37359/ under my username it is shown.

@tasn

This comment has been minimized.

Copy link
Member

commented Sep 12, 2019

Have you tried adding them again? Maybe there was an issue before. I tested here when I closed this issue, and it seemed to work, but I'm happy to test again early next week if you can confirm even adding again didn't fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.