Error in vcard multiple REV/PRODID fields - This card is corrupted and has been fixed #129

snapet · 2017-02-10T16:32:00Z

Steps to reproduce

Login, choose contacts app
Since latest version, I see an exclamation point next to many contacts, stating to check automatically fixed data and request to force a save. I do this by removing or adding some data. Exclamation point disappears.
Switch to other Nextcloud application and back to Contacts. Exclamation point is back and changes are not saved.

Expected behaviour

I would expect the indicated fixed issue with the vcard to be permanently fixed and changed information to be saved.

Actual behaviour

I keep seeing the exclamation points and changed data is not saved.

Server configuration

Operating system:
Debian Testing 9.0

Web server:
Apache/2.4.25 (Debian)

Database:
postgresql-9.6 9.6.1-2 amd64

PHP version:
php7.0 7.0.15-1
php -v
PHP 7.0.15-1 (cli) (built: Jan 25 2017 10:17:41) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
with Zend OPcache v7.0.15-1, Copyright (c) 1999-2017, by Zend Technologies

Nextcloud version: (see Nextcloud admin page)
Nextcloud 11.0.1 (stable)
Contacts version: (see Nextcloud apps page)
1.5.3
Updated from an older Nextcloud or fresh install:
Nextcloud updated several times, but problem started after latest Contact app update.
Signing status:
No errors have been found.

List of activated apps:
Enabled:

activity: 2.4.1
admin_audit: 1.1.0
bookmarks: 0.9.1
calendar: 1.5.0
comments: 1.1.0
contacts: 1.5.3
dav: 1.1.1
federatedfilesharing: 1.1.1
federation: 1.1.1
files: 1.6.1
files_pdfviewer: 1.0.1
files_sharing: 1.1.1
files_texteditor: 2.2
files_trashbin: 1.1.0
files_versions: 1.4.0
files_videoplayer: 1.0.0
firstrunwizard: 2.0
gallery: 16.0.0
logreader: 2.0.0
lookup_server_connector: 1.0.0
mail: 0.6.2
nextcloud_announcements: 1.0
notifications: 1.0.1
ojsxc: 3.0.2
password_policy: 1.1.0
provisioning_api: 1.1.0
serverinfo: 1.1.1
sharebymail: 1.0.1
survey_client: 0.1.5
systemtags: 1.1.3
theming: 1.1.1
twofactor_backupcodes: 1.0.0
updatenotification: 1.1.1
workflowengine: 1.1.1
Disabled:
encryption
external
files_accesscontrol
files_automatedtagging
files_external
files_retention
ocsms
templateeditor
user_external
user_ldap
user_saml

The content of config/config.php:

{
    "system": {
        "instanceid": "ochf368ebq3c",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.example.com"
        ],
        "datadirectory": "\/var\/nc_data",
        "overwrite.cli.url": "https:\/\/nextcloud.example.com",
        "dbtype": "pgsql",
        "version": "11.0.1.2",
        "dbname": "nextcloud",
        "dbhost": "dbserver.example.com",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "localhost",
            "port": 6379
        },
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "mail_smtpmode": "smtp",
        "mail_from_address": "webmaster",
        "mail_domain": "example.com",
        "mail_smtpauthtype": "PLAIN",
        "mail_smtphost": "smtp.example.com",
        "mail_smtpport": "25",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "singleuser": false,
        "loglevel": 2,
        "debug": false,
        "appstore.experimental.enabled": true,
        "theme": ""
    }
}

Are you using external storage, if yes which one: local/smb/sftp/...
no
Are you using encryption: yes/no
yes
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no

Client configuration

Browser:
Firefox, Chromium

Operating system:
Opensuse Tumbleweed
CardDAV-clients:
Not relevant probably as I experience this problem from the browser interface.
Akonadi connector, CardDAV-Sync on Android 0.4.20, Outlook CalDav Synchronizer

Logs

Web server error log

/var/log/apache2/error_nextcloud.log is empty

Nextcloud log (data/nextcloud.log)

Below is just one entry, there are many.

{"reqId":"i9AxBA1cpY\/b06\/NPHYu","remoteAddr":"2a02:1807:2600:100::101","app":"webdav","message":"Exception: {\"Message\":\"HTTP\\\/1.1 415 Validation error in vCard: PRODID MUST NOT appear more than once in a VCARD component\",\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\UnsupportedMediaType\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/CardDAV\\\/Plugin.php(294): Sabre\\\\CardDAV\\\\Plugin->validateVCard('BEGIN:VCARD\\\\r\\\\nVE...', false)\\n#1 [internal function]: Sabre\\\\CardDAV\\\\Plugin->beforeWriteContent('addressbooks\\\/us...', Object(Sabre\\\\CardDAV\\\\Card), 'BEGIN:VCARD\\\\r\\\\nVE...', false)\\n#2 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#3 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1104): Sabre\\\\Event\\\\EventEmitter->emit('beforeWriteCont...', Array)\\n#4 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(513): Sabre\\\\DAV\\\\Server->updateFile('addressbooks\\\/us...', 'BEGIN:VCARD\\\\r\\\\nVE...', NULL)\\n#5 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPut(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#6 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#7 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(479): Sabre\\\\Event\\\\EventEmitter->emit('method:PUT', Array)\\n#8 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#9 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Server.php(227): Sabre\\\\DAV\\\\Server->exec()\\n#10 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(30): OCA\\\\DAV\\\\Server->exec()\\n#11 \\\/var\\\/www\\\/nextcloud\\\/remote.php(165): require_once('\\\/var\\\/www\\\/nextcl...')\\n#12 {main}\",\"File\":\"\\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/CardDAV\\\/Plugin.php\",\"Line\":397,\"User\":\"peter\"}","level":4,"time":"2017-02-10T15:08:29+00:00","method":"PUT","url":"\/remote.php\/dav\/addressbooks\/users\/peter\/contacts\/185f70a9-5ca0-4e38-a1bf-43ec601e059b.vcf","user":"peter","version":"11.0.1.2"}

Browser log

From Chromium console

dav.js?v=52e17de…:3641 PUT https://nextcloud.example.com/remote.php/dav/addressbooks/users/peter/contacts/185f70a9-5ca0-4e38-a1bf-43ec601e059b.vcf 415 (Unsupported Media Type)
send @ dav.js?v=52e17de…:3641
callee$2$0$ @ dav.js?v=52e17de…:3156
tryCatch @ dav.js?v=52e17de…:121
invoke @ dav.js?v=52e17de…:294
Gp.(anonymous function) @ dav.js?v=52e17de…:330
onFulfilled @ dav.js?v=52e17de…:3737
(anonymous) @ dav.js?v=52e17de…:3726
co @ dav.js?v=52e17de…:3722
send @ dav.js?v=52e17de…:3134
updateObject @ dav.js?v=52e17de…:3415
updateCard @ dav.js?v=52e17de…:1799
updateCard @ dav.js?v=52e17de…:1524
update @ contact_service.js:171
ctrl.updateContact @ contactDetails_controller.js:72
fn @ VM597:4
$eval @ angular.js?v=52e17de…:17682
(anonymous) @ angular.js?v=52e17de…:24916
(anonymous) @ angular.js?v=52e17de…:27691
forEach @ angular.js?v=52e17de…:321
$$writeModelToScope @ angular.js?v=52e17de…:27689
writeToModelIfNeeded @ angular.js?v=52e17de…:27682
(anonymous) @ angular.js?v=52e17de…:27676
validationDone @ angular.js?v=52e17de…:27603
processAsyncValidators @ angular.js?v=52e17de…:27586
$$runValidators @ angular.js?v=52e17de…:27530
$$parseAndValidate @ angular.js?v=52e17de…:27669
$commitViewValue @ angular.js?v=52e17de…:27637
(anonymous) @ angular.js?v=52e17de…:27774
(anonymous) @ angular.js?v=52e17de…:19612
completeOutstandingRequest @ angular.js?v=52e17de…:5964
(anonymous) @ angular.js?v=52e17de…:6243
dav.js?v=52e17de…:3649 Uncaught (in promise) Error: Bad status: 415
    at XMLHttpRequest.request.onreadystatechange (dav.js?v=52e17de…:3649)

The text was updated successfully, but these errors were encountered:

snapet · 2017-02-11T08:39:02Z

I noticed there is a save button at the top in the contacts pages that have the exclamation point, next to the download and delete icons. However, when I click that I get the same error I documented above. When I try to download the card it fails. In my browser at the bottom it says: "Failed - Server problem" (This also fails when trying to download the contacts that have not been "fixed" and have no exclamation point in the interface.) I can click resume or cancel. These also fail.
No entries in my apache error logs when I try any of this.
And finally, when changing the "fixed" contacts from the contacts application in Kontact on Opensuse Tumbleweed, this works and no errors get logged.

skjnldsv · 2017-02-11T13:52:13Z

Okay, that what I thought :)
Could you post the impacted vcard here pls?

snapet · 2017-02-11T17:35:34Z

No, I get an error when I try to download it. Is there another way to get at it? SQL query against postgres?

cheers

skjnldsv · 2017-02-11T20:53:16Z

What is the error? Try opening the link to a new tab, I had this bug once! :)

snapet · 2017-02-12T09:13:59Z

After rebooting just now I am able to download one of the cards. I changed some names and numbers, the rest should be as I downloaded it from the nextcloud contacts interface. Had to rename it to txt to upload also.
a108c5bd-c089-4771-aa8e-378585e6bf6b.vcf.txt

skjnldsv · 2017-02-12T14:49:20Z

Yep, this is coming directly from your vcard. There is too many REV fields, VERSION and PRODID. see required cardinality
We might add a protection against this in the next version though, I will keep this open for this purpose only.

For your fix, just edit the raw vcard and remove the extra unwanted lines.

skjnldsv · 2017-02-12T15:09:01Z

Also, the error you get about the corrupted card comes from the vact that you use a comma into your groups. In your provided card, are you using two groups or one?

snapet · 2017-02-13T07:56:33Z

Ok, thanks for looking into this. I have a couple of final questions however.

Since the interface states it has fixed things, have other things been fixed or is it referring to these multiple occurrences?
Is there a way to correct these multiple occurrences with a query in the database? (I have hundreds of these...)
Can I find out where these multiple occurrence come from?
The comma is not in the interface, it is shown in the export only. Is this wrong? I have no idea where or how I should fix this. What I did notice is that when I edit contacts with kontact my groups are removed upon saving. Maybe this is related, but I don't know where the comma comes from, I never type this.
Finally, which of the PRODID lines should I keep? (Sabre I would think but "owncloud" is so common still in nextcloud code that it may be a candidate as well.)

snapet · 2017-02-13T08:12:49Z

This may be an error after all I have the feeling. All flagged contacts do indeed have several groups assigned. I have no comma's in my groups, the comma is only exported (escaped as you see in the example I posted here) when downloading the contact.
In the cardinality link you sent I read that the categories are actually supposed to be comma separated, so this is probably ok?
The only thing I can think of now is that that comma should not be escaped when downloading, as it is a separator and not part of my group(name). Could something be amiss there?

skjnldsv · 2017-02-13T14:52:14Z

The comma is unescaped, so the group is displayed as 'xxx,zzz' instead of two groups: 'xxx' and 'zzz'.
This is why your card has the warning at first.

In the meantime, the error you get isn't coming from the warning about the groups. It is happening upon save when you edit/click the save button on this card because it has invalid fields cardinality (PRODID, REV)

I can't tell you which to remove, but it shouldn't cause a lot of trouble if you keep the one you like the most. You can also remove the REV and PRODID. They will be replaced when you'll save the card later on. Keep only one VERSION and you'll be fine.

snapet · 2017-02-13T15:53:44Z

The error can't come from having comma's in my groups, I don't have comma's in my groups.
Also nowhere are my groups displayed comma separated in the web interface or concatenated with that comma.
I do have two groups for many contacts and they display just fine. I meant in the vcf file I uploaded the comma is escaped, there is a "" in front of it. When I create a new contact, assign it two groups and download the vcf file, the comma is not escaped. Don't know why that is.
What I did see is that for that new contact the CATEGORIES are in the carddata field of the oc_cards table. For the other contact it sits in the properties table oc_cards_properties with name=CATEGORIES and value is a comma separated list of my groups. Maybe either one of those methods escapes the comma's when I download?
Solving that cardinality is not trivial, a lot of work and not knowing where that problem came from makes me itchy.
Thanks again for your work... and patience.

skjnldsv · 2017-02-13T16:09:55Z

I assumed you had issues with your group based on the vcard you posted... 😕

Could you post a screenshot of the contact before you save please?

MeCias · 2017-02-13T17:35:28Z

Hi,
I am not sure if that helps, but I faced the same issue described above for ten of 120 contacts.
So I compared the working contacts with the highlighted ones and in my case all issues were related to the groups as well. I noticed that there was a backslash within the non working contacts.

If I stick to the example that snapet gave I adjusted as follows:
from

(I had to attach an screenshot, otherwise the two backslashes that I removed were not displayed)

to

BEGIN:VCARD
VERSION:3.0
N:papa;Aiko;;;
FN:Aiko papa
TEL;TYPE=CELL,VOICE:+32 5555 0 89 81
CATEGORIES:City,Noel
END:VCARD

After a re-import it seems to work again for me. I believe it had to do with those backslashes.

skjnldsv · 2017-02-13T18:03:05Z

Don't remove the backslash on the tel parameter. 😲
I said it already, the only thing you need is to remove the REV and PRODID lines and keep only one VERSION.

You can also remove the REV and PRODID. They will be replaced when you'll save the card later on. Keep only one VERSION and you'll be fine.

It will fix the contact. Then if the contacts app displays a warning, you should be able to save the card again.

MeCias · 2017-02-13T20:08:51Z

Was just one contact anyway that had that backslash on the tel parameter. Had no impact at all. I re-added that contact, just in case. The other ones I cleaned up as you said

remove the REV and PRODID lines and keep only one VERSION

and now it works quite well, even the import of all contacts is now working as without any errors.

snapet · 2017-02-18T08:24:10Z

Hi,
Thanks for the info MeCias.
I read into that document you linked earlier skjnldsv. You are right in every remark you make but what is puzzling is that often removing \ from , doesn't seem to make any difference.
Also, does anyone know what could be responsible for the double entries and the fact that the comma's in my groups ended up being escaped where they really must not be? I find no info on this, probably because while that is the spec most programs handle these multiple tags gracefully. Moreover, this (, in CATEGORIES) is one clear instance where the rules apparently did not apply as I still ended up with separate groups. I am kind of hoping this isn't going to be a recurring problem.

Anyway, after searching quite long I ended up with the following update statement to clean up the double VERSION and PRODID entries and remove the REV line for one address book . I couldn't get my head around unescaping only the comma's in my CATEGORIES lines only, but that was only a save click, which you guys provide.
BACKUP before you do this, it takes half a second to run, if something goes wrong it will take long to repair. (I noticed the hard way REV is likely to occur in contact pictures...)
UPDATE oc_cards set carddata=convert_to(regexp_replace(convert_from(carddata, 'UTF8'), E'(BEGIN.?\\015\\012)(VERSION:3.0\\015\\012)+?(PRODID:.\\015\\012)+(.)(REV.?\\015\\012)?(.*)', E'\\1\\2\\3\\4', 'g'), 'UTF8') where id = ANY (SELECT id from oc_cards where (convert_from(carddata, 'UTF8') ~ E'(VERSION:3.0\\015\\012){4,4}') AND addressbookid=2 );

nursoda · 2017-02-27T11:49:11Z

I had a similar case that drove me crazy: I edited one certain contact and saved, no error. But upon reloading or syncing my smartphone contacts, I did not see my edits. Took me a while until I figured that the log contains a silent error which leads to the contact not being saved at all. In my case, it was a double REV:. Please make sure that ALL entries of the database are valid, and throw errors on the USER. Upon importing, upon saving, upon database migration, app update, etc. It's insane to have data INSIDE the database that the app cannot handle without noticing the user.

elgaard · 2018-02-15T16:49:49Z

I had the same problem with Nextcloud 13.

Contacts could not be updated from the webinterface. But they could be updated from e.g., a phone using caldav. And after that those contact could be updated from the webinterface.

I fixed it for all contacts by:

update oc_cards set carddata=REPLACE(carddata,'PRODID:-//ownCloud//NONSGML Contacts 0.5.0.0//EN\r\n','');

billboule · 2018-06-15T15:24:13Z

@elgaard Thanks for your SQL line.... you saved me so much time !

digitalfox · 2021-02-05T23:39:51Z

same issue here. Various owncloud / sabre multiple declaration:

VERSION:3.0
VERSION:3.0
VERSION:3.0
VERSION:3.0
VERSION:3.0
PRODID:-//ownCloud//NONSGML Contacts 0.4.0.1//EN
PRODID:-//Sabre//Sabre VObject 3.4.2//EN
PRODID:-//ownCloud//NONSGML Contacts 0.2.5//EN

A quick fix:

make a backup before, a mistake is so easy...
export from nextcloud web interface all your contacts as one .vcf file
change PRODID with exact same name and remove duplicate consecutive lines with somethings like that:

cat contacts.vcf | sed 's#PRODID.*#PRODID:-//fox_script 0.0.1//EN#'|uniq > contacts_cleaned.vcf

delete contacts on nextcloud web interface
upload newly created contacts_cleaned.vcf

skjnldsv added the bug Something isn't working label Feb 10, 2017

skjnldsv added invalid This doesn't seem right 1. to develop Accepted and waiting to be taken care of labels Feb 12, 2017

skjnldsv self-assigned this Feb 12, 2017

skjnldsv changed the title ~~This card is corrupted and has been fixed. - Not working for me.~~ Error in vcard multiple REV/PRODID fields - This card is corrupted and has been fixed Feb 12, 2017

skjnldsv added medium Medium priority and removed invalid This doesn't seem right labels Jul 26, 2017

This was referenced Jul 26, 2017

Export generates dupe PRODID tags which break Import #209

Closed

Validate rev prodid version and validation edit #137

Merged

skjnldsv added 3. to review Waiting for reviews and removed 1. to develop Accepted and waiting to be taken care of labels Jul 26, 2017

skjnldsv modified the milestone: 1.5.4 Jul 26, 2017

irgendwie closed this as completed in #137 Sep 14, 2017

schneidr mentioned this issue Mar 1, 2021

Unable to add the Birthday field to existing contacts #2096

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error in vcard multiple REV/PRODID fields - This card is corrupted and has been fixed #129

Error in vcard multiple REV/PRODID fields - This card is corrupted and has been fixed #129

snapet commented Feb 10, 2017 •

edited by skjnldsv

snapet commented Feb 11, 2017

skjnldsv commented Feb 11, 2017

snapet commented Feb 11, 2017

skjnldsv commented Feb 11, 2017 •

edited

snapet commented Feb 12, 2017

skjnldsv commented Feb 12, 2017

skjnldsv commented Feb 12, 2017

snapet commented Feb 13, 2017

snapet commented Feb 13, 2017

skjnldsv commented Feb 13, 2017 •

edited

snapet commented Feb 13, 2017

skjnldsv commented Feb 13, 2017

MeCias commented Feb 13, 2017 •

edited

skjnldsv commented Feb 13, 2017

MeCias commented Feb 13, 2017

snapet commented Feb 18, 2017 •

edited

nursoda commented Feb 27, 2017

elgaard commented Feb 15, 2018

billboule commented Jun 15, 2018

digitalfox commented Feb 5, 2021

Error in vcard multiple REV/PRODID fields - This card is corrupted and has been fixed #129

Error in vcard multiple REV/PRODID fields - This card is corrupted and has been fixed #129

Comments

snapet commented Feb 10, 2017 • edited by skjnldsv

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

Client configuration

Logs

Web server error log

Nextcloud log (data/nextcloud.log)

Browser log

snapet commented Feb 11, 2017

skjnldsv commented Feb 11, 2017

snapet commented Feb 11, 2017

skjnldsv commented Feb 11, 2017 • edited

snapet commented Feb 12, 2017

skjnldsv commented Feb 12, 2017

skjnldsv commented Feb 12, 2017

snapet commented Feb 13, 2017

snapet commented Feb 13, 2017

skjnldsv commented Feb 13, 2017 • edited

snapet commented Feb 13, 2017

skjnldsv commented Feb 13, 2017

MeCias commented Feb 13, 2017 • edited

skjnldsv commented Feb 13, 2017

MeCias commented Feb 13, 2017

snapet commented Feb 18, 2017 • edited

nursoda commented Feb 27, 2017

elgaard commented Feb 15, 2018

billboule commented Jun 15, 2018

digitalfox commented Feb 5, 2021

snapet commented Feb 10, 2017 •

edited by skjnldsv

skjnldsv commented Feb 11, 2017 •

edited

skjnldsv commented Feb 13, 2017 •

edited

MeCias commented Feb 13, 2017 •

edited

snapet commented Feb 18, 2017 •

edited