[Bug] Quote chararacters turned into strange characters when copying from Zotero 6.0.4 to Zettlr

[jeroenmuller]

Description

When copying notes or citations from Zotero 6.0.4 to Zettlr 2.2.4, the quotations marks included in the note/citation are somehow turned into strange characters in the Zettlr editor. LEFT DOUBLE QUOTATION MARK (“) is turned into “ and RIGHT DOUBLE QUOTATION MARK (”) is turned into â€�. These seem to be the underlying UTF-8 characters (see https://www.cogsci.ed.ac.uk/~richard/utf-8.cgi?input=2E&mode=bytes). When copying to other text editors, for example gedit or the terminal, this issue does not occur. When the text is first copied to gedit, and then from gedit to Zettlr, the problem also does not occur. This leads me to believe that the issue is due to a specific interaction between Zotero and Zettlr (perhaps due to Zotero using UTF-8 encoding and CodeMirror using UTF-16 encoding).

This is a problem because these characters are widely used in Zotero and in the new built-in note pdf annotator in Zotero 6 (see https://www.zotero.org/blog/zotero-6/). The problem makes my workflow with Zotero built-in notes and Zettlr very cumbersome. I don't know if the problem is caused by Zettlr, or if Zotero somehow does something wrong in a way that most other appications can deal with.

Reproducing

Steps to reproduce

1. Set Zotero language to English (Preferences -> Advanced -> Language). Localization affects quote symbols in Zotero so this issue may not occur in the same way for other languages
2. In Zotero preferences -> Export, set Note format to "Markdown + Rich Text" (the default option).
3. Add a new item to the library by clicking File -> New item -> Document. Enter a title for the document, e.g. Test
4. Drag the item from the library pane to the Zettlr editor.

Alternatively, create an item with a PDF attachment and add a note with the built-in PDF editor including an annotation (the highlighted text will be enclosed in quotes). Drag the note item from the library view into Zotero.

Expected behavior

The text should show up as “Test,” n.d. (it shows up like that when dragged to gedit)

Actual behaviour

The text shows up as “Test,� n.d.

Zettlr Version

Stable (most recent version)

Specify version

2.2.4

Your Platform

• Windows
• macOS
• Linux

Architecture

• Intel 64bit
• ARM 64bit

Operating System Version

Ubuntu 21.10 with xorg session

Additional Information

It is possible to circumvent this issue in some cases by changing the note template in Zotero to not use this type of quote:
https://www.zotero.org/support/note_templates

[boring-cyborg]

Thanks so much for opening up your first issue here on the repository! 🎉 We would like to warmly welcome you to the community behind the app! ☺️ We'll check in soon and have a look at your issue. In the meantime, you can check your issue and make sure it aligns with our contribution guidelines! Here's the comprehensive list:

Enhancements

An enhancement takes a feature and improves or alters its behaviour. Please make sure to argue how your proposition will aid non-technical text workers, and why it can't be emulated easily with other features or apps!

Feature requests

Feature requests introduce whole new features into the app. This requires a lot of work, so these might be turned down if the implementation costs supersede the benefits we expect to see from implementing it. Please do not be disappointed if that happens. It likely has nothing to do with your great request but simply with us and our missing resources!
You can of course always ask someone to implement this feature, because a PR with a working new feature has much higher chances of being merged! :)

Bug reports

Please note that one of the main reasons for why bug reports cannot be addressed is that there's not enough information for us to find and fix the bug you describe, so make sure you try to pinpoint the bug as close as possible.
The ideal bug report for us has two qualities:

1. The bug is always reproducible, at least within a certain context. 2. We know exactly what specifically goes wrong, and there is consensus on
   what should happen instead.

Please note that if you encounter behaviour that does not align with your expectations of what would happen, this might as well be simply intended behaviour and we need to simply clarify why the behaviour is the way it is. This is not to be considered a bug and such issues may be closed! Suggest an enhancement instead!
But now, have a great day and thank you again!

[nathanlesage]

Just tried this, works like a charm. My system is macOS 12.2.1, Zotero 6.0.4, and Zettlr 2.2.4+dev (i.e. most recent state of the develop branch). This is the note's contents in Zotero:

Literally dragging and dropping results in this:

---

Additionally, I just inspected the source code and can confirm that Zettlr treats all incoming text definitely as utf-8. What might be the cause of this could be that the turndown engine has a hiccup in 2.2.4. Pasting as plain text should definitely work (Ctrl+Shift+V).

---

That being said, it could be that today's nightly may already be a working version, feel free to try it out: https://nightly.zettlr.com/

Please report back whether or not this solves this issue.

[jeroenmuller]

Hi, thank you for the quick response. Pasting as plain text indeed works fine. I just tried the nightly and the problem persists, and it also persists when I switch to the Ubuntu display manager (Mutter) instead of xorg. This problem also existed for me with Zettlr 2.2.2 and with earlier versions of Zotero 6, so I do not think it is related to recent changes in Zettlr.

[nathanlesage]

Hm. Could be a problem with the operating system as well. It sounds to me as if something produces an encoding error when taking the string from Zotero into the clipboard and/or when taking the string from the clipboard into Zettlr.

I don't think that this one is Zettlr's fault.

[RuizSerra]

FWIW, I am having the same issue “Some highlighted quote.â€ with:

• Ubuntu 22.04
• Zotero 6.0.20
• Obsidian 1.1.9

[nathanlesage]

@RuizSerra This sounds as if this may be a Zotero problem … I am going to close this issue because I am unable to reproduce this issue, but I am also frequently updating Zotero, and it is likely that they fixed this in the meantime.

[RuizSerra]

The problem is that when wrapping the annotation in quotes, Zotero uses “ and ” (i.e. "Left Double Quotation Mark" U+201C and "Right Double Quotation Mark" U+201D) as opposed to " (i.e. "quotation mark" U+0022).

When dropping the dragged annotation into a plain text editor it shows the double quotes, but when dropping into Obsidian (and I assume Zettlr), it converts the double quotes into the “ and â€ for left and right respectively.

So I guess Zotero should change this to the "basic" quotation mark? Or should the note-taking apps somehow accept the more "fancy" quotations marks?

[nathanlesage]

This has nothing to do with fancy quotation marks. Zettlr uses these everywhere as well. This is an encoding issue. Zettlr internally works with UTF-8, and I believe Obsidian as well. My assumption is that the Zotero drag behavior may possibly encode strings in some ANSI or ISO encoding, possibly an oversight because many people don't drag and drop text, and rather use Ctrl-C/CTRL-V for that. In light of the fact that this only happens with Zotero and doesn't occur if you copy and paste text, I don't think it would be wise for Zettlr to implement an exception for that one case.