Copy n Paste problems under Linux

[davidbannon]

Just as I released 0.14 I realised there is a significant issue, under Linux, wrt copy and paste when Unicode characters are involved. Specifically -

1. Copying Unicode between two tomboy-ng notes or within one note fails badly. Copying Unicode into or out of tomboy-ng works fine !
2. Copying anything into a note that has bulleted text causes the bullet character (a unicode char) to mess up. This also affects Mac.
3. I also believe Linux has an occasional issue applying to the first copy after the whole application is started. Its fine after that. This issue has been reported before and then withdrawn by the reporter ...

[davidbannon]

OK, temp and very ugly fix for now. Seems OK under linux but must also test against Mac. means that text copied and pasted within tomboy-ng will loose its formating. Better than what it does now.....

These hacks are to KControls and KMemo, not to tomboy-ng's code base. Hopefully, TK will be interested ......

In KControls.pas we will force ClipboardLoadStreamAs() to always return with plain text in AText, even when it thinks the RTF is OK. This should be a waste of cycles but later on we will force KMemo to only paste Plain Text, never RTF because the RTF is not unicode clean.

So, about line 1410 of KControls, add the ifdefs and stuff between them -

  Clipboard.GetFormat(AFormat, AStream);
  Result := True;
 {ifdef UNIX}
  AText := AsText;          // DRB April, 2018, chasing linux unicode paste bug
  // Force return with a copy of the plain text even when "we think" the RTF os OK
  // Over in KMemo #4790 we will force use of this plain text rather than the RTF
  // which is flawed. Until I find out whats wrong with it.
 {endif}
end else
begin
  AText := AsText;

And about #4786 of KMemo

  ExecuteCommand(ecClearSelection);
  OldSelectableLength := ActiveBlocks.SelectableLength;
 {ifdef UNIX}      
  Stream.Size := 0;         // DRB April 2018, chasing linux unicode paste bug
  // This disables any RTF Paste, only paste plain text so lose any format.
  // tempory measure until I work out whats wrong with the copied RTF.
  // depends on addition around  line 1412 of KControls
  {endif}
  if Stream.Size > 0 then
  begin
    Stream.Seek(0, soFromBeginning);
    // Stream.SaveToFile('pasted.rtf'); //debug line
    LoadFromRTFStream(Stream, SelEnd);
  end else

[xypd]

First thank you for taking the mantle and continuing this project. I really find Tomboy/Gnote to be the best app for my needs (it one of 3 note taking solutions that works with an older linux device).

I was looking at using tomboy-mono, but didn't work great on osx. I was using Gnote which was great, but that stop working after 10.13. I was going to try and compile gnote or try homebrewing
a solution, when I found your endeavors. I'm very excited since at lease at the moment I can see and carefully edit my notes

So, I'm happy to do test some updated binary's for you since I'm running Mac OS X 10.13.3. Or help best I can when I have time.

[davidbannon]

Yep, thats the problem. For sure.
As I noted above, I have a work around, not one I am particularly proud of but will push it out as a temporary fix. And, maybe a proper fix with the next release ?

Anyway, in the next day or so, I will replace the Linux and Mac binaries in v0.14 and if you could test them that would be great.

I am guessing you don't mean you want to build your own ? You would need to install Lazarus and the KControls addon and then patch KControls as detailed above. Bit of a slow job but not hard....

[xypd]

Yeah, I'm currently rebuilding my system on a new machine so I'm trying to keep it to only the important tools and files. I'm considering a few dev like tools for compiling some software (home-brew, even Lazarus), but Lazarus would only be for this app ... so hard to justify at the moment. I will consider it though once I can build a VM or boot drive for stuff like this. So for now I happy to test your binaries. And can let you know if that changes.

[davidbannon]

OK, I have updated the binaries associated with v0.14. The changes made are to the build env not the tomboy-ng source !
My tests do indicate that we've fixed the "pasting unicode" and "messed up bullet points" problem but at the expense of a copy between notes not taking along its format on Linux. I think thats a small price and hopefully one we can claim back in a future release.

[xypd]

Tested the latest update (39fa78b). The uni-code works well now, thanks.

However formatting and the bullets seem to be missing now on the Mac.

Additionally, the search command doesn't seem to work. The new note command does work, but only when a note is already open.

[davidbannon]

OK, can I clarify what you mean about Bullets and Formatting missing ? If you have a Note A and it contains the words "My Text" in bold, italic and you copy it to another note, Note B, you will end up with "My Text" in plain text in Note B. Thats what I mean about "not taking its formatting along".
Before this fix, that same paste would have failed completely to copy any unicode and would have killed any bullets that were already in Note B.

I don't rule out fixing the overall problem and have a copy between tomboy-ng notes retain the original formatting but it will involve me learning a lot more about codepages, RTF and UTF8. Obviously copy and paste between different applications almost always is plain text. And, you know what ? Personally, I suspect I'd rather it copied plain text most times !

Next, the search command ? Do you mean search in this note or search all notes ? Are you invoking it by using a control key combo or the menu item ? If menu, from a notes menu or from the top of screen menu ? I don't, personally, use the control (etc) keys very much on the Mac because I never know what keys have already been defined for something else.

Ctrl-F (search within a note) without a note having focus does not make sense but maybe Ctrl-Shift-F (search all notes) does. And I guess Ctrl-N (new note) probably does too. I think I can apply same tricks I used to make [that crossy thing key] Q work (quit the app). This would be Mac only code because Win/Linux users must use the mouse to activate the TrayIcon and then choose a menu item.

Thanks for your help !

[davidbannon]

Some progress with a 'real' fix. Sadly, it still needs to be applied to KControls rather than the tomboy-ng code and its not a full fix. Sigh.

Near end of kmemortf.pas there is a procedure TKMemoRTFWriter.WriteUnicodeString(const AText: TKString);

It has a line#4099, "if (Ansi <> '') and (Ansi <> #0) then"
Add another test, one to ensure the block is only entered if ANSI is only one byte -
if (Ansi <> '') and (Ansi <> #0) and (Length(ANSI) < 2) then

This will ensure that a multibyte char is passed on to next part of the procedure that appears as if it was always intended to do just that.

This model will now let us copy and paste unicode text and when you do so it does not mess up any pre existing bullet point characters. Good.

But I note that when we copy text that contains bullets, the first character of each bullet is dropped !
Ahhhh !

This appears to be a flaw in the pasting engine, not the copy side of things (exporting as RTF now works cleanly). So, as they say, further research is indicated.

David

[xypd]

Let me see if I can answer your questions.

Bullets:
I agree with how you explained your example. I was unsure if you previously were including the bullets in your unicode formatting issues. It seem we were talking about the samething ... mostly. So if this is what the current solution is I'm okay with it, since I normally don't need to copy from one note to another. Additional I can do this on my mobile device with Tomboy compatible program.

Search Command:
1.Do you mean search in this note or search all notes ? 2. Are you invoking it by using a control key combo or the menu item ? 3. If menu, from a notes menu or from the top of screen menu ?

1. Search in all notes.
2. I tried the control & command key + s.
3. The menu works as expected and brings up the search window, no problem.

As far as defined key as long as your app is currently in the foreground you can use most key for shortcuts (actions). so in this app current you're presented with neither a single note or an All notes window. So it not so confusing to go to the menu, however the previous tomboy compatible I was using had all notes showing at startup. Perhaps this can be a choice in the setting.

So I'm not sure how difficult it would be to add. But I think there's an opportunity to add control/command that would work on all platforms. And I think using Ctrl-F & Ctrl-Shift-F could work nicely too.

Thanks

[davidbannon]

xypd - lets try and keep things simple.

Could you please open a separate ticket for this shortcut key issue ? Best not to try to track multiple issues with one ticket. Can get confusing. Especially so as your one is a Mac only issue and I suspect we will strick some problems with keyboard shortcuts as I migrate Mac's tomboy-ng to 64bit cocoa.

[davidbannon]

Copy n Paste.
I have now worked out where, in KMemo, this copy and paste goes bad. In fact, in two places. I have documented my fixes at http://wiki.lazarus.freepascal.org/KControls/KmemoNotes and reported it to the author of KMemo. Hopefully he will release a fixed version, in the mean time, anyone wanting to build tomboy-ng needs to apply those fixes to KControls during their Lazarus install.
I'll leave this ticket open for awhile while we see if the that fixed version is, in fact, rolled out.

[kernelOfTruth]

I can confirm that formatting and bullets also seem to be missing:

especially highlighting doesn't seem to work on imported tomboy notes,

bold font + italics seem to be working.

When copying notes from e.g. gnote to tomboy-ng no formatting or bullets are preserved - it ends up as plain text

[davidbannon]

Next release will have this problem fixed in the binaries. But its not a source issue, its 'upstream' in KControls. So, anyone building from source must apply the patch mentioned above to KControls before compiling.
But copying notes from (eg) gnote and pasting into tomboy-ng is probably not going to preserve formating. The way copy and paste works is an application copying will put its content into the clipboard as plain text and possibly in other formats that it thinks are a good idea. Tomboy-ng uses KControls and KControls uses plain text AND RTF. So, when pasting, an app that thinks to check if the clipboard has any RTF may well copy over formatting. I don't know just what format gnote puts into the clipboard, if its RFT (unlikely) then next release will work.

I suspect better approach might be to transfer via file. I can easily make tomboy-ng import RTF and, with some effort, some other format as long as its well documented.

Worth looking to see what format gnote can export in perhaps ?

[davidbannon]

OK, finally committed a fix for the third and final problem identified in first post of this issue. Closing.