When Copying, Line Wrapped Text Is Inconsistently Broken Into Multiple Lines

[ghost]

Environment

Platform = Win32NT
Version = 10.0.19041.0
Version String = Microsoft Windows NT 10.0.19041.0
Windows Terminal Version = 1.0.1811.0
SSH Version = OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
WSL Version = 2

Steps to reproduce

1. SSH into any linux server or open a WSL tab (My testing specifically used an SSH session).
2. create a file called 'test.txt' containing a line long enough to wrap twice with the last wrap having just a few characters.
3. Use more test.txt to list the contents of the create file.
4. Copy the output and paste it into notepad/notepad++
5. Resize the Terminal Window (you can resize and then put it back to the same size)
6. Copy the output again and paste it into notepad/notepad++

The 'more' command was the easiest way to get a reproducible result but I have seen this behavior with many other Linux programs some inconsistently and some consistently.

Expected behavior

Pasted text should be a single line, for both steps 4 and 6, as the copied output was wrapped by the terminal and not broken into separate lines by the output program.

Actual behavior

For step 4, the single line gets broken into multiple lines depending on the size or your terminal and how many lines the text got wrapped into. Then on step 6, the line is correctly copied as a single line.

settings.json.txt

Possible related to #5113 but was directed to create new issuel.

[zadjii-msft]

Does this repro with an ssh from within WSL as well, or does it only repro with ssh.exe?

[ghost]

I've seen it directly in WSL, SSH from inside WSL, and most consistently from ssh.exe to a Linux host. The first two can be hit or miss in reproducing the bug. Even with the most consistent method I posted above it doesn't happen every time. But I did find that in a instance of the Windows Terminal where it happens it will happen consistently but closing out completely and relaunching can sometimes fix the behavior.

[garyo]

Happens to me too. Inconsistent; I can't detect a pattern so far.

[zadjii-msft]

Alright well, I could have sworn there was a dupe for the newer version of this, but I cannot find it. This is now the thread. I'm gonna link this up to #5800.

Unfortunately without a consistent repro I'm not sure how we'll be able to investigate this, but hopefully by the end of #5800, then this will just go away on it's own.

[vadimkantorov]

Also hit by this (in context of copy-pasting large commands):

[zadjii-msft]

@vadimkantorov do you have a consistent repro for how to get the Terminal into this state?

[vadimkantorov]

Unfortunately not. But I have this tab opened. If you can suggest how to dump some diagnostics info without restarting terminal, I can do that

[zadjii-msft]

@vadimkantorov Unfortunately there's not really a good way to do that with the Terminal at the moment. If we did have a dump of both the windowsterminal.exe and the openconsole.exe (for that tab), we could maybe try to draw some conclusions about what was wrong based off the state in the Terminal and the VtEngine, but there might not a good way to investigate without a live repro.

There's also a good number of changes coming to this area as a part of #5800, so a part of me expects this issue to either go away as a part of that, or be much easier to repro.

[vadimkantorov]

I can do a memory dump. I'll send it to you! Thanks for looking into this.

[atwok]

Running emacs in terminal mode over SSH. I see this every time cut and paste into a split windows.

Is this the same issue?

[ivanpu]

Maybe will be helpful for investigation:
I now had this with a tab split into 4 panes (2 horizontally, then each split into 2 vertically), and wanted to copy-paste a command from one pane (using copyOnSelect) to another (using right click).
It has obviously inserted a spurious new line character in the command:

Version: 1.7.1033.0

Edit: the shell is Cmder in all panes

[inphobia]

i'm not sure if this is related, but i can reproduce the line break thing quite easily.

let's get the details out of the way first

Platform ServicePack Version      VersionString
-------- ----------- -------      -------------
 Win32NT             10.0.19043.0 Microsoft Windows NT 10.0.19043.0

Windows Terminal
Version: 1.9.1942.0
(also tested & reproduced with 1.10.1933.0)

what is is exactly i can reproduce

pasting anything over 500 characters (found this number in another issue, but seems to be correct) will make the pasted text remove/add new lines and paste blocks randomly through each other. the exact location where these incorrect characters get inserted seems mostly random. pasting the same text in the same vi session sometimes appear at the same place, sometimes not. this problem only manifest when pasting into a cygwin terminal, when you copy text from a cygwin window running in windows terminal and paste in another application (either a wsl2 instance in another tab of the same terminal, or in notepad) the text pastes just fine.

there are/were 2 profiles i could reproduce this:

1. by using zsh from cygwin in windows terminal (or bash). just updated everthing cygwin related to be sure but have seen the issue for several months.
2. when using psreadline 2.0.0beta2 with powershell 5.1.19041.1023 (no longer have the powershell error message, seem to recall it saying something like System.ArgumentOutOfRangeException: The value must be greater than or equal to zero and less than the console's buffer size in that dimension. and then giving a negative value for either cursor position or something)

i could not (or can't anymore with the versions i'm running now) reproduce this with:

1. debian or opensuse 15.2 running from windows terminal - wsl2
2. opensuse 15.0 running from windows terminal - wsl1
3. powershell 5.1 with psreadline 2.1 running from windows terminal
4. zsh/vi running from cygwin's native mintty.exe console

how to reproduce

in a cygwin tab paste any text longer as 500 chars... that's it

• TERM setting doesn't matter, tried several options
• editor doesn't seem to matter. have it in vi, nano, and even with cat <<EOF
• ssh to a remote system from the cygwin tab does not fix the issue, same paste problems will persist.
• can be reproduced with default windows terminal settings (well, you'll need to add a cygwin profile)
• cygwin's zsh must be started from windows terminal. everything works fine when running from mintty.
• running git for windows might also have this issue since it uses cygwin underneath.

the following did not matter

• copying/pasting in the same cygwin tab. when you paste, problem manifests.
• copying in 1 cygwin tab, pasting in a second tab. when you paste, problem manifests.
• copying from notepad. when you paste, problem manifests.
• copy method: mouse select, ctrl+c, whatever. when you paste, problem manifests.
• paste method: shift+insert, right mouse button or paste from the command palette. when you paste, problem manifests.
• when copying with mouse, mouse position doesn't matter. you can either nicely start from the beginning of the line and stop at the last character several lines down, or just start & stop wherever you want. when you paste, problem manifests.

why not just stop using zsh/cygwin 😄 ?

• habbit, have been using it for 10+years
• powershell is great, but i like zsh better for cli work.
• wsl2 doesn't play nice with windows always on vpn for my setup
• and a host of other semi technical/semi personal reasons

and now for some files.

1. input text. copied from notepad via ctrl+c

                  GNU GENERAL PUBLIC LICENSE
                     Version 3, 29 June 2007

Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.

                          Preamble

The GNU General Public License is a free, copyleft license for
software and other kinds of works.

The licenses for most software and other practical works are designed
to take away your freedom to share and change the works.  By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.  We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors.  You can apply it to
your programs, too.

When we speak of free software, we are referring to freedom, not
price.  Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.

To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights.  Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.

For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received.  You must make sure that they, too, receive
or can get the source code.  And you must show them these terms so they
know their rights.

Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.

For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software.  For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.

2. pasted text in windows terminal. pasted in vi using shift+insert
   

3. my settings.json (but also tried with a non modified - other as adding the cygwin profile - settings file in terminal 1.10.1933.0)
   settings.json.txt

[ElvenSpellmaker]

by using zsh from cygwin in windows terminal (or bash). just updated everthing cygwin related to be sure but have seen the issue for several months.

As a long-time Cygwin user myself I find it invaluable compared to WSL2 (although I do have it) and still use it as my main terminal. I don't use Terminal as I prefer mintty.

Any time I call out to a Windows programme (and so uses ConPTY) I get this same issue, it's a nightmare.

It's ConPTY at fault here as far as I can tell.

[zadjii-msft]

line breaks are introduced, but they shouldn't be

• When Copying, Line Wrapped Text Is Inconsistently Broken Into Multiple Lines #6901 (this thread)
• Copy of a single wrapped line from bash introduces new-lines #14035
• Extra newline when copying line with wide glyph on the edge #11739

multiple lines get concatenated together when they should be separate

• Line breaks are ignored inconsistently when copying text in tmux copy mode #8976
• Copying text from vim.exe in Windows Terminal removes newlines #9933
• Console.WriteLine occasionally doesn't break the line #15455

other / unrelated

• Rendering errors in tmux split panes #6987
  • The illusive "something causes tmux panes to get overwritten into"
• git bash for windows in terminal does not break lines correctly #9359
  • mingw is outputting lines separated by \n instead of \r\n?
• pasting anything over 500 characters into a cygwin tab in terminal is wack: When Copying, Line Wrapped Text Is Inconsistently Broken Into Multiple Lines #6901 (comment)
• Mixed language input breaks line wrap #11459
  • Yea this isn't related
• UIA in conhost: incorrect word boundaries when text is selected #11809
  • Unrelated. UIA gets the wrong boundaries in conhost, for a wrapped line. That's weird, but not conpty related.
• Text Wrapping with ConPTY  #405 (comment)

Related(?) PRs

• Make Conpty emit wrapped lines as actually wrapped lines #4415
• We've been trying to reach you about your WriteCharsLegacy's extended Emoji support #15567
• Ensure that delayed EOL wrap is reset when necessary #14936
• Merge the LineFeed functionality into AdaptDispatch #14874

[mailinglists35]

probably worth mentioning the workaround of holding shift while selecting.

[gyorokpeter]

I can reproduce, but in my case shift+select is not a workaround as it causes every newline to be removed, while what I'm copying is an ASCII rendered table that gets broken up by the incorrectly introduced newlines. I want to copy the table such that it preserves its table format.

I'm attaching a file demonstrating what the layout of the table looks like.
table.txt
Also I set my terminal settings to: 120 columns, (30 rows), font size 10. typeing the file will result in this:

And if I copy this output, the extra newlines are preserved. What I'm expecting is that only the newlines that were in the output are preserved, so the copied version matches that in the file.

For comparison in the legacy console app I can do this:

And if I copy this with Ctrl+A, the extra newlines are not copied.

[fireattack]

I didn't have this issue when using the actual Windows Terminal, however I have this issue when using the integrated terminal in VS Code: microsoft/vscode#166247

The dev there said it is/was a WT bug since they use conpty, which is part of WT. They encouraged me to report it here, so I guess it's worth a mention.

[zadjii-msft]

Huh. Just had this thread open and was curious if the #15567, #14936, #14874 work cleaned this up at all. Then noticed this interesting thing:

The first paste into notepad, from cat, looks like it maintained the wrap. But more didn't. And that's highly reproducible. Perhaps there is something about the way that more outputs the lines to the buffer thats... manually breaking them?

Heck, that repros in gnome-terminal too:

---

And the type table.txt, then copy/paste it seems fixed:

So I think 1.18 fixed a bulk of "there's line breaks where there shouldn't be" issues.

---

That brings us to the rest of the stuff linked in #6901 (comment):

• multiple lines get concatenated together when they should be separate
• pasting anything over 500 characters into a cygwin tab in terminal is wack

[ElvenSpellmaker]

Hi @DHowett, a long stretch here but I think this has been a problem in Cygwin for some time and it's great to see a fix finally in.

Cygwin uses conhost/conPTY for improving interfacing Windows native binaries to Cygwin and so the fixes here would be great to be seen there too.

How does the conhost release process work if you'd not mind me asking. Is this something that's just wait for Windows Update to pick up a change to conhost and it'll magically fix itself once that update is out?