Got tabs replaced by literal '\t' when saving

[jkyeung]

Describe the bug
When saving tab characters back to the IBM i, the string '\t' (that is, the two-character string composed of a literal backslash and a literal 't') get written into the source member.

To Reproduce

1. Open a source member in VS Code.
2. Set indentation to "Indent Using Tabs".
3. Insert a tab.
4. Save.

Expected behavior
A tab character in the appropriate CCSID gets written to the source member.

Screenshots

SEU before editing:

Open in VS Code:

Setting indent mode:

Code containing tab:

SEU after saving:

Environment (please complete the following information):

• Code for IBM i v1.6.8
• IBM i OS version 7.4

Additional context
I realize you would rarely want actual tabs in your code, especially if it will also be worked on with SEU. But if the tabs are already there in the source member on the IBM i, VS Code opens them as tabs, and they need to make the round-trip intact.

[worksofliam]

Huh, fun bug. Likely something to do with VS Code and it writing files. I will look into it. Thanks!

[dbilyeu67]

So I was going to make a new issue but I decided to look through the open issues and came across this. I have been experiencing this as well in certain situations. I have a SQL source file that had a tab character in it most likely from when it was made. When I pull it down from production to work on it using VS Code the characters don't show up until I make a change and save it. When I reopen it the \t is put in place of the tab characters.

So I started testing to see where and how this is getting put in. Came to the conclusion that when I disable the Source dates option, VS Code will start translating the tab characters correctly and replace it with a space or however you may have your tab indentation setup. When I re-enable Source dates it creates the \t again.

Now I have no idea why this is the case because what would source dates have to do with translating tab characters but here we are.

If you want me to make a new issue I totally can but I just figured I would throw this comment on this open issue.

[jkyeung]

Excellent detective work! I'm glad you found this issue; commenting here was absolutely the best thing to do. I'm sure your new information will help someone track down the bug.

[chrjorgensen]

It's the Source date option in combination with the Source date tracking mode:

When set to diff, all tabs are converted to \t...!

You can easily replicate this by running the VS Code command Convert Indentation to Tabs... - this will trigger this error when the member is saved.

This is definitely a bug!

[jkyeung]

It's the Source date option in combination with the Source date tracking mode:

When set to diff, all tabs are converted to \t...!

So, does this mean you can find it in the code? Or are you just noting more specifics about the behavior?

[chrjorgensen]

So, does this mean you can find it in the code? Or are you just noting more specifics about the behavior?

I was only doing some extra testing - I didn't check the code. I've been involved in the save operation (particularly regarding hex color codes) and was curious as to what was causing this.

My initial thought about this problem was that having EBCDIC tab converted into \t after conversion to UTF-8 (VS Code) and back to EBCDIC (member) should not be categorized as an error... since non-displayed hex values in source code is a pain in the neck and should be avoided, and this made the non-display code visible and apparent and fixable!

But seeing that tabs inserted by VS Code was also converted into \t changed my mind - because this is of course not the intention and is of course a bug.

But at least now we know in which specific part of the code where to look for the bug...

[steph-beneschan-256]

It looks like the issue is with the escapeString function in extendedContent.ts; is that right?

I found that the function is executed for each line of a source member when the member is saved, but only if Enable Source Dates is turned on and the date tracking mode is set to diff. The function replaces each instance of \t with \\t; I tried changing the function to replace \t with 0x05 instead (since this is evidently the tab character in EBCDIC), and after preliminary testing it appears to work as expected.