⎕ucs 10 uglies still not fixed in ]link for character data

[bernecky]

I conflated two distinct issues into one. Mea culpa. Here is one of them:

And then, as they say in Russia, it got worse:
I am trying to use ]link to keep things under git. Here is
what a similar file looks like under bash, when maintained
by ]link:

cat FRGquotequad.apla 
('% Code fragment for quote-quad input and probably others',(⎕UCS 10 10),'%Fragment qdd XX1 X X C .',(⎕UCS 10),'inline char[.] $FNAME()',(⎕UCS 10),'{ /* Quote-quad input */',(⎕UCS 10),'  arbitrarylimit = 100000;',(⎕UCS 10),'  z = drop([-1], tochar(scanl(arbitrarylimit)));',(⎕UCS 10),'  return( z);',(⎕UCS 10),'}',(⎕UCS 10 10))

The real data looks like this:

apex@boa:~/apexgit/Source/Root/sac/fragments$ cat quotequad.frg 
% Code fragment for quote-quad input and probably others
%Fragment qdd XX1 X X C .
inline char[.] $FNAME()
{ /* Quote-quad input */
  arbitrarylimit = 100000;
  z = drop([-1], tochar(scanl(arbitrarylimit)));
  return( z);
}

This is on a linux box, running the latest ]link, from the Ides of March: e626cb7
The treatment of LF (NL) makes it effectively impossible to use a normal text editor, or even
a grep that wants to know about NL, to operate on files resulting from ]link. They are also ugly as sin.
Uglier, actually.

[abrudz]

Thanks for bringing this up, @bernecky
Considering that one cannot rely on line endings being preserved across platforms, and that the character vector specifically contains LFs (not just "generic line endings"), do you have any suggestions as to how Link could store such a value while keeping it distinct from say a character vector that is identical other than with CR in place of the the LFs?

[abrudz]

Maybe the result type could be stored as a tag, similar to how case-coding works?

Example file name	Explanation
mytext.1.apla	vector of vectors
mytext.2.apla	matrix
mytext.10.apla	LF-separated
mytext.13.apla	CR-separated
mytext.13-10.apla	CRLF-separated
mytext.133.apla	NEL-separated
mytext.16.apla	hex dump
mytext.apla	APL Array Notation

[e9gille]

That might be overkill. How about, just like acre, use 3 distinct extensions and where it matters (vector of vectors and matrix) deduce the line breaks as ⎕NGET does?

[e9gille]

Remember that if collaborating on a project that stores a vector of vectors on file, when switching platforms default git configuration is to translate EOL on checkout. So storing it with LF on linux will result in CR-LF on windows on checkout.

[abrudz]

What wouldn't have helped in this case then. ⎕NGET uses variable line breaks on the file side, not on the APL side, where it always uses LF.

[abrudz]

Remember that if collaborating on a project that stores a vector of vectors on file, when switching platforms default git configuration is to translate EOL on checkout. So storing it with LF on linux will result in CR-LF on windows on checkout.

That is exactly the problem. We can't use the file's actual content to deduce what the APL array should look like, so this must be stored in the file name or in the extension.

Personally, I don't like Acre's extensions because .charstring and .charlist aren't explicit enough. APL doesn't have strings as such, and .charlist sounds like it is a list (i.e. a vector) of chars (i.e of characters), but it isn't; it is actually a vector of character vectors!

[mkromberg]

I'm slowly drifting in the direction of adding extensions to declare a particular kind of transformation on serialisation and deserialisation. I wonder whether anyone really uses "charmat"s anymore, but a .aplcv (char vec) and .aplvv (vector of vectors) or perhaps .strings extension to make it a little APL-agnostic seem to make sense as something we could add as a standard component in the next round of Link development. I think it would be very unfortunate to do anything other than "normalise" new lines to ⎕UCS 10, if that doesn't work well in current APL we should work on how APL handles it.

[abrudz]

The Editor seems to handle use whichever of 13 and 10 is found first, as the separator for that variable. The other becomes a "normal" inline character.

Two issues:

1. It defaults to 13 if neither 13 nor 10 is found.
2. It doesn't recognise 13 10 as a single line break.

Also, I'm a bit worried about the proliferation of file extensions. I'd rather that the extension indicates the type of item (afonci: array function operator namespace class interface) rather than the internal structure of the item (nested, vector, matrix). Similarly, we don't use a separate extension for one-liner dfns and multi-line dfns, even if they are quite distinct in behaviour on the APL side.

[mkromberg]

I would maintain that the differences between different types of dfns is trivial compared to the difference between a simple vector with embedded newlines and a vector of vectors. The only alternative to using different extensions seems to be some kind of extension to the array notation, which allows you to directly write text variables without any decorators - is that possible?

[e9gille]

@bernecky 's input on his use cases is welcome here and @PhilLast might have some more insight into the issues encountered with implementing the .char* extensions in Acre.

My gut feeling is that the common scenarios of working with plain text files outside the APL editor that in APL materialize as either matrix or vector of vectors or as a simple string would be good. If you want a matrix or vector of vectors in APL where items (cells in matrix or lines in vov) contain embedded EOL characters then you have to resort to array notation.

[e9gille]

and why not secondary extension?
file.cm.txt for matrix
file.vv.txt for vector of vectors
file.txt for string

[mkromberg]

Sounds good, my real opposition to the secondary extension is to use it to provide information about what sort of newlines you want in the string case.

[abrudz]

Are we proposing that Link should see .txt as code? That might have interesting effects. How about:

• file.cm.apla for character matrix
• file.vv.apla for vector of vectors
• file.cv.apla for character vector

[mkromberg]

Yup, that makes sense. Link is under strict orders to ignore anything that is not "APL source", so that you are not forced to separate the APL source from other artefacts that the application needs. An actual project manager component would be at liberty to decide such things, but Link should, IMHO, not.

[PhilLast]

Without offering any opinion as to which way Link should be improved I'll just do what Gil is asking. We realised immediately that files containing --------------- ['phil''s' 'char' 'matrix'] ------------ ('phil''s' 'char' 'list') ------- and -------- 'phil''s/rtext/rstring' ----------- would be unacceptable given that array notation is supposed to be an improvement and the data is explicit requiring only a couple of APL primitives to turn the content of an ordinary text file into any of them. Hence our three additional file extensions: .charlist, .charmat and .charstring. The outcome for strings of mixed delimiters is as follows cr nl←⎕UCS 13 10 str←'phil''s',cr,nl,'char',cr,'string' file str.charstring contains ------------- phil's \nchar string ---------- str←'phil''s',nl,cr,'char',cr,'string' file str.charstring contains ------------- phil's\n char\nstring ---------- We use ⎕ED, ⎕NPUT and ⎕NGET so the default delimiter is left to Dyalog to dictate but acre uses CR as does Dyalog. The round trip is successful. Adding "\r" in a .charstring file results in an additional break in the string. Beyond this my research is lacking as it's seemed to work as expected. yours Phil

…

On 2021-03-16 09:25, Gilgamesh Athoraya wrote: @bernecky <https://github.com/bernecky> 's input on his use cases is welcome here and @PhilLast <https://github.com/PhilLast> might have some more insight into the issues encountered with implementing the .char* extensions in Acre. My gut feeling is that the common scenarios of working with plain text files outside the APL editor that in APL materialize as either matrix or vector of vectors or as a simple string would be good. If you want a matrix or vector of vectors in APL where items (cells in matrix or lines in vov) contain embedded EOL characters then you have to resort to array notation. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#240 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIGXUEEQBCQIPUCNY22Z7Q3TD4PZFANCNFSM4ZHI2KCA>.

[e9gille]

thanks Phil

[PhilLast]

Erratum: The second example should have been:

…

---------- str←'phil''s',nl,cr,'char',nl,'string' file str.charstring contains ------------- phil's\n char\nstring ---------- while

---------- str←'phil''s',nl,cr,'char',cr,'string' file str.charstring /would/ contain ------------- phil's\n char string ---------- P

On 2021-03-16 11:36, Phil Last wrote: Without offering any opinion as to which way Link should be improved I'll just do what Gil is asking. We realised immediately that files containing --------------- ['phil''s' 'char' 'matrix'] ------------ ('phil''s' 'char' 'list') ------- and -------- 'phil''s/rtext/rstring' ----------- would be unacceptable given that array notation is supposed to be an improvement and the data is explicit requiring only a couple of APL primitives to turn the content of an ordinary text file into any of them. Hence our three additional file extensions: .charlist, .charmat and .charstring. The outcome for strings of mixed delimiters is as follows     cr nl←⎕UCS 13 10     str←'phil''s',cr,nl,'char',cr,'string' file str.charstring contains ------------- phil's \nchar string ----------     str←'phil''s',nl,cr,'char',cr,'string' file str.charstring contains ------------- phil's\n char\nstring ---------- We use ⎕ED, ⎕NPUT and ⎕NGET so the default delimiter is left to Dyalog to dictate but acre uses CR as does Dyalog. The round trip is successful. Adding "\r" in a .charstring file results in an additional break in the string. Beyond this my research is lacking as it's seemed to work as expected. yours Phil On 2021-03-16 09:25, Gilgamesh Athoraya wrote: > @bernecky <https://github.com/bernecky> 's input on his use cases is > welcome here and @PhilLast <https://github.com/PhilLast> might have > some more insight into the issues encountered with implementing the > .char* extensions in Acre. > > My gut feeling is that the common scenarios of working with plain text > files outside the APL editor that in APL materialize as either matrix > or vector of vectors or as a simple string would be good. If you want > a matrix or vector of vectors in APL where items (cells in matrix or > lines in vov) contain embedded EOL characters then you have to resort > to array notation. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#240 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AIGXUEEQBCQIPUCNY22Z7Q3TD4PZFANCNFSM4ZHI2KCA>. > >

[bernecky]

I think my line(s?) of thinking aligns with those of Gilgamesh. However, let's back up a half-century or so.
Unixish systems used LF (Wir sprechen ASCII.) as a line separator or new line character.
IBM systems, and APL, in particular, used a unique new line character, neither fish nor fowl.
2741style terminals (electric typewriters, basically), performed [CR,LF] (as one character), space, and backspace,
as their only non-printing(ish) functions.
Then, BillWare(tm) came along and imitated the teletype, which had distinct mechanisms for cursor-to-left-margin(CR)
and cursor-down-one-line(LF). Rather than using a single character for new line in text files, BillWare(tm) required
text files to contain [CR,LF] to delimit lines, thereby torpedoing any ability to copy/move files among different
operating systems in a portable fashion. That attack seems to have worked.

Now, we have the above mess, and I think it is safe to assume that those eggs will remain broken.

So, let's look at the important design goals (Mine!) of ]link in that light. What do I want to use
]link for, and how do I want to use it?

1. I want to maintain APL source code functions in a host system file-per-function form. This enables me to use, with no headaches, tools such as git, grep, vi, wc, etc. ]link does most, or all, of this for me quite nicely.
2. I want to maintain APL source code variables the same way. I can not comment on how ]link handles numeric or nested arrays, but it handles text vectors very poorly, hence this issue.
3. Host system files, by and large, do not have any concept of arrays. All files are vectors. End of story.
4. How important is inter-system (Unix-Windoze) communication of files generated by ]link? For me, it is not important at all, because all my systems are linux-based. That is the case today, but obviously, if I want someone running BillWare(tm) to be able to run the github-maintained apex compiler, problems could arise.

The treatment of ]link on character vector variables was designed ONLY for use under BillWare(tm). Unix
users are tolerated, rather than supported, by ]link. I am unable to most of my Unix tools with files
generated from character vectors, because, e.g., cat produces results that are unreadable by humans;
vi text is unreadable by humans because a multi-line (unix) variable becomes a single line in vi;
wc tells me that all my variables are one line long; grep and friends now produce wrong answers
when I want to make searches based on end-of-line.

So, since ]link produces rubbish files from character variables, perhaps we should look at
my commonest use cases:

1. I rarely move files between different end-of-line architectures.
2. Every day, I need to use the above sorts of unix tools in my work, without headaches such as those created by the present incarnation of ]link.
3. When I do have to move (e.g., compiler source code) files between architectures, there are many factors involved in the task, including things including file names, path delimiter, and end-of-line character(s).

I suspect that most users are in the same boat as I am. So, perhaps ]link should
be OS-centric in how it treats character vectors, and leave my LFs as LFs. It does this perfectly
well for functions, and the non-APL problems described above do not arise.

This would result in host-system files that are completely amenable to analysis
and manipulation by non-APL apps, and those rare times when
inter-end-of-line "architecture" (We know that Bad Design is never Architecture...)
movements are required, a case by case analysis
could be performed, for those even rarer cases when control characters
are embedded in text vectors.

This change would keep people running under either architecture utterly happy.
The current situation liketh me not.

[abrudz]

The treatment of ]link on character vector variables was designed ONLY for use under BillWare(tm).

I think you've misunderstood something. Currently, all character vectors are stored using ',(⎕UCS nn),' no matter what line endings appear in them, with no special treatment afforded CR or CRLF. Indeed, Link behaves identically across all platforms.

[bernecky]

Ah, I see. My apologies for using the term "designed".

Bob