Rewinding this repo and encouraging community

[gordonwoodhull]

I work with @lkoutsofios at AT&T Labs Research and volunteered to see if I could help restore the repo to a state the community is comfortable with. This was last discussed in #1464 (comment) and onward.

I am not an expert in ksh or the technical issues, and I don't know the entire history, but my understanding is that

1. ksh93u+ is the final official release from AT&T. This can be found in the branch 2012-08-01-master
2. A later version, ksh93v-, was contributed by the original authors after they left AT&T, but this version is not considered stable. This can be found in the tag ksh93v
3. @krader1961 has been maintaining the master branch for some years. He has tried to streamline the codebase (and only support ksh, not the rest of ast), but this has caused compatibility issues.

The original authors of ksh have all left AT&T. We still have plenty of users inside the company, but no one who can maintain and manage the project.

My proposal is:

• Move the current master to a branch "ksh2020" (or whatever is appropriate).
• Clearly mark README.md and CHANGELOG.md on that branch as unmaintained, point to segfault with extended globs #1464 and this issue for further info
• Reset the master branch to 2012-08-01-master
• Remove external collaborators from the repo. Since no one in AT&T is maintaining this, it doesn't make sense to have this fork be the active one. Git and Github allow anyone to take the code in whatever direction they want, in their own space.
• Clean up README.md to reflect the current state: "This repo contains the AT&T ksh93u+ and experimental ksh93v- versions of AST. There may be later forks of AST and ksh out there."
• Change CONTRIBUTING.md and issue template to be clear we are not actively maintaining this fork and do not promise to review anything.
• Shorten the repo description to "AST - AT&T Software Technology" (Streamline this repository's description #1441)

After this is complete, we at AT&T may contribute patches to ksh93u+ in the master branch. We probably won't review or accept contributions unless they are trivial (build issues, etc.) Eventually, we will archive this repo.

Currently I don't think anyone in the company plans to create a repo just for ksh. However, I think this is fairly straightforward and can be taken up by the community.

If anyone wants to create a community fork, I'd suggest creating a new GitHub Organization. This is free for open source projects. The bigger commitment is time, making contribution and maintenance guidelines clear, possibly creating a code of conduct, etc. Of course if anyone wants to fork this repo to an existing organization, they are welcome to do that too.

[jghub]

@gorodnwoodhull: my most recent comments are here: #1464 (comment)

sorry for overlooking this (better) place. will take note from now on...

[gordonwoodhull]

I went ahead with removing the external collaborators and fixing the description. I'll wait to see if there is further discussion about the other points before continuing with those.

[lkoutsofios]

thanks @gordonwoodhull for helping out.

my original motivation for going along with @krader1961's proposal was to keep at least KSH alive since no one else was volunteering to work on the AST project as a whole. but I think we were all clear that backwards compatibility was mandatory. KSH's strongest advantage over other SH/KSH clones was its programming interface. if the new version breaks old scripts, there's no point in using it. it seems from the recent issues on this repo that this isn't working out. so @gordonwoodhull's changes seem like the way to go. we'll see what happens with KSH. I do have some fixes for the u+ version that should get it to build and work on all recent distros of redhat, ubuntu, and opensuse, that I can push out. unfortunately, I don't have the time to be more involved.

[DavidMorano]

Just some friendly observational comments:

I have tried to follow both the recent discussion about rewinding the repository, et cetera, and the progress on this project since @siteshwar and then @krader1961 started making major contributions back in 2017. For the record, my own experience w/ KSH dates from its very (albeit public) inception, back in 1983 (I was at Bell Labs at the time and following). I have used KSH almost entirely exclusively and continuously since that time to the present (including very recently on ksh2020). Some items:

• this code is very complicated and somewhat messy by the best modern coding practices (that may be something of a gross understatement); it is NOT suited for the casual programmer -- or team of programmers -- to dive into; despite people possibly feeling otherwise, I think that @krader1961 is-was fairly competent as a programmer and developer; where @krader1961 seemed to possibly lack something was his understanding of the various historical customer uses of KSH over the last 37 years -- being much more intensive and far-reaching than he seemed to realize; I will not dwell on this now, but KSH was Perl before Perl was, and it was Python before Python was; it had substantial enterprise applications written in it before the end of the 1980s (and the first book only came out in 1988)

• the code creates a program that is substantially defined by three standards: 1) POSIX, 2) the KSH book (by David Korn), and 3) the existing behavior of the ksh93u+ branch; this is not something to be trifled with lightly; extreme care needs to be exercised to not break any existing documented or expected behavior (this may be an area were @krader1961 might have appeared to not be as sensitive as people would have expected)

• despite some possible (arguably relatively minor) personality issues, @krader1961 has put an exceptionally enormous amount of work into fixing problems that even existed in ksh93u+ that were not always readily observed or complained about; to replicate @krader1961's work starting from scratch at the ksh93u+ branch will be extremely non-trivial (please, @siteshwar give your feeling on this if you would)

• I am aware that even ksh93u+ had severe bugs and problems that needed to be eventually fixed; many of these did not manifest readily for most users, but they needed to be addressed eventually none-the-less; these are things such as strays writes, buffer overflows, and the like; there were a variety of minor (and not so minor) problems also that had been in there for about 20+ years (also not readily complained about by most regular uses); @krader1961 had made a very substantial start at getting some of the more serious of these (the memory related stuff) fixed (with the help of, among other things, ASAN and Valgrind); starting again from ksh93u+ means a huge amount of dog-work in fixing up again what @krader1961 has already done

• not all distros are equal! some distros are more important and have more say in the world than others! sadly for some, the Red-Hat distro is more important now-a-days than pretty much all other distros combined (yes my opinion!); and I am not one who even likes Linux derived distros in the first place! ; Linux itself is very much inferior in many ways to older non-Linux distros (yes that is also my opinion); so the short of it is, that I think that @siteshwar should have more say in how KSH proceeds into the future than all the rest of us (distro maintainers or not) combined! I am sorry, but that is what I feel should happen; @siteshwar is not a dumb guy; he knows 'the score"; he knows full well that maintaining KSH is like managing fire in your hands; he works for Red Hat!!! Red Hat understands managing real (hard core, porn) enterprise (and other) customers and their requirements and demands; I would ask that more deference be given to @siteshwar's thoughts on this whole matter going forward; no joke

• my fear is that if the ksh2020 branch is essentially abandoned in favor of starting from ksh93u+ again, a huge amount of time and work will have been forfeited; I want to hear from @siteshwar more of this point; this kind of decision should not be taken lightly

• even separating out KSH from the rest of the AST code was not a totally trivial endeavor; @siteshwar and @krader1961 did this w/ ksh2020 at no little (no little) effort in itself; although I had some trepidation at the time, I feel that this was a useful and important thing to do to modernize the KSH code base in particular; this is just one example of something that would have to be done again if starting from scratch

• my own preference going forward would be something along the lines of the following: 1) bring the ksh93u+ branch to a new state (calling it something else, ksh93uu+ whatever) so that it can be easily compiled by the community (easily, maybe using make, autotools, CMake, whatever) so that distros have a fairly reliable and readily accessible base to port from (in the near term), but then 2) continue w/ the ksh2020 branch and make it more conforming to historical and expected behavior; ksh2020 would be the base for future -- possibly feature enhanced -- versions going forward

• and finally, for those who think that just using ksh93u+ is some sort of magical cookie to the golden land, it is very likely that making any fixes to ksh93+ (which will likely be required) will break some conforming behavior in unexpected ways; this is likely part of what happened with ksh2020 -- the act of fixing some previous bad stuff, exposed some bad stuff that was still in there already from the past, but not previously readily observable by most customers; this happens all of the time with old code, but to get through it one has to often just confront bug after bug and plow ahead through the hard times; for myself, I feel that ksh2020 has started down that path (as hard as it may be); starting fresh from ksh93u+ may never get past the hump of fixing some of its worst bugs lurking within it

• KSH is a complicated and still -- in many quarters -- a mission critical program; in my opinion, it needs the attention and contributions from the entire world to focus on a single main code base branch in order for it to flourish into the future, rather than just staying "useable" at the ksh93u+ level; ksh93u+ had (has) many bugs that need to be eventually fixed if there is any hope for real future feature enhancements to likely work; the code base needs to be modernized also (not a trifle in itself)

• just my two cents on this whole thing!

[jelmd]

@gordonwoodhull, I think you got it right and your proposal is good.

However, some nits:

* [ ]  Move the current master to a branch "ksh2020" (or whatever is appropriate).

Yes. But to avoid confusion [1] [2], that people use it as an official or stable version (actually what happened with version v-), README.md as well as CHANGELOG.md in this branch should make it very clear, that this is an experimental branch (e.g. by adding an appropriate level 1 header at the top) and the mentioned articles refer to an experimental release (or that there is no ksh2020.*, etc.).

* [ ]  Rewind the master branch to 2012-08-01-master

Adding @lkoutsofios patches is a good idea - thanks @lkoutsofios for spending your time! Direct merge, separate *.patch files or PR, whatever you like is probably ok.

* [ ]  Clean up README.md to reflect the current state, and point to any active forks from there. If desired, transfer relevant issues to child forks.

If no-one at AT&T has time to maintain it, or prefers one fork over another, it should be kept neutral and just mention, that there are forks which are not AT&T driven. I think, if there is a valuable, active fork, people interested in ksh93 will find it by themselves or by getting hints from others...

[gordonwoodhull]

@DavidMorano thanks for your perspective. I don't know the history but I agree in principle with what you're saying.

I will second that I hope ksh2020 starts an organization, perhaps led by @siteshwar and @krader1961, starting from what is currently on master.

[gordonwoodhull]

@jelmd, sure, sounds good - I've edited the tasks above.

[jghub]

@DavidMorano, I appreciate your perspective, especially concerning the "KSH was Perl before Perl etc" aspect and its implications.

regarding what has been achieved (or not...) in ksh2020 I respect your assessment, although it seems too "understanding" in my view. I would readily agree that I cannot really judge the technical difficulties involved when rewriting/rephrasing the C-code . I am in the position of a "typical" KSH user who uses the KornShell language for scripting w/o knowing the C innards of the ksh program .

let's say I cannot build a car but I can drive it and if I drive ksh2020 around the block I notice strange noises from the motor department and assorted malfunctions. it's slower, too... before the overhaul the car was just fine, only had some peculiarities the driver knew (mostly) to avoid. so while the overhaul obviously involved technical skills (at the C-level) beyond my own I can judge that the job has not produced something better but something worse.

I maintain that is the current state of ksh2020. choose any metric you like (except better formatted C-code and "tidiness").

as you explained to me elsewhere (I forgot so far to again thank you for this!) #1449 (comment), you especially value the malloc vs. vmalloc changes for reasons of your special interests although you also agree that vmalloc might be uncritical for the 99.9% crowd not writing their own (multi-threaded) builtins. of course, if it turns out, that indeed replacing vmalloc is clearly better (but for whom and according to what measures would have to be decided) one might consider to do that to 93u+ at some point down the road. I am not sure that there is clear evidence that this is worth it -- but not for me to decide....

and it sure would have been the responsibility of @krader1961 and @siteshwar to listen to people like you and many others (I was a latecomer in the queue of people raising concerns...) from the start and adjust their actions how to deal with the code, the features etc. I feel they (at least krader) had the wrong agenda. and the decision to start from the 93v- beta is not comprehensible to me, given the circumstances (known buggy beta state, no advice from dgk etc.).

I do not know of any "problem/bug driven" TODO list with which ksh2020 was started. there was the "we must first overhaul the code to adhere to modern standards" and the fuzzy "keep ksh relevant" approach and that without any adequate understanding of either the code or KornShell. the issue history is there for everybody to go through (it is not that extremely long): a clear story evolves of misconceptions, false claims and -- and that was the central problem throughout -- an incomprehensible rigidity and refusal to adjust course and actions in the small or in the big issues on the side of krader.

so I believe your rather benevolent description of krader's "possible (arguably relative minor) personality issues" is objectively not accurate (and this is not irrelevant since it crucially has determined what has happened in ksh2020 -- it could have been a different project -- and is also crucial for deciding how to proceed from here).

for me this implies that ksh2020 should (no: must!), indeed pursue its goals independently of ksh93. I also insist that terminology-wise there is need for a strict and hopefully agreed-upon "branding". it was proposed to call ksh2020 "krsh" for obvious reasons, but I really believe the "ksh2020" name is more sensible and "catchy" and accurate enough, so why not call it that for the next 25 years just like ksh93 still is called that...

altogether I am still not quite sure, what exactly the goals of ksh2020 are: so far it has eliminated most or all of the new features of 93v- since they could not debug those (being, so far, not yet or not fully functional in 93v-) while incompatibilities and regressions and new bugs crept in and performance went southward. maybe, that all can be fixed and at some point ksh2020 is better than a (patched) 93u+ but I really do not see that right now.

to reiterate: in my view, ksh93 and ksh2020 development should not proceed in a common repository. this can not work out. what could work out in my view is a friendly coexistence (in the long run, at least, when potential hard feelings have waned) and, hopefully, bug fix exchange (the latter for the time being could very well be unidirectional from ksh2020 to ksh93u+ for the rather few cases (AFAICS) where ksh2020 has fixed things that apply to 93u+ and not only to 93v-).

I am looking forward to implementation of the strategy outlined by @gordonwoodhull: even if ksh2020 could proof at some point to be the "future", I believe making ksh93u+ easily available again for everybody ASAP is much more important right now. this requires separate projects as the history of the ksh2020 project proofs beyond reasonable doubt in my view.

[gordonwoodhull]

I clarified that we won't immediately archive the repo, and we'll contribute what patches we have, but we probably won't accept contributions unless they are trivial.

I'll proceed with the plan in the next week.

[DavidMorano]

@jelmd
I think that we agree (and most everyone agrees) with the growing consensus around the following:

• that a new workable version of KSH be made out of the ksh93u+ branch as its base, possibly replacing NMAKE w/ something else; this would serve the needs of distros that need a very reliable (as much as can be expected given everything) and behaviorally compliant KSH

• further, I have no objection to VMALLOC staying in the ksh93u+ follow-on product; if I ever want to use this new KSH product for myself, I will just replace VMALLOC w/ a regular system one as I did in the past

• neither do I take any issue w/ trying to bring the new (ksh93u+ derived) product up-to-date w/ any patches or fixes that can readily be put into it (the issue here is the time to do this at the level that ksh2020 has already achieved)

• I do not want to waste ksh2020, and neither do @krader1961 and @siteshwar as far as I would guess; but it seems that the consensus is that they open a new unique repo for its development, separate from this ATT/AST repo

• my guess is that @krader1961 and @siteshwar will likely continue to develop and orient ksh2020 as a possible next generally-released KSH also, and I encourage that work; but my preference would be that it try to adhere more to the standardized and expected behavior of prior KSHs -- circumscribed as I stated previously around 1) POSIX, 2) the KSH book (Korn), and 3) ksh93u+ behavior

For myself, I admit that I am optimistic that the benefits that @krader1961 has put into ksh2020 (much of it quite subtle, like cleaning up the various NV preprocessor flags mess, to name just one) more than offsets the regressions that people have complained about. But, yes, this is opinion on my part. But based on following many or almost all of @krader1961 commits. I have seen regressions in ksh2020 myself (like in the form of unexpected crashes). But in fairness, there were some unexpected crashes in many of the production KSHs over the last couple decades also. My hope is that, accidentally or not, @krader1961 might have actually fixed some of these (again, perhaps even by accident as it were).

Having a follow-on version of ksh93u+ available for distros (whatever its name is, I do not care about any names really) will take the pressure off of the ksh2020 development, so that they can concentrate on cleaning it up so that it appears to perform at least as reliably and conformally as the old ksh93u+ did (or its follow-on replacement).

I also have to admit that I hope that ksh2020 succeeds totally eventually. I have seen, and sometimes tried to understand, the code in ksh93u+ and it is often quite difficult (and I look at tons of bad code all of the time for my living). I tend to get spooked by and biased against code that has memory buffer overflows and the like, and I very much welcomed @krader1961 and other's initial attempts to clean some or much of it up -- as much as is exposed by bad behavior or by development tools (like ASAN or Valgrind, et cetra). I do not feel that the code base of ksh93u+ is really fit for the future. I would like to see the code come to a place where both fixes and feature enhancements become at some point much easier. But I think that the right path for this (following the project as I have) is with ksh2020, simply due to the amount of cleanup put into it (being perceived as useful or not to the end user). I could be wrong, but we will now see. Neither do I discourage future major fixes or enhancements to the new ksh93u+ branch. Let the race begin, as it were. The end code-builders or end-users can decide for themselves which one is the real future (or not).

[jelmd]

@DavidMorano, I totally understand your concerns. But what we need right now is to get back the trust from vendors and users and therefore the outlined plan is IMHO the right way to go.

FWIW, I'm basically repeating, what has been already said:

At the moment all distros ship a ksh93 package, have certain bugs already fixed with their own patches and the process to build the package, or apply a small patch from time to time is almost a no-brainer. Vendors/developers/users may even have a knowledge base, FAQ, ... wrt. workarounds for existing bugs/corner cases so that even patching is not really needed, as @jghub already pointed out. So on the vendor side there is no need to invest a lot of/any time and they can continue to ship it until a new star appears on the sky, which makes it redundant.

Redundant means for me full compatibility to ksh93u+. E.g. the getopts self documenting as well as the time conversion and formatter feature are essential and unique (skim over acme-ksh scripts to get some more details if you like). Last but not least I do not like to waste a lot of time to find out, which of several dozen man pages actually contains a description of the feature I wanna use (like zsh), and than skim over several pages again to finally find it. ${builtin} --man is such a useful thing, that I do not really care about the size of the ksh93 binary.

Anyway, new stuff like native json support, possibly ksh thread safety, ... would be nice to have, yes. But I think, even changing the build system only would requires a lot of "review" time at least on distro maintainers' side to make sure, that they still ship a stable/usable version. Debian has shown, that this is probably too challenging, decided to just trust the most active members of the repo and bundled something that at least I do not want to see on any of the 100+ zones/containers I need to manage (or others, which use scripts I wrote in the last 20+ years). Therefore, encouraging people to fork and give it a distinguishable name seems to be the best option. If it is good, distros will pick it up pretty fast and do not need to care about compatibility wrt. ksh93* ...

[kdudka]

But what we need right now is to get back the trust from vendors and users and therefore the outlined plan is IMHO the right way to go.

The necessary precondition to get back the trust of Red Hat is that upstream is able to competently and reasonably fast react on reports of important security issues. For example, how quickly would your team be able to fix the following security issue in ksh93u+?

https://access.redhat.com/security/cve/CVE-2019-14868

[siteshwar]

I would ask that more deference be given to @siteshwar's thoughts on this whole matter going forward; no joke

I give up and move on with my life. I made all my efforts with the goodwill to help the community and it turned out that I achieved quite the opposite. I hope that someone else would pick up this project and would make progress in the direction that's more acceptable to the community.

[DavidMorano]

@siteshwar

I give up and move on with my life. ...

I am very sorry to hear your decision on this matter.
I think that you were the best thing to happen to KSH in years!
Having RedHat (as represented by yourself) in the middle of decisions regarding KSH was likely the best thing to happen to KSH since the old haughty days of AT&T (pretty much long gone now).
I feel badly that the general community did not seem to show you more respect (deference).
Thanks very much for the service that you did give to this project over these past years.

All the best with whatever you do going forward!

[jelmd]

...

The necessary precondition to get back the trust of Red Hat is that upstream is able to competently and reasonably fast react on reports of important security issues. For example, how quickly would your team be able to fix the following security issue in ksh93u+?

https://access.redhat.com/security/cve/CVE-2019-14868

@kdudka, if no-one creates an repo issue, it is very questionable, that it get fixed "in time" by the members of the related repo at all - so thanx for bringing it finally (after 4+ month) to our attention. And BTW, silently inserting a very questionable patch, which just swings the big hammer, is ok? Or can you help me? I could not find a related issue in this repo using the keyword security or CVE . Even the commit contains neither the word "security" nor "CVE". Is this really your understanding of open source development?

[marcastel]

I would like to express an "external" (or consumer) point of view here on KSH for today and tomorrow.

Though I have not contributed any code to this GitHub repository, I have (since my #42 comment) maintained my own private build of ksh and a have some insight on its intrinsics -- and for one, I like its build system which, optimised, is better than the gmake/cmakes of the world, or their more recent python or javascript alternatives)

But back to the consumer view of ksh. In the comments I have seen so far we speak about the "community"... and I understand that to be the "community of developers" as opposed to the "community of users".

As a user, I am obviously very concerned with backwards compatibility, performance, and much less so about the build system. But more importantly I am concerned about the POSIX status of AT&T ksh which has landed this shell on all UNIX distributions.

This is what is important to me, as a consumer. I want my ksh to remain in my AIX, macOS or Linux distro "as-is". I don't want it to be called pksh, krsh, ... I want it to remain POSIX certified and to be the time-proofed shell I use for my production environments and secured managed operations.

As probably many "consumers" I am a system administrator who uses Korn shell over other shells because of its portability, its performance, and of its superior programming capabilities -- e.g. object-oriented REST interfaces for managed services using KAML (https://github.com/ISLEcode/KAML)

And as one of these "consumers" I wish we could upgrade modern operating systems from POSIX "sh" to POSIX "ksh". Imagine such a replacement in the GNU build system... ksh in lieu of sh and m4... lean and clean. Obviously a little provocative, only to emphasise the value of ksh by its functionality, its programability and its heritage.

All that to say this: Korn shell should not be just another open source project abandoned to a community of developers, however skilled and passionate these may be. If it is to survive as a POSIX certified shell, it also needs representation (and defence) in USENIX and the like associations. It needs governance and a roadmap aligned to modern operating system roadmaps.

I want ksh to survive as a POSIX shell... even if we are probably a decade late. I fully support (and ready to invest time) in any initiative to make this happen. The consumer view is the bird's eye view of where we want to go. This is the first step. The developer's view, which is how we get this done, is the next step.

I appreciate the initiative to bring governance to this repository. I hope the extended scope proposed here shall be considered.

[kdudka]

@jelmd Embargoed security issues cannot really be reported to a public bug tracker. There needs to be some confidential channel where security issues can be reported and fixes prepared before they are disclosed publicly. When the mentioned security issue was discovered, the confidential channel was @siteshwar + @krader1961. They took care of it code-wise and process-wise. If the new upstream team has some non-public channel where we can safely report embargoed security issues, please let us now. We will be happy to collaborate on them next time.

[jghub]

@marcastel

As a user, I am obviously very concerned with backwards compatibility, performance, and much less so about the build system. But more importantly I am concerned about the POSIX status of AT&T ksh which has landed this shell on all UNIX distributions.

This is what is important to me, as a consumer. I want my ksh to remain in my AIX, macOS or Linux distro "as-is". I don't want it to be called pksh, krsh, ... I want it to remain POSIX certified and to be the time-proofed shell I use for my production environments and secured managed operations.

its seems you are more or less paraphrasing what people have said here: ksh2020 is not a ksh93u+ drop-in replacement, so keep (or make again) ksh93u+ available, clarify ksh93u+ being "the" ksh for now and then proceed carefully (bug fix patches etc).

As probably many "consumers" I am a system administrator who uses Korn shell over other shells because of its portability, its performance, and of its superior programming capabilities -- e.g. object-oriented REST interfaces for managed services using KAML (https://github.com/ISLEcode/KAML)

yes. all of which was harmed to varying extend in ksh2020 already. I was just as you at the "receiving" end of it (as a user) ...

All that to say this: Korn shell should not be just another open source project abandoned to a community of developers, however skilled and passionate these may be. If it is to survive as a POSIX certified shell, it also needs representation (and defence) in USENIX and the like associations. It needs governance and a roadmap aligned to modern operating system roadmaps.

if the ATT people were able/willing to do that and provide a project lead in the spirit of dgk, that would be great. but it seems that this is completely out of the question if I get their comments right. so the envisaged strategy as outlined by @gordonwoodhull seems to be the next best thing. and then it depends on 'the community' -- in absence of any other option. and it will hopefully then be able to pool sufficiently many reasonable guys in a single fork of the archived ATT repo to make it "the" ksh93 repo which distros could honour and use. something like that...

I want ksh to survive as a POSIX shell... even if we are probably a decade late. I fully support (and ready to invest time) in any initiative to make this happen. The consumer view is the bird's eye view of where we want to go. This is the first step. The developer's view, which is how we get this done, is the next step.

I appreciate the initiative to bring governance to this repository. I hope the extended scope proposed here shall be considered.

it seems everybody who expressed his opinion here (or earlier in the cause of the project) agrees with your "conservative" approach. I for one, do .

[jelmd]

@jelmd Embargoed security issues cannot really be reported to a public bug tracker. There needs to be some confidential channel where security issues can be reported and fixes prepared before they are disclosed publicly.

@kdudka, hmmm, not sure like other projects handle it, e.g. like apache httpd, where one is able to find, which CVEs have been already fixed by just browsing the commit logs, or reading the file which documents all important changes in a more human readable style - e.g. CHANGES.

When the mentioned security issue was discovered, the confidential channel was @siteshwar + @krader1961. They took care of it code-wise and process-wise.

Well, and this approach has proved to fail. In a healthy environment I would expect a github issue, which has at least a security tag, contains some kind of a thread analysis, perhaps some suggestions how to fix it and sooner or later one or more proposed fixes, which can be discussed and properly adjusted. Under such conditions, I guess the mentioned patch (which is, to be polite, far far away from being "brilliant") would have never made it into a main or stable branch. But it is still there and not even a "serious performance breakdown" issue (#1449) was able to bring it back to a review.

So it shows, that what happened in this repo, was basically a one-man-show: the self-appointed "god" writes a patch and the dazzled believer just gave its 'LGTM' or 1+, and often not even that - the patch got simply committed w/o giving the opportunity to review it before. So IMHO we can't be sure, what other Easter eggs got already implanted by krader and a new repo is required to start from scratch, i.e. ksh93u+.

If the new upstream team has some non-public channel where we can safely report embargoed security issues, please let us now. We will be happy to collaborate on them next time.

Thanx for the info. Unfortunately I don't know, whether there will be "a new upstream". I hope, that we will not see an incarnation of this repo but many forks and sooner or later one gets eventually the focus, is acceptable and can be used as "the upstream repo". I don't know either, whether it (or any other) will have such a non-public channel. On one hand it makes somehow sense, on the other hand it seems to be in contradiction to open source development. And last but not least I do not know, whether I would be part of such a channel. At least wrt. C I do not consider myself a C programmer (not even sure, when I wrote last time a single line of C code - probably some years ago). Wrt. ksh93 I would say, I'm 99% on its user side, and only 1% on its developer side ...

So, IMHO distro maintainers should make their own choice, which ones are trustable, open a corresponding issue on demand and wait for whatever happens.

[jghub]

@jelmd

Well, and this approach has proved to fail. In a healthy environment I would expect a github issue, which has at least a security tag, contains some kind of a thread analysis, perhaps some suggestions how to fix it and sooner or later one or more proposed fixes, which can be discussed and properly adjusted. Under such conditions, I guess the mentioned patch (which is, to be polite, far far away from being "brilliant") would have never made it into a main or stable branch. But it is still there and not even a "serious performance breakdown" issue (#1449) was able to bring it back to a review.

I looked at that patch, too, and would agree that is is a "symptomatic treatment" (disable the respective ksh capability of passing such data at that point altogether). but I can see that security holes might better first be patched "in secret" to exclude exploitation by so far unaware actors. but otherwise you are fully right: no information afterwards, no discussion and no attempt of a better solution, not documenting that ksh2020 now simply behaved otherwise by rejecting certain input -- all that is bad.

So it shows, that what happened in this repo, was basically a one-man-show: the self-appointed "god" writes a patch and the dazzled believer just gave its 'LGTM' or 1+, and often not even that - the patch got simply committed w/o giving the opportunity to review it before. So IMHO we can't be sure, what other Easter eggs got already implanted by krader and a new repo is required to start from scratch, i.e. ksh93u+.

I wrote "booby traps" rather than "easter eggs" elsewere but my concern is exactly yours: the erratic way "development" was handled (and yes, very quickly it became a one man show) including, e.g., massive monolithic checkins, the danger is eminent of a multitude of regressions and completely new problems buried in all the unreliably or not correctly documented tinkering.

the obvious ones that immediately hit you are at least reported (I recall "segfault after typing typeset -f
as an example, affecting also the "stable" release AFAICR) and thus "known" (if one memorizes the whole issue list...) and partly they were fixed. it is the hidden layer below that that concerns me even more than the already documented misbehaviour of ksh2020.

If the new upstream team has some non-public channel where we can safely report embargoed security issues, please let us now. We will be happy to collaborate on them next time.

Thanx for the info. Unfortunately I don't know, whether there will be "a new upstream". I hope, that we will not see an incarnation of this repo but many forks and sooner or later one gets eventually the focus, is acceptable and can be used as "the upstream repo". I don't know either, whether it (or any other) will have such a non-public channel. On one hand it makes somehow sense, on the other hand it seems to be in contradiction to open source development. And last but not least I do not know, whether I would be part of such a channel. At least wrt. C I do not consider myself a C programmer (not even sure, when I wrote last time a single line of C code - probably some years ago). Wrt. ksh93 I would say, I'm 99% on its user side, and only 1% on its developer side ...

here I disagree regarding what would be desirable. as said, if the present repo/project would reset and restart under official curation/supervision of one of the ATT people (specifically @lkoutsofios would seem a viable "candidate" as far as I understand) and include further people (but this time not letting them loose on the code unsupervised like the last time...) with modest goals like "maintain availability of 93u+by making it build everywhere" (even if only with old build system for a starter) and than start to compile a list of good patches from solaris or elsewhere and a list of operational bugs that really would need fixing that would help in my view.

so I would prefer another attempt at "keeping it all together". but in view of the fact that the ATT people won't/can't commit to this scenario, yes, forks are the only solution. hopefully, than, the modest number of ksh users around here who care can join forces to do the necessary.

I believe I understand the initial incentive of @siteshwar (and the position of @kdudka) as employees of RedHat regarding the wish to once and for all overhaul/refactor ksh so that it is "ordinary" maintainable software afterwards. but if RedHat really wants that they would have to do it by assigning their own people to it dedicatedly (and restrict it to refactoring of ksh93u+ in order to not run into the mess of trying to refactor while needing/trying to bug-fix ksh93v- bugs accompanied by misguided intentional breaking changes (including feature elimination) all at the same time as has happened in ksh2020...). but the idea that krader would do the "dirty work" for RedHat for free, competently, definitely did not work out. quite to the contrary.

So, IMHO distro maintainers should make their own choice, which ones are trustable, open a corresponding issue on demand and wait for whatever happens.

despite everything that has been said regarding unmaintainable code, opaque build system and known limitations/bugs of ksh93u+ and the hope expressed by @DavidMorano that ksh2020 still can be the future (I really do not believe that...) I am sure that the "make ksh93u+ build for everybody w/o hassle" will suffice to satisfy the needs of 99.9% of all ksh users out there simply because ksh93u+ is "close enough". the ideas tried in 93v- might be revisited at a later time

[aweeraman]

so I would prefer another attempt at "keeping it all together". but in view of the fact that the ATT people won't/can't commit to this scenario, yes, forks are the only solution. hopefully, than, the modest number of ksh users around here who care can join forces to do the necessary.

A question that I have is what is the upstream of 93u+ moving forward, that will host these curated defect fixes and enhancements. @jghub are you volunteering to be the curator and maintainer of the pristine 93u+ official fork which distros can rely upon and not consider this project or the community as dead or abandoned?

[gordonwoodhull]

I've changed the branches as discussed, and added messages to the ksh2020 and master branches. See 5058182 and 0be8255 respectively.

If anyone wants to fork ksh2020, please start from ksh2020^ to remove the messages. And again, I'm happy to transfer issues to any relevant fork.

Rewinding master removed the issue template and CONTRIBUTING.md. I didn't see the point of adding them, but if anyone has requests, I'm game.

I think it's best to leave the repo open for comments so that people can discuss forks and future plans.

I will monitor the repo but I also encourage the rest of the community to answer bug reports with advice about what version to use, as @jghub did here: #1467 (comment)

[jghub]

@gordonwoodhull
thanks a lot for your help so far!

it also is good to leave the repo open for the reasons you state. thanks for this decision, too. :)

[jghub]

@aweeraman

so I would prefer another attempt at "keeping it all together". but in view of the fact that the ATT people won't/can't commit to this scenario, yes, forks are the only solution. hopefully, than, the modest number of ksh users around here who care can join forces to do the necessary.

A question that I have is what is the upstream of 93u+ moving forward, that will host these curated defect fixes and enhancements. @jghub are you volunteering to be the curator and maintainer of the pristine 93u+ official fork which distros can rely upon and not consider this project or the community as dead or abandoned?

maybe you can open a new issue "future plans" or something to that extent (this one is closed now...)? other people should have opinions, too :).

regarding your specific question: I have not considered to volunteer for that role so far since I am sure not the best qualified person to do so. if everything else fails: maybe... but I definitely would prefer to do it as a group -- and having at least one package maintainer (the one from debian ;)) included would be a good start I feel...

[floppym]

Thanks for the ping. As a distro package maintainer myself, I really just need something that actually compiles. If another fork happens and someone actually makes it buildable, just ping me if you would like my help getting it included in Gentoo Linux.

[cschuber]

A quick question. master is now almost equivalent to 93u. However looking through the log 93v is newer. Why was the repo rewound back to 93u instead of 93v?

tag 93u
Tagger: Kurtis Rader krader@skepticism.us
Date: Sat May 19 08:29:17 2018 -0700

AT&T AST release of version ksh93u+ to Github

commit e79c292 (tag: ksh93u, tag: 93u)
Author: Siteshwar Vashisht svashisht@redhat.com
Date: Mon May 15 20:56:58 2017 +0200

iffe depends on cc -E not inserting newlines between tokens

tag 93v
Tagger: Kurtis Rader krader@skepticism.us
Date: Sat May 19 08:30:16 2018 -0700

AT&T AST release of version ksh93v- to Github

commit 2f2b1b8 (tag: ksh93v, tag: 93v)
Author: Lefteris Koutsofios ek@research.att.com
Date: Mon Jan 11 15:58:01 2016 -0500

Version: 2016-01-10-beta

[jelmd]

grep -- '--- Release' lib/package/*.html

A trailing dash ('-') means beta ...

[jghub]

@cschuber
long version (@jelmd should correct me where I get the details wrong): 93v development had to be aborted by the ATT team around D. Korn which had to leave in 2013(?). 93v- was published as a last resort in 2014 and then dgk et al had to give up on the code base completely.

93v- for a time was adopted by repos but most/all(?) noted rapidly the remaining massive problems (stability, severely dysfunctional new features -- which is a pity since there are very good ideas there) and reverted to 93u+ as the only alternative to have a reliable stable predictable ksh93. see e.g. this remark #1466 (comment) from the opensuse maintainers of ksh.

personally I recall my 2-3 month try with ksh93v- in place of ksh93u+ back in 2014/early 2015 IIRC (to get hold of the new features like programmable command completion, builtin grep etc): most of the new stuff was really fragile and dysfunctional to varying degrees and stability overall was bad so it really was by no means an adequate, let alone superior replacement to ksh93u+ -- so I also went back to 93u+.

and to reiterate: it was a crucial error for ksh2020 to start from 93v- (the beta state, that is) for the above described situation. recently, finally, even krader acknowledged so much #1461 (comment) although obviously still not getting it that ksh93v- never was an official release but the given beta-state #1461 (comment)

also note that ksh2020 actually removed all new 93v- features that I am aware of, the important, wanted ones like command completion and new builtins, anyway -- except keeping a set of predefined mathematical constants like $((E)) (krader might clarify whether more was kept but I suspect he does not know because he is not informed about the feature differences between 93u+ and 93v-). they did delete the new features for the simple reason that a) they did not work properly and b) they were not able to fix the problems.

featurewise ksh2020 is basically only a dismantled ksh93v- with no relevant capabilities beyond ksh93u+ while having inherited all bugs/regressions that crept into ksh93v- relative to 93u+. what misbehaviour exactly has been fixed in ksh2020 relative to ksh93v-? such an overview should be compiled...

[cschuber]

Thanks for the explanation.

I have updated the ksh93 and ksh93-devel ports in FreeBSD, only in my git tree so far, to 93u and to the head of master, respectively, to reflect the change and will git svn dcommit the commits later today or tomorrow. We still have ksh2020 which I hope to track ksh-community with. And ast-ksh, maintained @saper, which will continue to provide 93v-, unless he chooses otherwise.

[jghub]

@cschuber

Thanks for the explanation.

I have updated the ksh93 and ksh93-devel ports in FreeBSD, only in my git tree so far, to 93u and to the head of master, respectively, to reflect the change and will git svn dcommit the commits later today

understood. thanks for the update.

or tomorrow. We still have ksh2020 which I hope to track ksh-community with. And ast-ksh, maintained @saper, which will continue to provide 93v-, unless he chooses otherwise.

maybe you get this wrong: ksh2020 is currently only available on the respective branch in the present repo and I don't know whether the ksh2020 authors change their mind and actually do fork and go on with their work (so far they both unfortunately seem instead to prefer to invest their time trying to harm ksh-community with their recent postings in several issues culminating in the broadcast of @siteshwar to distro maintainers (edit: so that nobody misses it, this one: #1466 (comment)). but hey!) so if you are primarily interested in ksh2020 you need to stay with this repo right now.

ksh-community, however, will definitely steer clear of the ksh2020 code base completely and proceed from the original ATT/AST sources in order to get a stable easily buildable ksh93u+ into "circulation" ASAP again. but also see the thoughts of @saper here #963 (comment) and my reply which I believe is mostly in sync with all ksh-community members (hopefully).

[cschuber]

I know it's in a separate branch. FreeBSD ports will fetch a github hash/tag/branch. The ksh2020 port lists the ksh2020 branch. The magic is in the secret sauce.

I will point it to ksh-community when it becomes active or delete it if ksh-community never takes off. For now it points to the ksh2020 branch.

[siteshwar]

I am keeping a backup of changes me and @krader1961 had done in this repository. I don't plan to actively work on it, however.