Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewinding this repo and encouraging community #1466

Closed
gordonwoodhull opened this issue Feb 4, 2020 · 75 comments
Closed

Rewinding this repo and encouraging community #1466

gordonwoodhull opened this issue Feb 4, 2020 · 75 comments

Comments

@gordonwoodhull
Copy link
Member

@gordonwoodhull gordonwoodhull commented Feb 4, 2020

I work with @lkoutsofios at AT&T Labs Research and volunteered to see if I could help restore the repo to a state the community is comfortable with. This was last discussed in #1464.

I am not an expert in ksh or the technical issues, and I don't know the entire history, but my understanding is that

  1. ksh93u+ is the final official release from AT&T. This can be found in the branch 2012-08-01-master
  2. A later version, ksh93v-, was contributed by the original authors after they left AT&T, but this version is not considered stable. This can be found in the tag ksh93v
  3. @krader1961 has been maintaining the master branch for some years. He has tried to streamline the codebase (and only support ksh, not the rest of ast), but this has caused compatibility issues.

The original authors of ksh have all left AT&T. We still have plenty of users inside the company, but no one who can maintain and manage the project.

My proposal is:

  • Move the current master to a branch "ksh2020" (or whatever is appropriate).
  • Clearly mark README.md and CHANGELOG.md on that branch as unmaintained, point to #1464 and this issue for further info
  • Reset the master branch to 2012-08-01-master
  • Remove external collaborators from the repo. Since no one in AT&T is maintaining this, it doesn't make sense to have this fork be the active one. Git and Github allow anyone to take the code in whatever direction they want, in their own space.
  • Clean up README.md to reflect the current state: "This repo contains the AT&T ksh93u+ and experimental ksh93v- versions of AST. There may be later forks of AST and ksh out there."
  • Change CONTRIBUTING.md and issue template to be clear we are not actively maintaining this fork and do not promise to review anything.
  • Shorten the repo description to "AST - AT&T Software Technology" (#1441)

After this is complete, we at AT&T may contribute patches to ksh93u+ in the master branch. We probably won't review or accept contributions unless they are trivial (build issues, etc.) Eventually, we will archive this repo.

Currently I don't think anyone in the company plans to create a repo just for ksh. However, I think this is fairly straightforward and can be taken up by the community.

If anyone wants to create a community fork, I'd suggest creating a new GitHub Organization. This is free for open source projects. The bigger commitment is time, making contribution and maintenance guidelines clear, possibly creating a code of conduct, etc. Of course if anyone wants to fork this repo to an existing organization, they are welcome to do that too.

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 4, 2020

@gorodnwoodhull: my most recent comments are here: #1464 (comment)

sorry for overlooking this (better) place. will take note from now on...

@gordonwoodhull

This comment has been minimized.

Copy link
Member Author

@gordonwoodhull gordonwoodhull commented Feb 4, 2020

I went ahead with removing the external collaborators and fixing the description. I'll wait to see if there is further discussion about the other points before continuing with those.

@lkoutsofios

This comment has been minimized.

Copy link
Contributor

@lkoutsofios lkoutsofios commented Feb 4, 2020

thanks @gordonwoodhull for helping out.

my original motivation for going along with @krader1961's proposal was to keep at least KSH alive since no one else was volunteering to work on the AST project as a whole. but I think we were all clear that backwards compatibility was mandatory. KSH's strongest advantage over other SH/KSH clones was its programming interface. if the new version breaks old scripts, there's no point in using it. it seems from the recent issues on this repo that this isn't working out. so @gordonwoodhull's changes seem like the way to go. we'll see what happens with KSH. I do have some fixes for the u+ version that should get it to build and work on all recent distros of redhat, ubuntu, and opensuse, that I can push out. unfortunately, I don't have the time to be more involved.

@DavidMorano

This comment has been minimized.

Copy link

@DavidMorano DavidMorano commented Feb 5, 2020

Just some friendly observational comments:

I have tried to follow both the recent discussion about rewinding the repository, et cetera, and the progress on this project since @siteshwar and then @krader1961 started making major contributions back in 2017. For the record, my own experience w/ KSH dates from its very (albeit public) inception, back in 1983 (I was at Bell Labs at the time and following). I have used KSH almost entirely exclusively and continuously since that time to the present (including very recently on ksh2020). Some items:

  • this code is very complicated and somewhat messy by the best modern coding practices (that may be something of a gross understatement); it is NOT suited for the casual programmer -- or team of programmers -- to dive into; despite people possibly feeling otherwise, I think that @krader1961 is-was fairly competent as a programmer and developer; where @krader1961 seemed to possibly lack something was his understanding of the various historical customer uses of KSH over the last 37 years -- being much more intensive and far-reaching than he seemed to realize; I will not dwell on this now, but KSH was Perl before Perl was, and it was Python before Python was; it had substantial enterprise applications written in it before the end of the 1980s (and the first book only came out in 1988)

  • the code creates a program that is substantially defined by three standards: 1) POSIX, 2) the KSH book (by David Korn), and 3) the existing behavior of the ksh93u+ branch; this is not something to be trifled with lightly; extreme care needs to be exercised to not break any existing documented or expected behavior (this may be an area were @krader1961 might have appeared to not be as sensitive as people would have expected)

  • despite some possible (arguably relatively minor) personality issues, @krader1961 has put an exceptionally enormous amount of work into fixing problems that even existed in ksh93u+ that were not always readily observed or complained about; to replicate @krader1961's work starting from scratch at the ksh93u+ branch will be extremely non-trivial (please, @siteshwar give your feeling on this if you would)

  • I am aware that even ksh93u+ had severe bugs and problems that needed to be eventually fixed; many of these did not manifest readily for most users, but they needed to be addressed eventually none-the-less; these are things such as strays writes, buffer overflows, and the like; there were a variety of minor (and not so minor) problems also that had been in there for about 20+ years (also not readily complained about by most regular uses); @krader1961 had made a very substantial start at getting some of the more serious of these (the memory related stuff) fixed (with the help of, among other things, ASAN and Valgrind); starting again from ksh93u+ means a huge amount of dog-work in fixing up again what @krader1961 has already done

  • not all distros are equal! some distros are more important and have more say in the world than others! sadly for some, the Red-Hat distro is more important now-a-days than pretty much all other distros combined (yes my opinion!); and I am not one who even likes Linux derived distros in the first place! ; Linux itself is very much inferior in many ways to older non-Linux distros (yes that is also my opinion); so the short of it is, that I think that @siteshwar should have more say in how KSH proceeds into the future than all the rest of us (distro maintainers or not) combined! I am sorry, but that is what I feel should happen; @siteshwar is not a dumb guy; he knows 'the score"; he knows full well that maintaining KSH is like managing fire in your hands; he works for Red Hat!!! Red Hat understands managing real (hard core, porn) enterprise (and other) customers and their requirements and demands; I would ask that more deference be given to @siteshwar's thoughts on this whole matter going forward; no joke

  • my fear is that if the ksh2020 branch is essentially abandoned in favor of starting from ksh93u+ again, a huge amount of time and work will have been forfeited; I want to hear from @siteshwar more of this point; this kind of decision should not be taken lightly

  • even separating out KSH from the rest of the AST code was not a totally trivial endeavor; @siteshwar and @krader1961 did this w/ ksh2020 at no little (no little) effort in itself; although I had some trepidation at the time, I feel that this was a useful and important thing to do to modernize the KSH code base in particular; this is just one example of something that would have to be done again if starting from scratch

  • my own preference going forward would be something along the lines of the following: 1) bring the ksh93u+ branch to a new state (calling it something else, ksh93uu+ whatever) so that it can be easily compiled by the community (easily, maybe using make, autotools, CMake, whatever) so that distros have a fairly reliable and readily accessible base to port from (in the near term), but then 2) continue w/ the ksh2020 branch and make it more conforming to historical and expected behavior; ksh2020 would be the base for future -- possibly feature enhanced -- versions going forward

  • and finally, for those who think that just using ksh93u+ is some sort of magical cookie to the golden land, it is very likely that making any fixes to ksh93+ (which will likely be required) will break some conforming behavior in unexpected ways; this is likely part of what happened with ksh2020 -- the act of fixing some previous bad stuff, exposed some bad stuff that was still in there already from the past, but not previously readily observable by most customers; this happens all of the time with old code, but to get through it one has to often just confront bug after bug and plow ahead through the hard times; for myself, I feel that ksh2020 has started down that path (as hard as it may be); starting fresh from ksh93u+ may never get past the hump of fixing some of its worst bugs lurking within it

  • KSH is a complicated and still -- in many quarters -- a mission critical program; in my opinion, it needs the attention and contributions from the entire world to focus on a single main code base branch in order for it to flourish into the future, rather than just staying "useable" at the ksh93u+ level; ksh93u+ had (has) many bugs that need to be eventually fixed if there is any hope for real future feature enhancements to likely work; the code base needs to be modernized also (not a trifle in itself)

  • just my two cents on this whole thing!

@jelmd

This comment has been minimized.

Copy link

@jelmd jelmd commented Feb 5, 2020

@gordonwoodhull, I think you got it right and your proposal is good.

However, some nits:

* [ ]  Move the current master to a branch "ksh2020" (or whatever is appropriate).

Yes. But to avoid confusion [1] [2], that people use it as an official or stable version (actually what happened with version v-), README.md as well as CHANGELOG.md in this branch should make it very clear, that this is an experimental branch (e.g. by adding an appropriate level 1 header at the top) and the mentioned articles refer to an experimental release (or that there is no ksh2020.*, etc.).

* [ ]  Rewind the master branch to 2012-08-01-master

Adding @lkoutsofios patches is a good idea - thanks @lkoutsofios for spending your time! Direct merge, separate *.patch files or PR, whatever you like is probably ok.

* [ ]  Clean up README.md to reflect the current state, and point to any active forks from there. If desired, transfer relevant issues to child forks.

If no-one at AT&T has time to maintain it, or prefers one fork over another, it should be kept neutral and just mention, that there are forks which are not AT&T driven. I think, if there is a valuable, active fork, people interested in ksh93 will find it by themselves or by getting hints from others...

@gordonwoodhull

This comment has been minimized.

Copy link
Member Author

@gordonwoodhull gordonwoodhull commented Feb 5, 2020

@DavidMorano thanks for your perspective. I don't know the history but I agree in principle with what you're saying.

I will second that I hope ksh2020 starts an organization, perhaps led by @siteshwar and @krader1961, starting from what is currently on master.

@gordonwoodhull

This comment has been minimized.

Copy link
Member Author

@gordonwoodhull gordonwoodhull commented Feb 5, 2020

@jelmd, sure, sounds good - I've edited the tasks above.

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 5, 2020

@DavidMorano, I appreciate your perspective, especially concerning the "KSH was Perl before Perl etc" aspect and its implications.

regarding what has been achieved (or not...) in ksh2020 I respect your assessment, although it seems too "understanding" in my view. I would readily agree that I cannot really judge the technical difficulties involved when rewriting/rephrasing the C-code . I am in the position of a "typical" KSH user who uses the KornShell language for scripting w/o knowing the C innards of the ksh program .

let's say I cannot build a car but I can drive it and if I drive ksh2020 around the block I notice strange noises from the motor department and assorted malfunctions. it's slower, too... before the overhaul the car was just fine, only had some peculiarities the driver knew (mostly) to avoid. so while the overhaul obviously involved technical skills (at the C-level) beyond my own I can judge that the job has not produced something better but something worse.

I maintain that is the current state of ksh2020. choose any metric you like (except better formatted C-code and "tidiness").

as you explained to me elsewhere (I forgot so far to again thank you for this!) #1449 (comment), you especially value the malloc vs. vmalloc changes for reasons of your special interests although you also agree that vmalloc might be uncritical for the 99.9% crowd not writing their own (multi-threaded) builtins. of course, if it turns out, that indeed replacing vmalloc is clearly better (but for whom and according to what measures would have to be decided) one might consider to do that to 93u+ at some point down the road. I am not sure that there is clear evidence that this is worth it -- but not for me to decide....

and it sure would have been the responsibility of @krader1961 and @siteshwar to listen to people like you and many others (I was a latecomer in the queue of people raising concerns...) from the start and adjust their actions how to deal with the code, the features etc. I feel they (at least krader) had the wrong agenda. and the decision to start from the 93v- beta is not comprehensible to me, given the circumstances (known buggy beta state, no advice from dgk etc.).

I do not know of any "problem/bug driven" TODO list with which ksh2020 was started. there was the "we must first overhaul the code to adhere to modern standards" and the fuzzy "keep ksh relevant" approach and that without any adequate understanding of either the code or KornShell. the issue history is there for everybody to go through (it is not that extremely long): a clear story evolves of misconceptions, false claims and -- and that was the central problem throughout -- an incomprehensible rigidity and refusal to adjust course and actions in the small or in the big issues on the side of krader.

so I believe your rather benevolent description of krader's "possible (arguably relative minor) personality issues" is objectively not accurate (and this is not irrelevant since it crucially has determined what has happened in ksh2020 -- it could have been a different project -- and is also crucial for deciding how to proceed from here).

for me this implies that ksh2020 should (no: must!), indeed pursue its goals independently of ksh93. I also insist that terminology-wise there is need for a strict and hopefully agreed-upon "branding". it was proposed to call ksh2020 "krsh" for obvious reasons, but I really believe the "ksh2020" name is more sensible and "catchy" and accurate enough, so why not call it that for the next 25 years just like ksh93 still is called that...

altogether I am still not quite sure, what exactly the goals of ksh2020 are: so far it has eliminated most or all of the new features of 93v- since they could not debug those (being, so far, not yet or not fully functional in 93v-) while incompatibilities and regressions and new bugs crept in and performance went southward. maybe, that all can be fixed and at some point ksh2020 is better than a (patched) 93u+ but I really do not see that right now.

to reiterate: in my view, ksh93 and ksh2020 development should not proceed in a common repository. this can not work out. what could work out in my view is a friendly coexistence (in the long run, at least, when potential hard feelings have waned) and, hopefully, bug fix exchange (the latter for the time being could very well be unidirectional from ksh2020 to ksh93u+ for the rather few cases (AFAICS) where ksh2020 has fixed things that apply to 93u+ and not only to 93v-).

I am looking forward to implementation of the strategy outlined by @gordonwoodhull: even if ksh2020 could proof at some point to be the "future", I believe making ksh93u+ easily available again for everybody ASAP is much more important right now. this requires separate projects as the history of the ksh2020 project proofs beyond reasonable doubt in my view.

@gordonwoodhull

This comment has been minimized.

Copy link
Member Author

@gordonwoodhull gordonwoodhull commented Feb 5, 2020

I clarified that we won't immediately archive the repo, and we'll contribute what patches we have, but we probably won't accept contributions unless they are trivial.

I'll proceed with the plan in the next week.

@DavidMorano

This comment has been minimized.

Copy link

@DavidMorano DavidMorano commented Feb 6, 2020

@jelmd
I think that we agree (and most everyone agrees) with the growing consensus around the following:

  • that a new workable version of KSH be made out of the ksh93u+ branch as its base, possibly replacing NMAKE w/ something else; this would serve the needs of distros that need a very reliable (as much as can be expected given everything) and behaviorally compliant KSH

  • further, I have no objection to VMALLOC staying in the ksh93u+ follow-on product; if I ever want to use this new KSH product for myself, I will just replace VMALLOC w/ a regular system one as I did in the past

  • neither do I take any issue w/ trying to bring the new (ksh93u+ derived) product up-to-date w/ any patches or fixes that can readily be put into it (the issue here is the time to do this at the level that ksh2020 has already achieved)

  • I do not want to waste ksh2020, and neither do @krader1961 and @siteshwar as far as I would guess; but it seems that the consensus is that they open a new unique repo for its development, separate from this ATT/AST repo

  • my guess is that @krader1961 and @siteshwar will likely continue to develop and orient ksh2020 as a possible next generally-released KSH also, and I encourage that work; but my preference would be that it try to adhere more to the standardized and expected behavior of prior KSHs -- circumscribed as I stated previously around 1) POSIX, 2) the KSH book (Korn), and 3) ksh93u+ behavior

For myself, I admit that I am optimistic that the benefits that @krader1961 has put into ksh2020 (much of it quite subtle, like cleaning up the various NV preprocessor flags mess, to name just one) more than offsets the regressions that people have complained about. But, yes, this is opinion on my part. But based on following many or almost all of @krader1961 commits. I have seen regressions in ksh2020 myself (like in the form of unexpected crashes). But in fairness, there were some unexpected crashes in many of the production KSHs over the last couple decades also. My hope is that, accidentally or not, @krader1961 might have actually fixed some of these (again, perhaps even by accident as it were).

Having a follow-on version of ksh93u+ available for distros (whatever its name is, I do not care about any names really) will take the pressure off of the ksh2020 development, so that they can concentrate on cleaning it up so that it appears to perform at least as reliably and conformally as the old ksh93u+ did (or its follow-on replacement).

I also have to admit that I hope that ksh2020 succeeds totally eventually. I have seen, and sometimes tried to understand, the code in ksh93u+ and it is often quite difficult (and I look at tons of bad code all of the time for my living). I tend to get spooked by and biased against code that has memory buffer overflows and the like, and I very much welcomed @krader1961 and other's initial attempts to clean some or much of it up -- as much as is exposed by bad behavior or by development tools (like ASAN or Valgrind, et cetra). I do not feel that the code base of ksh93u+ is really fit for the future. I would like to see the code come to a place where both fixes and feature enhancements become at some point much easier. But I think that the right path for this (following the project as I have) is with ksh2020, simply due to the amount of cleanup put into it (being perceived as useful or not to the end user). I could be wrong, but we will now see. Neither do I discourage future major fixes or enhancements to the new ksh93u+ branch. Let the race begin, as it were. The end code-builders or end-users can decide for themselves which one is the real future (or not).

@jelmd

This comment has been minimized.

Copy link

@jelmd jelmd commented Feb 6, 2020

@DavidMorano, I totally understand your concerns. But what we need right now is to get back the trust from vendors and users and therefore the outlined plan is IMHO the right way to go.

FWIW, I'm basically repeating, what has been already said:

At the moment all distros ship a ksh93 package, have certain bugs already fixed with their own patches and the process to build the package, or apply a small patch from time to time is almost a no-brainer. Vendors/developers/users may even have a knowledge base, FAQ, ... wrt. workarounds for existing bugs/corner cases so that even patching is not really needed, as @jghub already pointed out. So on the vendor side there is no need to invest a lot of/any time and they can continue to ship it until a new star appears on the sky, which makes it redundant.

Redundant means for me full compatibility to ksh93u+. E.g. the getopts self documenting as well as the time conversion and formatter feature are essential and unique (skim over acme-ksh scripts to get some more details if you like). Last but not least I do not like to waste a lot of time to find out, which of several dozen man pages actually contains a description of the feature I wanna use (like zsh), and than skim over several pages again to finally find it. ${builtin} --man is such a useful thing, that I do not really care about the size of the ksh93 binary.

Anyway, new stuff like native json support, possibly ksh thread safety, ... would be nice to have, yes. But I think, even changing the build system only would requires a lot of "review" time at least on distro maintainers' side to make sure, that they still ship a stable/usable version. Debian has shown, that this is probably too challenging, decided to just trust the most active members of the repo and bundled something that at least I do not want to see on any of the 100+ zones/containers I need to manage (or others, which use scripts I wrote in the last 20+ years). Therefore, encouraging people to fork and give it a distinguishable name seems to be the best option. If it is good, distros will pick it up pretty fast and do not need to care about compatibility wrt. ksh93* ...

@kdudka

This comment has been minimized.

Copy link
Contributor

@kdudka kdudka commented Feb 6, 2020

But what we need right now is to get back the trust from vendors and users and therefore the outlined plan is IMHO the right way to go.

The necessary precondition to get back the trust of Red Hat is that upstream is able to competently and reasonably fast react on reports of important security issues. For example, how quickly would your team be able to fix the following security issue in ksh93u+?

https://access.redhat.com/security/cve/CVE-2019-14868

@siteshwar

This comment has been minimized.

Copy link
Contributor

@siteshwar siteshwar commented Feb 6, 2020

I would ask that more deference be given to @siteshwar's thoughts on this whole matter going forward; no joke

I give up and move on with my life. I made all my efforts with the goodwill to help the community and it turned out that I achieved quite the opposite. I hope that someone else would pick up this project and would make progress in the direction that's more acceptable to the community.

@DavidMorano

This comment has been minimized.

Copy link

@DavidMorano DavidMorano commented Feb 6, 2020

@siteshwar

I give up and move on with my life. ...

I am very sorry to hear your decision on this matter.
I think that you were the best thing to happen to KSH in years!
Having RedHat (as represented by yourself) in the middle of decisions regarding KSH was likely the best thing to happen to KSH since the old haughty days of AT&T (pretty much long gone now).
I feel badly that the general community did not seem to show you more respect (deference).
Thanks very much for the service that you did give to this project over these past years.

All the best with whatever you do going forward!

@jelmd

This comment has been minimized.

Copy link

@jelmd jelmd commented Feb 7, 2020

...

The necessary precondition to get back the trust of Red Hat is that upstream is able to competently and reasonably fast react on reports of important security issues. For example, how quickly would your team be able to fix the following security issue in ksh93u+?

https://access.redhat.com/security/cve/CVE-2019-14868

@kdudka, if no-one creates an repo issue, it is very questionable, that it get fixed "in time" by the members of the related repo at all - so thanx for bringing it finally (after 4+ month) to our attention. And BTW, silently inserting a very questionable patch, which just swings the big hammer, is ok? Or can you help me? I could not find a related issue in this repo using the keyword security or CVE . Even the commit contains neither the word "security" nor "CVE". Is this really your understanding of open source development?

@marcastel

This comment has been minimized.

Copy link

@marcastel marcastel commented Feb 7, 2020

I would like to express an "external" (or consumer) point of view here on KSH for today and tomorrow.

Though I have not contributed any code to this GitHub repository, I have (since my #42 comment) maintained my own private build of ksh and a have some insight on its intrinsics -- and for one, I like its build system which, optimised, is better than the gmake/cmakes of the world, or their more recent python or javascript alternatives)

But back to the consumer view of ksh. In the comments I have seen so far we speak about the "community"... and I understand that to be the "community of developers" as opposed to the "community of users".

As a user, I am obviously very concerned with backwards compatibility, performance, and much less so about the build system. But more importantly I am concerned about the POSIX status of AT&T ksh which has landed this shell on all UNIX distributions.

This is what is important to me, as a consumer. I want my ksh to remain in my AIX, macOS or Linux distro "as-is". I don't want it to be called pksh, krsh, ... I want it to remain POSIX certified and to be the time-proofed shell I use for my production environments and secured managed operations.

As probably many "consumers" I am a system administrator who uses Korn shell over other shells because of its portability, its performance, and of its superior programming capabilities -- e.g. object-oriented REST interfaces for managed services using KAML (https://github.com/ISLEcode/KAML)

And as one of these "consumers" I wish we could upgrade modern operating systems from POSIX "sh" to POSIX "ksh". Imagine such a replacement in the GNU build system... ksh in lieu of sh and m4... lean and clean. Obviously a little provocative, only to emphasise the value of ksh by its functionality, its programability and its heritage.

All that to say this: Korn shell should not be just another open source project abandoned to a community of developers, however skilled and passionate these may be. If it is to survive as a POSIX certified shell, it also needs representation (and defence) in USENIX and the like associations. It needs governance and a roadmap aligned to modern operating system roadmaps.

I want ksh to survive as a POSIX shell... even if we are probably a decade late. I fully support (and ready to invest time) in any initiative to make this happen. The consumer view is the bird's eye view of where we want to go. This is the first step. The developer's view, which is how we get this done, is the next step.

I appreciate the initiative to bring governance to this repository. I hope the extended scope proposed here shall be considered.

@kdudka

This comment has been minimized.

Copy link
Contributor

@kdudka kdudka commented Feb 7, 2020

@jelmd Embargoed security issues cannot really be reported to a public bug tracker. There needs to be some confidential channel where security issues can be reported and fixes prepared before they are disclosed publicly. When the mentioned security issue was discovered, the confidential channel was @siteshwar + @krader1961. They took care of it code-wise and process-wise. If the new upstream team has some non-public channel where we can safely report embargoed security issues, please let us now. We will be happy to collaborate on them next time.

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 7, 2020

@marcastel

As a user, I am obviously very concerned with backwards compatibility, performance, and much less so about the build system. But more importantly I am concerned about the POSIX status of AT&T ksh which has landed this shell on all UNIX distributions.

This is what is important to me, as a consumer. I want my ksh to remain in my AIX, macOS or Linux distro "as-is". I don't want it to be called pksh, krsh, ... I want it to remain POSIX certified and to be the time-proofed shell I use for my production environments and secured managed operations.

its seems you are more or less paraphrasing what people have said here: ksh2020 is not a ksh93u+ drop-in replacement, so keep (or make again) ksh93u+ available, clarify ksh93u+ being "the" ksh for now and then proceed carefully (bug fix patches etc).

As probably many "consumers" I am a system administrator who uses Korn shell over other shells because of its portability, its performance, and of its superior programming capabilities -- e.g. object-oriented REST interfaces for managed services using KAML (https://github.com/ISLEcode/KAML)

yes. all of which was harmed to varying extend in ksh2020 already. I was just as you at the "receiving" end of it (as a user) ...

All that to say this: Korn shell should not be just another open source project abandoned to a community of developers, however skilled and passionate these may be. If it is to survive as a POSIX certified shell, it also needs representation (and defence) in USENIX and the like associations. It needs governance and a roadmap aligned to modern operating system roadmaps.

if the ATT people were able/willing to do that and provide a project lead in the spirit of dgk, that would be great. but it seems that this is completely out of the question if I get their comments right. so the envisaged strategy as outlined by @gordonwoodhull seems to be the next best thing. and then it depends on 'the community' -- in absence of any other option. and it will hopefully then be able to pool sufficiently many reasonable guys in a single fork of the archived ATT repo to make it "the" ksh93 repo which distros could honour and use. something like that...

I want ksh to survive as a POSIX shell... even if we are probably a decade late. I fully support (and ready to invest time) in any initiative to make this happen. The consumer view is the bird's eye view of where we want to go. This is the first step. The developer's view, which is how we get this done, is the next step.

I appreciate the initiative to bring governance to this repository. I hope the extended scope proposed here shall be considered.

it seems everybody who expressed his opinion here (or earlier in the cause of the project) agrees with your "conservative" approach. I for one, do .

@jelmd

This comment has been minimized.

Copy link

@jelmd jelmd commented Feb 8, 2020

@jelmd Embargoed security issues cannot really be reported to a public bug tracker. There needs to be some confidential channel where security issues can be reported and fixes prepared before they are disclosed publicly.

@kdudka, hmmm, not sure like other projects handle it, e.g. like apache httpd, where one is able to find, which CVEs have been already fixed by just browsing the commit logs, or reading the file which documents all important changes in a more human readable style - e.g. CHANGES.

When the mentioned security issue was discovered, the confidential channel was @siteshwar + @krader1961. They took care of it code-wise and process-wise.

Well, and this approach has proved to fail. In a healthy environment I would expect a github issue, which has at least a security tag, contains some kind of a thread analysis, perhaps some suggestions how to fix it and sooner or later one or more proposed fixes, which can be discussed and properly adjusted. Under such conditions, I guess the mentioned patch (which is, to be polite, far far away from being "brilliant") would have never made it into a main or stable branch. But it is still there and not even a "serious performance breakdown" issue (#1449) was able to bring it back to a review.

So it shows, that what happened in this repo, was basically a one-man-show: the self-appointed "god" writes a patch and the dazzled believer just gave its 'LGTM' or 1+, and often not even that - the patch got simply committed w/o giving the opportunity to review it before. So IMHO we can't be sure, what other Easter eggs got already implanted by krader and a new repo is required to start from scratch, i.e. ksh93u+.

If the new upstream team has some non-public channel where we can safely report embargoed security issues, please let us now. We will be happy to collaborate on them next time.

Thanx for the info. Unfortunately I don't know, whether there will be "a new upstream". I hope, that we will not see an incarnation of this repo but many forks and sooner or later one gets eventually the focus, is acceptable and can be used as "the upstream repo". I don't know either, whether it (or any other) will have such a non-public channel. On one hand it makes somehow sense, on the other hand it seems to be in contradiction to open source development. And last but not least I do not know, whether I would be part of such a channel. At least wrt. C I do not consider myself a C programmer (not even sure, when I wrote last time a single line of C code - probably some years ago). Wrt. ksh93 I would say, I'm 99% on its user side, and only 1% on its developer side ...

So, IMHO distro maintainers should make their own choice, which ones are trustable, open a corresponding issue on demand and wait for whatever happens.

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 8, 2020

@jelmd

Well, and this approach has proved to fail. In a healthy environment I would expect a github issue, which has at least a security tag, contains some kind of a thread analysis, perhaps some suggestions how to fix it and sooner or later one or more proposed fixes, which can be discussed and properly adjusted. Under such conditions, I guess the mentioned patch (which is, to be polite, far far away from being "brilliant") would have never made it into a main or stable branch. But it is still there and not even a "serious performance breakdown" issue (#1449) was able to bring it back to a review.

I looked at that patch, too, and would agree that is is a "symptomatic treatment" (disable the respective ksh capability of passing such data at that point altogether). but I can see that security holes might better first be patched "in secret" to exclude exploitation by so far unaware actors. but otherwise you are fully right: no information afterwards, no discussion and no attempt of a better solution, not documenting that ksh2020 now simply behaved otherwise by rejecting certain input -- all that is bad.

So it shows, that what happened in this repo, was basically a one-man-show: the self-appointed "god" writes a patch and the dazzled believer just gave its 'LGTM' or 1+, and often not even that - the patch got simply committed w/o giving the opportunity to review it before. So IMHO we can't be sure, what other Easter eggs got already implanted by krader and a new repo is required to start from scratch, i.e. ksh93u+.

I wrote "booby traps" rather than "easter eggs" elsewere but my concern is exactly yours: the erratic way "development" was handled (and yes, very quickly it became a one man show) including, e.g., massive monolithic checkins, the danger is eminent of a multitude of regressions and completely new problems buried in all the unreliably or not correctly documented tinkering.

the obvious ones that immediately hit you are at least reported (I recall "segfault after typing typeset -f
as an example, affecting also the "stable" release AFAICR) and thus "known" (if one memorizes the whole issue list...) and partly they were fixed. it is the hidden layer below that that concerns me even more than the already documented misbehaviour of ksh2020.

If the new upstream team has some non-public channel where we can safely report embargoed security issues, please let us now. We will be happy to collaborate on them next time.

Thanx for the info. Unfortunately I don't know, whether there will be "a new upstream". I hope, that we will not see an incarnation of this repo but many forks and sooner or later one gets eventually the focus, is acceptable and can be used as "the upstream repo". I don't know either, whether it (or any other) will have such a non-public channel. On one hand it makes somehow sense, on the other hand it seems to be in contradiction to open source development. And last but not least I do not know, whether I would be part of such a channel. At least wrt. C I do not consider myself a C programmer (not even sure, when I wrote last time a single line of C code - probably some years ago). Wrt. ksh93 I would say, I'm 99% on its user side, and only 1% on its developer side ...

here I disagree regarding what would be desirable. as said, if the present repo/project would reset and restart under official curation/supervision of one of the ATT people (specifically @lkoutsofios would seem a viable "candidate" as far as I understand) and include further people (but this time not letting them loose on the code unsupervised like the last time...) with modest goals like "maintain availability of 93u+by making it build everywhere" (even if only with old build system for a starter) and than start to compile a list of good patches from solaris or elsewhere and a list of operational bugs that really would need fixing that would help in my view.

so I would prefer another attempt at "keeping it all together". but in view of the fact that the ATT people won't/can't commit to this scenario, yes, forks are the only solution. hopefully, than, the modest number of ksh users around here who care can join forces to do the necessary.

I believe I understand the initial incentive of @siteshwar (and the position of @kdudka) as employees of RedHat regarding the wish to once and for all overhaul/refactor ksh so that it is "ordinary" maintainable software afterwards. but if RedHat really wants that they would have to do it by assigning their own people to it dedicatedly (and restrict it to refactoring of ksh93u+ in order to not run into the mess of trying to refactor while needing/trying to bug-fix ksh93v- bugs accompanied by misguided intentional breaking changes (including feature elimination) all at the same time as has happened in ksh2020...). but the idea that krader would do the "dirty work" for RedHat for free, competently, definitely did not work out. quite to the contrary.

So, IMHO distro maintainers should make their own choice, which ones are trustable, open a corresponding issue on demand and wait for whatever happens.

despite everything that has been said regarding unmaintainable code, opaque build system and known limitations/bugs of ksh93u+ and the hope expressed by @DavidMorano that ksh2020 still can be the future (I really do not believe that...) I am sure that the "make ksh93u+ build for everybody w/o hassle" will suffice to satisfy the needs of 99.9% of all ksh users out there simply because ksh93u+ is "close enough". the ideas tried in 93v- might be revisited at a later time

@aweeraman

This comment has been minimized.

Copy link
Contributor

@aweeraman aweeraman commented Feb 8, 2020

so I would prefer another attempt at "keeping it all together". but in view of the fact that the ATT people won't/can't commit to this scenario, yes, forks are the only solution. hopefully, than, the modest number of ksh users around here who care can join forces to do the necessary.

A question that I have is what is the upstream of 93u+ moving forward, that will host these curated defect fixes and enhancements. @jghub are you volunteering to be the curator and maintainer of the pristine 93u+ official fork which distros can rely upon and not consider this project or the community as dead or abandoned?

@gordonwoodhull

This comment has been minimized.

Copy link
Member Author

@gordonwoodhull gordonwoodhull commented Feb 8, 2020

I've changed the branches as discussed, and added messages to the ksh2020 and master branches. See 5058182 and 0be8255 respectively.

If anyone wants to fork ksh2020, please start from ksh2020^ to remove the messages. And again, I'm happy to transfer issues to any relevant fork.

Rewinding master removed the issue template and CONTRIBUTING.md. I didn't see the point of adding them, but if anyone has requests, I'm game.

I think it's best to leave the repo open for comments so that people can discuss forks and future plans.

I will monitor the repo but I also encourage the rest of the community to answer bug reports with advice about what version to use, as @jghub did here: #1467 (comment)

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 8, 2020

@gordonwoodhull
thanks a lot for your help so far!

it also is good to leave the repo open for the reasons you state. thanks for this decision, too. :)

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 8, 2020

@aweeraman

so I would prefer another attempt at "keeping it all together". but in view of the fact that the ATT people won't/can't commit to this scenario, yes, forks are the only solution. hopefully, than, the modest number of ksh users around here who care can join forces to do the necessary.

A question that I have is what is the upstream of 93u+ moving forward, that will host these curated defect fixes and enhancements. @jghub are you volunteering to be the curator and maintainer of the pristine 93u+ official fork which distros can rely upon and not consider this project or the community as dead or abandoned?

maybe you can open a new issue "future plans" or something to that extent (this one is closed now...)? other people should have opinions, too :).

regarding your specific question: I have not considered to volunteer for that role so far since I am sure not the best qualified person to do so. if everything else fails: maybe... but I definitely would prefer to do it as a group -- and having at least one package maintainer (the one from debian ;)) included would be a good start I feel...

uqs pushed a commit to freebsd/freebsd-ports that referenced this issue Feb 9, 2020
ksh93u+ and v-. See github commit 0be82553e98be77238577bc0eaafda0f1cf807fe.

To learn how and why our att/ast upstream made this decision see
att/ast#1464 and
att/ast#1466.

The next steps will be to update shells/ksh93-devel to att/ast master.
shells/ksh93 will likely be based on att/ast master at
0be82553e98be77238577bc0eaafda0f1cf807fe or some future tag or branch.


git-svn-id: svn+ssh://svn.freebsd.org/ports/head@525624 35697150-7ecd-e111-bb59-0022644237b5
@marcastel

This comment has been minimized.

Copy link

@marcastel marcastel commented Feb 12, 2020

@jghub

Thanks for your repeated feedback. I believe we are inline on short term priorities and this should reflect if the previously submitted draft roadmap document.

Consequently, if the KornShell is to survive, it needs a community. But more importantly it needs an organisation and a roadmap. This is, at present -- and in my opinion, more important than deciding on the build system, on the source code versioning environment, or even on the channel for classified confidential discussions.

I do not completely agree regarding priorities. or rather, I'd like to see initially only a very modest short-term roadmap agreed upon and a general consensus that the priority in the short run should be to conserve ksh93u+ and ensure that it builds "everywhere" (the latter to be defined ...) so that its availability can easily be guaranteed.

@marcastel

This comment has been minimized.

Copy link

@marcastel marcastel commented Feb 12, 2020

@jghub

Clear roles and responsibilities are essential to give a consistent and coherent view to the outside world. They also allow to establish commitment levels within the team. Most, if not all, participants will have a day job. By clearly identifying roles and responsibilities, everybody has a more precise idea of the time that has to be invested by each participant. We move away from the best effort model for better planning of our activities and deliverables.

We dramatically need this visibility. Recent discussions here -- which we should relocate in a proper discussion thread, demonstrate the urge to clarify the situation for the KornShell. Its very existence as a primary shell on UNIX distributions now being at stake.

We need to clearly identify roles and responsibilities both at the technical level but also at the marketing level to promote KornShell to all major UNIX and derivative distributions.

not now.

@marcastel

This comment has been minimized.

Copy link

@marcastel marcastel commented Feb 12, 2020

@jghub: of course it would be good if ksh93 would have a decent new website pooling historical sources scattered currently over the internet (and partly vanishing already) and up-to-date information like a good FAQ and documentation/help beyond the manpage. but this sounds like a lot of work and would have to wait for now in my view.

Yes documentation is hard work. But I, for one, am ready to invest time and effort to get this, if not done, at least started. Investment being on content not on layout -- the latter can wait. As an example, I very much appreciate the one-man job done by John MacFarlane to maintain Pandoc's documentation and mailing lists. Good documentation means visibility. Visibility means on boarding more people.... hence better chances to keep the KornShell alive 😄

@jelmd

This comment has been minimized.

Copy link

@jelmd jelmd commented Feb 12, 2020

@jghub Actually, I cannot really think of any information/discussions that require a confidental channel here. Wasn't this one piece of all this krader repo mess that he probably also used confidential channels with some third-parties?!? IMHO all future development and discussions should he fully performed in public and through official GitHub channels to prevent the same from happening again...

@jens-maus, I agree with you, but it seems to be an established model used by almost every more or less big SW community. The idea of the private repo is having a closed room, where only all org members (and I really mean all) are able to see, what's going on. All other probably do not care anyway, but well, if the distro maintainers/security officers feel better with it, why not if it works.

At least my intention is to stay on github with the public repos (because of performance and some habits). Only if needed, we would switch over to the private repo and do the same things as in the public repo. If done and thumbs are up, we PR the work into the public repo and have it properly documented/tagged/commit messages assigned. And because if done via repo, we would have an archive of the discussion and and playgrounds to test/submit patches, tests, etc. easily. Nothing gets lost.

I suggested gitlab for the private repo, to have something suitable now. There the number of team members is unlimited. I expect, that at the beginning the org has 5-10 members + sooner or later ksh package maintainers of OS/etc. distros. So a repo limited to 5 or 10 members is not going to work, and I do not like to do all this stuff via [private] mailing lists.

Anyway, I think @jghub is right: we probably will need it very rarely. So is this something you could live with?

@jelmd

This comment has been minimized.

Copy link

@jelmd jelmd commented Feb 12, 2020

@jelmd No worries. The backported patches will be available for everybody to be able to (re)build ksh themselves. Red Hat Enterprise Linux is open source. I was talking about ksh upstream that should coordinate the effort across distributions, like upstreams of other projects do.

@kdudka, My problem here is, that you? released an CVE for ksh and put a link to the patch for the 2020 version, only. IMHO irresponsible and just another attempt to make people think, that 2020 replaced 93u+. Really, really dirty.

@jelmd

This comment has been minimized.

Copy link

@jelmd jelmd commented Feb 12, 2020

@gordonwoodhull, thanks for adjusting ksh2020 releases. Your idea is as usual even better :)

@kdudka

This comment has been minimized.

Copy link
Contributor

@kdudka kdudka commented Feb 12, 2020

@jelmd I think it is a common practice that security advisory contains link to the upstream commit that fixes it. As far as I know, there was no other upstream of ksh when the security issue was handled. But no worries, this is open source. Everybody who has an e-mail account can comment on the bug and attach backported patches for any releases of ksh. Please feel free to do so.

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 12, 2020

@marcastel, my feedback to your roadmap draft:

this is impressively detailed and I appreciate your initiative and thoughts. many things I could agree with, some others: not so much. and other people would have to have their say, of course.

but for very principal reasons (manpower, optimal use of the present momentum for priority short-term goal achievement) I would opt for postponing this roadmap issue for a couple of weeks or so. it is not essential right now (but it will be in the long run!).

I don't know how others see the situation but I see certain parallel's between ksh and Tcl in their respective "ecosystems" (and yes, I am aware that dgk once was hoping for ksh being a "better Tcl" or similar): there are competitors that have 95% (or whatever) of the total user base (bash vs. ksh, python vs. Tcl) and there is no realistic chance (but no need, either!) to change that. both projects lost their creators as project lead at some point.

but Tcl definitely has managed to survive and is actively developed and used -- by a small community.
as far as I understand, Tcl managed that with a "core group" of developers who jointly decide. no benevolent dictator or similar seems necessary or desirable for them. I would be in favour of something similar for ksh for the simple reason that there is nobody available I am aware of that really could act as "the leader" right now (if this would have to be based on much deeper insight into the code, much clearer grasp of necessities etc.). if such a person at some point materialises: new situation.

you also define so many roles that right now we would have fewer candidates than roles to fill... of course, somewhere down the road that can change if we are lucky...

versioning: ksh2020 tried (sort of) a semantic versioning. this now allows to talk of ksh2020 vs. ksh93 which is rather important to distinguish past and future ksh93 from the (seemingly failed, at least right now stalling) ksh2020 effort. so for that very simple reason I would opt for staying with the old versioning somehow and explicitly emphasize the 93u+ lineage. I would not think that releasing some "ksh 1.0" would send the right message to any new user ;). also, this decision can be postponed until one really arrives at something better than a pure bug fix release (whether that ever happens remains to be seen I'd say...).

bottom line:
maybe we should now just "jump", create that ksh organisation, decide on github repo name (your konshell.plus proposal is OK but I would prefer something more specific like KornShell93) and do the fork.

and then my preference would be to define ASAP together an agreed upon shortlist of immediate (and realistic) todos so that the repo can act as the upstream for all distros to provide ksh93u+. than review of additional patches from "everywhere" and integration of these after thorough testing. than a bugfix release. "ksh93u+ (2020-08-28)" could be a birthday present to dgk if the work is done right this time. :)

@marcastel

This comment has been minimized.

Copy link

@marcastel marcastel commented Feb 12, 2020

@jghub. I mostly agree with your remarks. Let us materialise this TODO list. A lot of comments have been made. We now need a GO and a practical location to mature thoughts and ideas, and to start working. In my opinion the where is not so important, and nor is its name as we can change this subsequently. It simply needs to exists rapidly.

Count me as a helping hand where and when needed 😄

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 12, 2020

@jelmd, @jens-maus:
it seems you two are the ones being most suitable to execute the formal/technical setup (organisation+fork/new repo) -- or is someone else around who wants to do it? could you try to sync and propose a definite organisation/repo name here (so that people can object or approve) and than just do it?

regarding naming, @marcastel proposed "kornshell.plus". I would prefer "KornShell93". other suggestions are welcome.

@marcastel

This comment has been minimized.

Copy link

@marcastel marcastel commented Feb 12, 2020

Naming is not of vital importance now as this can be changed in the coming weeks as consensus is gained. However I would encourage not to relate to 'ksh93' in the name. We wish to exist on the long term where the product will be the KornShell not the ksh93 incarnation that marked its end of life at AT&T.

As for the .plus TLD -- despite the eye wink to the release state of software at AT&T, this was mostly opportunistic to reserve an available domain at a reasonable price.

Simply 'KornShell' is a lasting and flexible name. And it is the name given by David Korn. The kornshell GitHub repository is already taken.. interestingly since 2017 and with no activity since. Perhaps a candidate member sharing similar thoughts?

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 12, 2020

I still believe it would be preferable to refer to the '93' in the name. there are decent ksh88 clones out there. there is no ksh93 clone at all. it thus seems to be important to distinguish between ksh88 (clones) and ksh93 (and also against ksh2020, if that project continues) for the foreseeable future or even forever.

"ksh93" is synonymous to "the new Korn shell programming language" syntax as described in the bolsky/korn book. so as long as the syntax is not changed in an incompatible or massive way (like when going from ksh88 to ksh93) there seems no need to ever change the name again.

but what do others think? waiting for further opinions....

@jelmd

This comment has been minimized.

Copy link

@jelmd jelmd commented Feb 12, 2020

@marcastel, just some notes on the go: Awesome work and if we would be 50+ people, thumbs up. Sooner or later we need it definitely, but for how it is IMHO a little bit over the the top. As @jghub said, we all do not have so much time and probably shrug, when reading it, because we know, we are right now 5-10 people, only. So let us start with small steps and than, when the things flow, tackle the other stuff.

So lets get to establish the org and therefore we need a name. I suggested ksh-ast because in german it is a little bit funny (Ast is the german noun for branch) and a reference to its roots as well (and most of us are admins) ;-). Last but not least it is short - IMHO very important for day-by-day work and presentation.

Anyway, the names floating around are in lowercase (i.e. what is visible in URLs) - I assume that you already made sure that we can actually use them:

  • kornshell

  • kornshell.plus

  • kornshell93

  • ksh-ast

  • ksh-community

  • ksh-team

  • ksh93u++

  • kshland

  • kshlang

  • kshreload

  • thekornshell

last list update: 2020-02-12 17:45

If no other suggestions, I think, we should close the list. After it everybody gets (n^2+n)/2 points and may distribute the to each name, but not more than n to a single name. n is the number of elements in the list. The name, which gets the most points wins. Is this ok? If so, lets do it.:)

@marcastel

This comment has been minimized.

Copy link

@marcastel marcastel commented Feb 12, 2020

@jelmd... you forgot kornshell which is the name I advocate and which is how this is called on David Korn's related website.

Please let me insist: 93 is 1993 and hence synonymous of old, legacy, deprecated... which is quite the contrary to what we are trying to achieve. Likewise references to AST or ATT are synonymous to abandoned, stopped, deprecated... again contrary to what a new community would want to achieve. Even though one can do limited object orientation with the Korn Shell language, OO is nots its forte, and a ++ suffix would be more confusing than helpful.

The binary is /bin/ksh and should remain that. Not ksh93. As previously mentioned, we need to clearly mention that this is not endorsed by AT&T. Thats it. And its a very good thing that ksh2020 be called this way... it will be stuck in time... and plain ksh will not.

Plain ksh is already POSIX and registered in /etc/shells. Anything else would need to redo that long winding road. Keeping the name and sticking to the ksh93 "philosophy" avoids such pains.

In conclusion:

  • ksh for the binary name
  • KornShell for the programming language (my vote)

and ksh93 as motto indicating the values and drivers for the community... and for the nostalgic like me who, before the KornShell went public, had horrendeous experiments with simulacres of ksh such as pdksh and the likes -- a brave effort but nothing near compatible with then ksh88.

I understand that @jghub is a fierce defender of the 93 component - and I do agree that ksh93 has been extensively used in literature and on-line articles -- I for one, used primarily that term when at IBM. That component was very important - then, when operating systems were not updated as frequently as today, were primarily proprietary, and where confusion between ksh88 and ksh93 was frequent. These are historic facts of interest to the elders like me.

Younger generations have probably never heard of neither of these and are naturally inclined to use Bash -- not sure that many of them know what the acronym stands for. The 93 component will have no extra value in their eyes. Simple names -- KornLang, KSHLang... or simply KornShell, will have more resonance on the long run.

@jelmd

This comment has been minimized.

Copy link

@jelmd jelmd commented Feb 12, 2020

Well, we are currently working on the org name. And thus having a kind of version prefix or suffix doesn't make much sense to me as well. As an org we can have several repos (step 2). A repo named ksh93 would make sense, since right now our primary goal is/should be to fix existing problems of ksh93, but having a single ksh repo with a ksh93 maintenance branch would be ok either. But one step after another ...

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 12, 2020

@jelmd: I was going to ask that just now: we are only deciding the org name now, right? I partly mixed it up in my head with the best repo name...

so my question now is: is (camel case!) KornShell possible/not yet taken? if not (as it seems) that would be fine with me except the single concern that searching github for "kornshell" still yields too many hits. @jens-maus proposal to make the "ownership" explicit in the name therefore starts to make more sense to me on second thought. so how about the org-name KornShell/community or KornShellCommunity or KornShell-Community (including capitalisation, I'd say)? Since "KornShell" as one word is the name of the language that seems to be a good approach (like "perl-community" or whatever).

@marcastel:
presuming we get the org-name sorted out along the above lines (but I also would be fine with ksh-ast if that were the consensus for the other people expressing their view) the next question would be the repo name. here I think, the project/repo name should be KSH93 (maybe in CAPS to distinguish from binary). I am rather decidedly (but not really fiercely ;)) in favour of this:

the binary should always be ksh, I agree. basically ksh93 defines "the ksh" these days. I of course would not argue for renaming the binary to ksh93...

OTOH, ksh88 clones like mksh can be and partly are used as /bin/ksh on certain systems (my understanding is, that mksh is exactly used in this way in android on all those billions of smartphones...). so /bin/ksh will NOT be guaranteed to be ksh93 on each and every system. this makes, in my view the naming as ksh93 (vs. mksh vs. ksh2020 vs. ksh88 (if still to be found somewhere ;))) important.

so: naming binary ksh: yes. naming the language and organisation KornShell: yes. dropping the 93 when talking/writing about ksh: no. the latter meaning that the new repo/project would state in its README that it is providing KSH93 (and the repo name should reflect that). not just "ksh". can that be agreed upon?

regarding you last concern:

Younger generations have probably never heard of neither of these and are naturally inclined to use Bash -- not sure that many of them know what the acronym stands for. The 93 component will have no extra value in their eyes. Simple names -- KornLang, KSHLang... or simply KornShell, will have more resonance on the long run.

I don't think this is a valid reason to avoid unambiguity in our naming scheme. if they accept python2 or sqlite3, why not ksh93? ;) "KornShell" as one word is good as the language name but not good as the everyday name. no one is using the "Born Again Shell". they all are using "bash". and we would be "ksh93 users" -- which installs as ksh -- but which others might do as well (ksh2020 included...) -- which we can't help.

does that make sense?

@marcastel

This comment has been minimized.

Copy link

@marcastel marcastel commented Feb 12, 2020

@jghub Your arguments are good and I appreciate your conciliating approach; I give up my not so important opinions so as to limit further iterations on this topic.

We will re-open this discussion when we finally get to addressing the POSIX compliance of the AT&T KornShell successor of same name. But that may be quite some time down the line.

I was not aware of the Android "issue". Interesting.

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 13, 2020

@marcastel: attention -- off topic ;)

I was not aware of the Android "issue". Interesting.

I double-checked. here is the "history": https://stackoverflow.com/a/18844732

and I was told in the mksh IRC channel that android uses it as /system/bin/sh. specifically, Android 10 has version "MIRBSD KSH R57 2019/03/01 Android"

mksh is a descendant of oksh which is a descendant of pdksh. mksh is very descent. not fast (but still bash-like...), very small and complete regarding ksh88/POSIX compatibility I believe. plus support of a few ksh93 things like ${ command; }. I have a couple of interdependent shell functions (≈ 1000 LOC) that do a sort of "probabilistic" regexp-based cd stack management (i.e. lookup of dir names to which to cd based on partial name or regexp) initially written for ksh93. I later made them "shell agnostic" so that they also work with bash, zsh, and now mksh. adjustments for bash and zsh were partly tricky or in non-obvious places. mksh was easy. I like it.

@jelmd

This comment has been minimized.

Copy link

@jelmd jelmd commented Feb 13, 2020

@jghub, camel case is possible as well. However the base URL (http://github.com/${entity}/${repo}) is case-insensitive, and many people including me prefer all lowercase. Anyway, I've created issue #1470 to have the name poll on a separate page and put the camel case variants into the list as well.

Please vote by adding your list copy including points to #1470. If someone likes to add other names, just add a note no later than 2020-02-14 00:00:00 UTC. If there are no objection until the same time, voting ends one day later, on 2020-02-15 00:00:00. You may start now. The number of available points and how many one can give max. to a single name is shown at the end of the list.

@bitstreamout

This comment has been minimized.

Copy link

@bitstreamout bitstreamout commented Feb 13, 2020

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 13, 2020

@bitstreamout:
thanks a lot for this clarification (I seem to recall your name from the old times on the ATT ksh mailing list ;)).
I believe to get a good overview over existing sound patches from different sources(?) would be important in the early phase of the planned community effort. so if you can provide competent help there (or want to be directly involved?) that would be great! I am of course aware that early in the ksh2020 project, @siteshwar did some sort of comprehensive patch integration -- but into ksh93v-, not into ksh93u+. so I am not sure whether one could salvage the patch integration from the ksh2020 branch or is better advised to start over with it. any opinion?

I do not know of any contributors list. maybe @lkoutsofios can shed some light on this? and if it does not yet exist I would of course be in favour of starting such a list (better late than never) and put all the names into it which we know for sure to belong there - including you and michael schroeder it seems).

@gordonwoodhull

This comment has been minimized.

Copy link
Member Author

@gordonwoodhull gordonwoodhull commented Feb 13, 2020

Since all of the wiki pages were about ksh2020 I have archived them in a separate repo:

https://github.com/att/ksh2020-wiki/wiki

@lkoutsofios

This comment has been minimized.

Copy link
Contributor

@lkoutsofios lkoutsofios commented Feb 13, 2020

@bitstreamout:
thanks a lot for this clarification (I seem to recall your name from the old times on the ATT ksh mailing list ;)).
I believe to get a good overview over existing sound patches from different sources(?) would be important in the early phase of the planned community effort. so if you can provide competent help there (or want to be directly involved?) that would be great! I am of course aware that early in the ksh2020 project, @siteshwar did some sort of comprehensive patch integration -- but into ksh93v-, not into ksh93u+. so I am not sure whether one could salvage the patch integration from the ksh2020 branch or is better advised to start over with it. any opinion?

I do not know of any contributors list. maybe @lkoutsofios can shed some light on this? and if it does not yet exist I would of course be in favour of starting such a list (better late than never) and put all the names into it which we know for sure to belong there - including you and michael schroeder it seems).

I had copied all the AST public files when Dave and Glenn were leaving and I don't believe there's such a list in there. Dave probably kept all the email interactions he had but that would have been in his personal home directory.

@saper

This comment has been minimized.

Copy link
Contributor

@saper saper commented Feb 13, 2020

I have some patches for things like the INIT package based on 2014-12-24, mostly bug fixes, is there a chance those could be upstreamed here and a new release be done?

@jelmd

This comment has been minimized.

Copy link

@jelmd jelmd commented Feb 13, 2020

@saper, we are currently bootstrapping a new github organization, which will take over ksh maintenance and hopefully gets accepted as ksh upstream. We would be happy to get your patches as well. And certainly there will be a new release, when it is time.

BTW: If you (and everybody else) like, you are welcome to vote for the new org's name on #1470 as well. Voting ends on 2020-02-15 00:00:00 UTC - see details

@bitstreamout

This comment has been minimized.

Copy link

@bitstreamout bitstreamout commented Feb 14, 2020

@jghub

This comment has been minimized.

Copy link

@jghub jghub commented Feb 14, 2020

@bitstreamout :

A good starting point is here https://build.opensuse.org/package/show/shells/ksh and as maintainer had changed in 2014 as I had taken maintenance of systemd before, I add Michael Schröder to carbon copy.

thanks for this pointer.

if he has a github account, we could try to notify him, too, if relevant.

Note that I had switched to 2014-06-26 of ksh93v in July of 2014 but due to the trouble with this version (it was never stable, e.g. used not reentrant calls within signal handlers, as well as ksh93u is much faster) Michael switched back to ksh93u in Jun of 2015. Also he had done a lot of work of

that sure was wise ... (although the "much faster" is not correct in my benchmarks. it is more or less break even (93v- frequently only 10% slower but in other cases distinctly faster). ksh2020 indeed is much slower.)

backporting fixes found in ksh93v as well done many other fixes.

that sounds very interesting, indeed! probably exactly what will be needed here!

Most of those patches will not apply as the formating in ksh2020 had changed a lot AFAICS ...

in fact, ksh2020 descends from 93v-, not from 93u+... and it is not an issue anyway: as you can read in #1464 and in the present thread, the ksh2020 development is discontinued in this repo and master is rolled back to ksh93u+. in the next step we will fork a "community" repo from here and will try to make that a viable upstream for (incrementally patched) ksh93u+.

so if suse/you/michal do have good patches backported from ksh93v- to ksh93u+ (so these would be mostly patches of 93u+ problems from dgf and the original team which they integrated into 93v- I guess?) that would be highly wellcome! and help or active participation from your side would of course be appreciated.

@jelmd

This comment has been minimized.

Copy link

@jelmd jelmd commented Feb 14, 2020

@bitstreamout, @gordonwoodhull, @lkoutsofios, just in case github didn't notify you (because not explicitly mentioned): Voting for the new organization name ends within ~7 hours. So if you like, details are here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.