segfault with extended globs

[jghub]

this code

i=0
v=a
s=
while ((i < 10000)); do
  ((++i))
  s+=$v
done
[[ $s == +($v) ]]

reliably segfaults with current ksh2020, the problem is in this case inherited from ksh93v- (which in turn inherited it, for a change, from ksh93u+, so this is not a regression but an original ksh93 bug I presume).

the extended glob pattern in the final test is causing the segfault.

[krader1961]

Bug reports are great and welcomed. But given your intense interest in a project you've repeatedly claimed has been "hijacked" I'd be more impressed if you actually provided an analysis of the bug and, ideally, a fix. If nothing else you could have provided a backtrace of the failure.

[krader1961]

A backtrace of the failure shows the stack being exceeded. Here are just the last few lines of the backtrace:

    frame #27214: 0x000000010fc020c5 ksh`parserep + 393
    frame #27215: 0x000000010fc00324 ksh`regnexec_parse + 4989
    frame #27216: 0x000000010fc00d20 ksh`regnexec_parse + 7545
    frame #27217: 0x000000010fbfeced ksh`regnexec + 1125
    frame #27218: 0x000000010fc17633 ksh`strngrpmatch + 387
    frame #27219: 0x000000010fb7d411 ksh`test_strmatch + 157
    frame #27220: 0x000000010fbd9904 ksh`sh_exec + 6164
    frame #27221: 0x000000010fbb48cd ksh`exfile + 1437
    frame #27222: 0x000000010fbb543c ksh`sh_main + 1779
    frame #27223: 0x00007fff654307fd libdyld.dylib`start + 1
    frame #27224: 0x00007fff654307fd libdyld.dylib`start + 1

Note the number of call frames. The three frames from #27214 to #27216 repeat until the stack size limit is exceeded. I would argue this is not a bug in the usual sense of an implementation bug; e.g., a buffer overflow or off by one error. But there is a strong argument to be made that it is a design bug in how the ksh regexp engine handles that pattern using recursion. Especially since, in this case, the shortest match is all that is needed and thus no recursion whatsoever is necessary even if recursion is otherwise required to match other inputs.

[krader1961]

Also, if you're using ksh for this type of problem you are using the wrong tool for the job. There are problems where ksh is the best tool. This isn't one of them. Google "if all you have is a hammer."

[jghub]

Google "if all you have is a hammer."

this coming from you... so you can be funny (even if only involuntarily).

but let's leave it at that here: segfault reported. this one of the "111" kind (present in ksh93u+, present in ksh93v-, present in ksh2020). put it on the list.

oh, and maybe count the "111" segfaults and user-facing operational bugs you know of. and than count the bugs in the "011" set (only present in 93v- and 2020 but working in 93u+: I am suspecting this is the largest set) and the :"110" and "010" ones (that would be real fixes in 2020). and the "001" (only buggy in 2020) ones.

[jghub]

you also wrote:

But there is a strong argument to be made that it is a design bug in how the ksh regexp engine handles that pattern using recursion. Especially since, in this case, the shortest match is all that is needed and thus no recursion whatsoever

what you state is wrong in multiple ways:

• we are not talking "regexp" matching here but "extended glob patterns". I am semi-confident that you understand there is a difference and you just used the word "regexp" carelessly. but to be on the safe side this needs to be pointed out.

• obviously, you don't understand extended glob patterns properly. +(a) means "only a's from start to end of the line", i.e. it is equivalent to the regexp ^a+$. so the matcher has to absorb the whole string to decide whether it matches. you can not restrict the match to the leading substring.

• ksh's regexp matcher (which it also supports, in case you don't know) does not have the problem:

i=0
v=a
s=
while ((i < 10000)); do
  ((++i))
  s+=$v
done
[[ $s =~ ^a+$ ]] && print match!

works just fine.

[jghub]

Bug reports are great and welcomed. But given your intense interest in a project you've repeatedly claimed has been "hijacked" I'd be more impressed if you actually provided an analysis of the bug and, ideally, a fix. If nothing else you could have provided a backtrace of the failure.

fake fact fighting cont'd:

• I don't have an "intense interest" in this project. I do have interest in availability of a fully functional ksh93 -- and ksh2020 is becoming something different and inferior in your hands.

• it was someone else who described what has happened to the ast project as "hijack". you might check this and also whether I've adopted and used that term at all afterwards (I don't think so).
  "hijack" is close but I'd rather describe the situation as a "hostile takeover" since the ksh community very obvious and nearly unanimously does not agree with your actions. and "hostile" is a good summary of your overall attitude anyway.

concerning the rest of your comment: nobody wants to impress you. nobody is impressed by you. you received a bug report. react any which inappropriate way you want.

[krader1961]

react any which inappropriate way you want.

Okay, I'm closing this since the problem is due to an algorithm chosen long ago by the old AST team. Their choice of a recursive algorithm for that pattern is a pretty common choice. It is not an implementation bug. We're not going to change the algorithm they implemented since doing so would be incredibly risky with respect to backward compatibility.

@jghub I'll grant that you know more about the behavior of the ksh language than I do (despite my using ksh for more than 15 years). Heck, you might be the leading expert on the planet on the topic of ksh behavior. But you are rude, argumentative, and obnoxious. Please, go away.

[jghub]

I'll grant that you know more about the behavior of the ksh language than I do (despite my using ksh for more than 15 years). Heck, you might be the leading expert on the planet on the topic of ksh behavior.

knowing more about KornShell language than you do is not difficult, unfortunately. so let's say I am just an average "real" ksh user while you are not and your grasp of the KornShell language and the ksh communities' needs is decidedly sub-standard (and manifestly inadequate for maintaining future ksh development).
if you would appreciate (and be willing to respect/follow) expert advice on finer details of core ksh (POSIX) behaviour I'd point you to @mirabilos -- if he could be bothered to suffer your ways gladly.

But you are rude, argumentative, and obnoxious. Please, go away.

I trust people will be able to judge by themselves to whom these attributes apply consistently in the first place.

[jghub]

@siteshwar:

I'd expect that you reopen this issue. a reproducible segfault is a bug. what else? krader is acting and arguing foolishly by closing it. he seems to draw satisfaction from this sort of actions.

I have reported the bug intentionally very concisely and did not want to get into another stupid argument with krader but --- see above. I also made it very clear that this is broken even in ksh93u+ (which is the rare case in this project not the regular one).

I agree with krader that this bug cannot be fixed easily (not by him, that's for sure) and it might be there to stay. in which case it needs a clear documentation in the manpage under KNOWN BUGS/LIMITATIONS in my view (and hint at using regexp as a workaround). but it needs to remain documented in this project as a BUG. I do hope you do agree and will reopen the issue accordingly.

[jghub]

@siteshwar:
I also would request that you break your silence and speak out how you view the state of affairs in this project. are you still caring or have you given up? I feel it is getting more alarming and ridiculous by the day. and there is a single person responsible for this sad situation: krader. it most definitely is not just a problem of "deficient communication skills" as you put it somewhere, I believe. many different people have voiced similar opinions by now. nobody has ever welcomed what krader is doing to ksh.

you know my position:
I believe he should do his thing in a private fork (where he might alter ksh beyond recognition without affecting anybody) but he definitely has no beneficial influence or purpose in the "official" ksh project. so he should not have commit rights. it was a grave error to grant him those rights (but apparently that was not obvious at the time). you should acknowledge that and correct this error.

what needs to happen seems rather clear: get together some sane and competent people and restart from ksh93u+ (presuming that nobody appears that is able to fully understand the ksh93v- code and to complete D. Korn's work to create a "ksh93v+").

"phase one" would be to replace the build system either by this meson/ninja thing (possible) or just by gmake/Cmake (preferable, probably). that is doable and would solve the single central issue of all the package maintainers out there as far as I have understood the situation: make ksh93u+ build easily and maintainable in the different distros with standard build tools.

considering all the breakage that still lurks in the ksh93v- heritage of ksh2020 and the additional booby traps planted by krader's actions of two years (case in point: what has happened regarding reverting the silly changes to cd in #1445? nothing, I bet) , considering the distinctly inferior performance of ksh2020 (#1449) while offering not a single new feature in comparison to ksh93u+ that I am aware of (I know that 93v- had new features but they do not yet work reliably and have already partly been removed by you like the "toupper", "tolower" string processing stuff...), it is very very obvious that ksh93u+ is a better choice as "the future of ksh" than ksh2020.

so the project now first and foremost should concentrate on 93u+ w/o for the time being touching the code at all (beyond what is necessary to make it build with gmake/Cmake).

I believe that almost everybody out there would be rather happy with this situation: ksh93u+ is already good enough for almost all purposes I'd say. but of course it does contain certain bugs (the one I reported here is one example). so "phase two" then would mean to start trying to fix known (or newly reported) real bugs of 93u+. but carefully and thoroughly. it goes without saying that your and krader's work at ksh2020 than could serve as one source where to look for successful fixes to problems that are common to 93u+ and 93v-.

it would be good if this strategy could be agreed on and further interference of krader be prevented.

so, what do you think?