SUBSTR and STRIP Performance Gains

[Peter-Jacob]

After optimising the SUBSTR, STRIP, POS, and LASTPOS functions over the past few weeks, I ran a performance comparison between the current version and a build from early June. The updates focused on faster UTF-8 string handling and expanded support for various types of whitespace beyond simple spaces. Given their frequent use in REXX, SUBSTR and POS are among the most performance-critical functions in the language. The measurements were taken by running each function one million times to ensure consistent and meaningful results.

⏱️ Performance Comparison

Operation	Old Elapsed Time	New Elapsed Time	Improvement Factor
SUBSTR(..., 1, 45)	1.930 s	0.493 s	3.91×
SUBSTR(..., 19, 5)	0.584 s	0.517 s	1.13×
SUBSTR(..., 52, 5)	0.634 s	0.544 s	1.17×
SUBSTR(..., 9, 60)	2.458 s	0.919 s	2.67×
STRIP(..., B) (both ends)	10.585 s	1.867 s	5.67×
STRIP(..., L) (left only)	6.263 s	1.399 s	4.48×
STRIP(..., T) (right only)	5.820 s	0.686 s	8.48×
POS("only", ..., 1)	0.828 s	0.246 s	3.37×
POS("about", ..., 1)	6.209 s	0.316 s	19.64×
POS("about", ..., 70)	1.223 s	0.253 s	4.83×
LASTPOS("only", ..., -1)	5.752 s	0.446 s	12.90×
LASTPOS("tomorrow", ..., -1)	3.477 s	0.443 s	7.85×
Total Time	45.763 s	7.683 s	5.96×

⚡ Overall speedup: ~6× faster

STRIP and POS operations saw the largest gains. Although the performance has improved significantly, I was still surprised by how expensive the STRIP operation remains.

[Peter-Jacob]

This is the REXX I used for the comparison:

count=1000000
elp=0.0

/* Test 1, test substr starting from position 1 */
   mystring="The only limit to our realization of tomorrow is our doubts of today. 😊 think about it!"
   elp=elp+tsubstr(count,mystring,1,45)

/* Test 2, test substr starting from the middle of the string */
   elp=elp+tsubstr(count,mystring,19,5)

/* Test 3, test substr starting near the end of the string */
   elp=elp+tsubstr(count,mystring,52,5)

/* Test 4, test large substr in the middle of the string */
   elp=elp+tsubstr(count,mystring,9,60)

/* Test 5, strip white spaces */
   mystring="         The only limit to our realization of tomorrow is our doubts of today. 😊 think about it!          "
   elp=elp+tstrip(count,mystring,'B')
/* Test 6, strip left white spaces */
   mystring="         The only limit to our realization of tomorrow is our doubts of today. 😊 think about it!          "
   elp=elp+tstrip(count,mystring,'L')
/* Test 7, strip right white spaces */
   mystring="         The only limit to our realization of tomorrow is our doubts of today. 😊 think about it!          "
   elp=elp+tstrip(count,mystring,'T')

/* Test 8, test pos function near beginning */
   mystring="The only limit to our realization of tomorrow is our doubts of today. 😊 think about it!"
   elp=elp+tpos(count,'only',mystring,1)

/* Test 9, test pos function near the end */
   elp=elp+tpos(count,'about',mystring)

/* Test 10, test pos function near end, but with offset */
   elp=elp+tpos(count,'about',mystring,70)

/* Test 11, test lastpos function near beginning */
   mystring="The only limit to our realization of tomorrow is our doubts of today. 😊 think about it!"
   elp=elp+tlpos(count,'only',mystring)

/* Test 12, test lastpos function near the end */
   elp=elp+tlpos(count,'tomorrow',mystring)

/* Summary of tests */
   say "Summary "
   say '>>> elapsed time 'elp' seconds'
exit 0
tSubstr: procedure=.float
  arg maxcount=.int,mystring=.string,from=.int,len=999
  start=time('us')
  do i=1 to maxcount
     result=substr(myString,from,len)
  end
  elp=(time('us')-start)/1000000
  say 'SUBSTR('mystring','from','len')'
  say '>>> elapsed 'elp' seconds, result="'result'"'
return elp

tStrip: procedure=.float
  arg maxcount=.int,mystring=.string,mode="B"
  start=time('us')
  do i=1 to maxcount
     result=strip(myString,mode)
  end
  elp=(time('us')-start)/1000000
  say 'STRIP('mystring','mode')'
  say '>>> elapsed 'elp' seconds, result="'result'"'
return elp

tpos: procedure=.float
  arg maxcount=.int,search=.string,mystring=.string,offset=1
  start=time('us')
  do i=1 to maxcount
     result=pos(search,myString,offset)
  end
  elp=(time('us')-start)/1000000
  say 'POS('search','mystring','offset')'
  say '>>> elapsed 'elp' seconds, result="'result'"'
return elp

tlpos: procedure=.float
  arg maxcount=.int,search=.string,mystring=.string,offset=-1
  start=time('us')
  do i=1 to maxcount
     result=lastpos(search,myString,offset)
  end
  elp=(time('us')-start)/1000000
  say 'LASTPOS('search','mystring','offset')'
  say '>>> elapsed 'elp' seconds, result="'result'"'
return elp

[rvjansen]

Hi Peter, unfortunately, lots of other testcases fail now on macOS: 13 - db_decimal_tests (Failed) 18 - ts_c2x-rxvm (Failed) 19 - ts_c2x-rxbvm (Failed) 30 - ts_d2x-rxvm (Failed) 31 - ts_d2x-rxbvm (Failed) 34 - ts_date2-rxvm (Failed) 35 - ts_date2-rxbvm (Failed) 40 - ts_format-rxvm (Failed) 41 - ts_format-rxbvm (Failed) 44 - ts_left-rxvm (Failed) 45 - ts_left-rxbvm (Failed) 58 - ts_right-rxvm (Failed) 59 - ts_right-rxbvm (Failed) 72 - ts_translate-rxvm (Failed) 73 - ts_translate-rxbvm (Failed) 92 - ts_x2c-rxvm (Failed) 93 - ts_x2c-rxbvm (Failed) 94 - ts_x2d-rxvm (Failed) 95 - ts_x2d-rxbvm (Failed) 125 - arrays_test (Failed) ... even more than last time, when I had a mail in draft that did not make it out (and after the next commit they were normal again). Reason I did not send it was that I also wanted to check Linux - and got sidetracked more than once. So I will check that after sending this one. best regards, René.

…

On 28 Jul 2025, at 09:23, Peter-Jacob ***@***.***> wrote: After optimising the SUBSTR and STRIP functions, I ran a performance comparison between the current version and one from early June. The updates were made over the past four weeks and include faster UTF-8 string handling as well as improved support for various types of whitespace beyond simple spaces. ⏱️ Performance Comparison Operation Old Elapsed Time New Elapsed Time SUBSTR(..., 1, 45) 2.038 s 0.496 s SUBSTR(..., 19, 5) 0.720 s 0.507 s SUBSTR(..., 52, 5) 0.716 s 0.544 s SUBSTR(..., 9, 60) 2.566 s 0.939 s STRIP(..., B) (both ends) 11.501 s 1.896 s STRIP(..., L) (left only) 6.659 s 1.445 s STRIP(..., T) (right only) 6.443 s 0.712 s Total 30.643 s 6.539 s ⚡ Performance improved ~4.7× overall. STRIP operations saw the largest gains. Although the performance has improved significantly, I was still surprised by how expensive the STRIP operation remains. — Reply to this email directly, view it on GitHub <#470>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC3WJQOZNEPIYLY3FXFL2AD3KXFVTAVCNFSM6AAAAACCQBXCUKVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZYGY2DINZXGA>. You are receiving this because you are subscribed to this thread.

[Peter-Jacob]

sorry to hear, I will have a look!

[rvjansen]

yes, also Linux (and all the same testcases)

…

On 28 Jul 2025, at 11:33, Peter-Jacob ***@***.***> wrote: sorry to hear, I will have a look! — Reply to this email directly, view it on GitHub <#470 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC3WJQJ42XES62ZRCIOFTT33KXU7JAVCNFSM6AAAAACCQBXCUKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGOJQHAYTMMA>. You are receiving this because you commented.

[Peter-Jacob]

My apologies — I didn’t fully think some changes through to the end. All errors have now been corrected, except:
 18 – ts_c2x-rxvm (Failed)
 92 – ts_x2c-rxvm (Failed).
A decision is still needed on how to handle these.

[rvjansen]

Hi Peter, really no apologies required. Great that they are fixed. So - I thought we also had these tests clean at one moment? best regards, René.

…

On 28 Jul 2025, at 14:09, Peter-Jacob ***@***.***> wrote: My apologies — I didn’t fully think some changes through to the end. All errors have now been corrected, except:  18 – ts_c2x-rxvm (Failed)  92 – ts_x2c-rxvm (Failed). A decision is still needed on how to handle these. — Reply to this email directly, view it on GitHub <#470 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC3WJQKQJXGDSCEVQUTVSCL3KYHJHAVCNFSM6AAAAACCQBXCUKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGOJQHE3DMMY>. You are receiving this because you commented.