superDoIt

BRANCH	STATUS
v4.2
v4.1
v3
v3.1
v2
v2.0
v2.1

Versions

v4.2

Development version intended for use with GsDevKit_stone:v2 and GemStone 3.5.3 and newer versions of GemStone. github actions for superDoit test 3.6.4 thru 3.6.6, and 3.7.0. while the GsDevKit_stones github actions tests 3.5.3, 3.5.8, 3.6.0. Note that GemStone either 3.7.0 or 3.7.1 is required for all solo scripts, which allows for supporting versions older that 3.6.4...

v4.1

Stable branch intended for use with SmalltalkCI, GsDevKit_stones:v1.1.x and versions of GemStone that are shipped with an extent0.rowan.dbf in $GEMSTONE/bin (tested from GemStone 3.6.4 thru 3.7.0).

v3

Intended for .solo scripts used with versions of GemStone that are shipped with an extent0.rowan.dbf in $GEMSTONE/bin, GemStone 3.6.4 and newer versions of GemStone (currently tested thru 3.6.5).

.stone scripts may be used with GemStone versions as old as 3.4.0.

v3.1

Stable branch for use by smalltalkCI

v2

Intended for use with older versions of GemStone: 3.6.1, 3.6.0, 3.5.7, 3.5.0 (tested versions)

What is superDoit?

superDoit is a scripting framework for writing shell scripts in GemStone Smalltalk using GemStone Topaz. To install:

	cd <install dir>
  git clone git@github.com:dalehenrich/superDoit.git
	superDoit/bin/install.sh

Current best practices

Current best practices for writing a topaz solo bash scripts to report the sum total size of tranlog files in the given directory involves creating 3 separate files (see Scripting with topaz solo using sh-bang for resons why 3 files are needed to handle command line arguments in topaz solo scripts):

a bash script driver script named gettranlogspace:

#!/bin/bash
export GEMSTONE=/lark1/users/gsadmin/3.6
$GEMSTONE/bin/topaz -lq -I $GEMSTONE/scripts/myini -S
$GEMSTONE/scripts/reporttranlogspace.tpz -- $1

a solo .topazini file named myini:

set user DataCurator pass swordfish
set gemstone gs64stone
set solologin on

a topaz command file named reporttranlogspace.tpz, that contains the Smalltalk code to calculate and report the sum of the tranlog sizes in the directory:

login
run
  | dir files sz |
  dir := System commandLineArguments last.
  files := GsFile 
     contentsOfDirectory: dir
     onClient: false.
  sz := 0.
  files do: [:ea | sz := sz + (GsFile sizeOfOnServer: ea)].
  GsFile gciLogClient: dir, ': tranlogs consume total ', 
     (sz / 1024) asInteger asString, ' KB'.
%
logout
exit

The bash driver script would then be executed:

unix> ./gettranlogspace /lark1/users/gsadmin/tranlogs
/lark1/users/gsadmin/tranlogs: tranlogs consume total 98477 KB

superDoit solution

superDoit not only eliminates the need to create three separate files to run a solo script, but provides support for declaring command line options, help, debugging and more.

Here is the superDoit version of reporttranlogspace.solo:

#!/usr/bin/env superdoit_solo
options
{
  SuperDoitRequiredOptionWithRequiredArg long: 'tranlogDir' short: 't'.
}
%
usage
-----
USAGE $basename [--help | -h] [--debug | -D] --tranlogDir=<tranlog-directory-path>

DESCRIPTION
  Calculate and report the sum of the tranlog sizes in the given <tranlog-directory-path>

OPTIONS
  -t, --tranlogDir=<tranlog-directory-path>
                             path to tranlog directory
  -h, --help                 display usage message
  -D, --debug                bring up topaz debugger in the event of a script error

EXAMPLES
  $basename --help
  $basename -D -t <tranlog-directory-path>
  $basename -t <tranlog-directory-path>
  $basename --tranlogDir=<tranlog-directory-path>
-----
%
doit
  | sz |
  sz := 0.
  self tranlogDir asFileReference files
    do: [:tranlogFile |
      sz := sz + tranlogFile size ].
  self stdout
    nextPutAll: self tranlogDir;
    nextPutAll: ': tranlogs consume total ';
    nextPutAll: (sz / 1024) asInteger asString, ' KB'.
  ^ self noResult
%

Executing ./reporttranlogspace.solo -t $GS_HOME/server/stones/gs_361/tranlogs produces:

/home/dhenrich/rogue/_homes/rogue/_home/server/stones/gs_361/tranlogs: tranlogs consume total 39617 KB

Executing ./reporttranlogspace.solo -h produces:

-----
USAGE reporttranlogspace.solo [--help | -h] [--debug | -D] --tranlogDir=<tranlog-directory-path>

DESCRIPTION
  Calculate and report the sum of the tranlog sizes in the given <tranlog-directory-path>

OPTIONS
  -t, --tranlogDir=<tranlog-directory-path>
                             path to tranlog directory
  -h, --help                 display usage message
  -D, --debug                bring up topaz debugger in the event of a script error

EXAMPLES
  reporttranlogspace.solo --help
  reporttranlogspace.solo -D -t <tranlog-directory-path>
  reporttranlogspace.solo -t <tranlog-directory-path>
  reporttranlogspace.solo --tranlogDir=<tranlog-directory-path>
-----

Executing ./reporttranlogspace.solo produces:

UserDefinedError: The required option 'tranlogDir' was not set.

Executing ./reporttranlogspace.solo -D and entering where at topaz prompt produces:

-----------------------------------------------------
 Near line 62 of file /home/dhenrich/rogue/_homes/rogue/_home/shared/repos/superDoit/gemstone/superdoit.tpz
GemStone: Error         Nonfatal
a UserDefinedError occurred (error 2318), reason:halt, The required option 'tranlogDir' was not set.
Error Category: 231169 [GemStone] Number: 2318  Arg Count: 1 Context : 20 exception : 102530817
Arg 1: [20 size:0 primitiveSize:0 cls: 76289 UndefinedObject] nil
Stopping at line 62 of /home/dhenrich/rogue/_homes/rogue/_home/shared/repos/superDoit/gemstone/superdoit.tpz
topaz 1> where
==> 1 UserDefinedError (AbstractException) >> _signalToDebugger @10 line 8   [methId 101346817]
2 UserDefinedError (AbstractException) >> defaultAction @2 line 18   [methId 101347073]
3 UserDefinedError (AbstractException) >> _defaultAction @4 line 4   [methId 101346305]
4 UserDefinedError (AbstractException) >> _signal @2 line 20   [methId 3833089]
5 UserDefinedError (AbstractException) >> signal @2 line 47   [methId 3848449]
6 SuperDoitExecutionClass (Object) >> error:    @6 line 7   [methId 6400001]
7 [] in SuperDoitExecution >> getAndVerifyOptions @21 line 12   [methId 102538497]
8 Dictionary >> keysAndValuesDo:                @9 line 11   [methId 12241665]
9 SuperDoitExecutionClass (SuperDoitExecution) >> getAndVerifyOptions @10 line 8   [methId 102499841]
10 [] in SuperDoitExecution >> doit              @6 line 5   [methId 102543617]
11 ExecBlock0 (ExecBlock) >> on:do:              @3 line 44   [methId 5850369]
12 SuperDoitExecutionClass (SuperDoitExecution) >> doit @2 line 8   [methId 102546689]
13 SuperDoitDoitCommand >> executeAgainst:       @27 line 19   [methId 102584833]
14 [] in SuperDoitCommandDefinition >> executeAgainst: @7 line 2   [methId 102500609]
15 OrderedCollection (Collection) >> do:         @5 line 10   [methId 3419649]
16 SuperDoitCommandDefinition >> executeAgainst: @3 line 2   [methId 102502913]
17 [] in SuperDoitCommandParser >> parseAndExecuteScriptFile: @22 line 15   [methId 102511105]
18 [] in AbstractFileReference >> readStreamDo:  @8 line 4   [methId 48130817]
19 ExecBlock0 (ExecBlock) >> ensure:             @2 line 12   [methId 5852161]
20 FileReference (AbstractFileReference) >> readStreamDo: @4 line 5   [methId 42354689]
21 SuperDoitCommandParser >> parseAndExecuteScriptFile: @2 line 3   [methId 102520577]
22 SuperDoitCommandParser class >> processInputFile @31 line 22   [methId 102519297]
23 Executed Code                                 @2 line 6   [methId 102553601]
24 GsNMethod class >> _gsReturnToC               @1 line 11   [methId 5673729]
  [GsProcess 102639105]
topaz 1>

Executing ./reporttranlogspace.solo --tranlogDirproduces:

UserDefinedError: Missing required argument for option 'tranlogDir' [b]

Executing ./reporttranlogspace.solo -tproduces:

UserDefinedError: Missing required argument for option 'tranlogDir' [d]

Executing ./reporttranlogspace.solo -t /tmp/oopsproduces:

DirectoryDoesNotExist: Path / 'tmp' / 'oops'

Of course adding -D or --debug to any of the the above, will bring up the debugger and allow you to debug the error yourself.

For the complete list of superDoit file sections see superDoit Sections

solo, stone, and topaz scripts

superDoit scripts come in three flavors: solo, stone, topaz. By convention each type of script is identified by a .solo, .stone, .topaz file extenstion, so that users will know how to run and what to expect from the script.

solo script

solo scripts use a solo login, a special kind of login, which does not require a running Stone. This allows a Gem session to start up and execute GemStone Smalltalk code (based on reading from a GemStone extent file), without any connection to a Stone. Without a Stone, a solo session is single-user, and changes to persistent objects cannot be committed. Solo sessions cannot, for example, run markForCollection in their own environments, nor execute methods that make or restore backups..

solo scripts start with a #!/usr/bin/env superdoit_solo shebang and <path-to-superDot-git-clone>/bin/superdoit_solo needs to be in your path:

#!/usr/bin/env superdoit_solo
doit
^ 3 + 4
%

solo scripts are run against an extent with Rowan v2.2 installed.

See .solo script installation for additional information about installing a topaz solo extent and product tree.

.stone script

The doit section contains the Smalltalk expressions to be executed and is terminated by %:

#!/usr/bin/env superdoit_stone
doit
^ 3 + 4
%

.topaz script

The doit section contains the Smalltalk expressions to be executed and is terminated by %:

#!/usr/bin/env superdoit_topaz
set u SystemUser p swordfish
set gemstone gs64stone
login

run
GsFile stdout nextPutAll: (3 + 4) printString; lf
%

superDoit sections

Here is a complete list of the sections that may be used in a superDoit script:

doit
options
usage
method
instvars
input
method:
classmethod:
customoptions
projectshome
specs
specurls.

doit section

A typical Smalltalk IDE will provide a facility for writing Smalltalk code in a workspace or playground, where you can quickly put together a collection of Smalltalk expressions for evaluation without having to resort to creating a class. It is natural that the simplicity of the workspace be preserved in the superDoit doit section. The doit section contains the Smalltalk expressions to be executed and is terminated by %:

#!/usr/bin/env superdoit_solo
doit
^ 3 + 4
%

The script invokes the superdoit_solo program that arranges for topaz solo to be used to execute the doit.

options section

As with any other scripting language it is an absolute requirement to allow for the declaration of command line arguments. In superDoit, the options section is used to declare a set of command line options for the script:

#!/usr/bin/env superdoit_solo
options
{
  SuperDoitRequiredOptionWithRequiredArg long: 'addend' short: 'a'.
}
%
doit
  ^ 3 + (self addend) asNumber
%

and the framework automatically arranges for a method to be created using the long name of the option. The value of the option from the command line is returned as a String by the method. Here is the full list on command line options that are currently supported:

options
{
	SuperDoitOptionalOptionWithNoArg long: 'noarg'.
	SuperDoitOptionalOptionWithNoArg long: 'noarg' short: 'n'.

	SuperDoitOptionalOptionWithRequiredArg long: 'optional'.
	SuperDoitOptionalOptionWithRequiredArg long: 'optional' default: 'default'.
	SuperDoitOptionalOptionWithRequiredArg long: 'optional' short: 'o'.
	SuperDoitOptionalOptionWithRequiredArg long: 'optional' short: 'o' default: 'default'.

	SuperDoitRequiredOptionWithRequiredArg long: 'required'.
	SuperDoitRequiredOptionWithRequiredArg long: 'required' short: 'r'.
}
%

The SuperDoitOptionalOptionWithNoArg returns true if the option is present on the command line and false if it is not present on the command line. The SuperDoitOptionalOptionWithRequiredArg returns nil or the default value if not specified on the command line or the value of the option as a String if specified on the command line. The SuperDoitRequiredOptionWithRequiredArg signals an error if the option is not present on the command and the value of the option as a String if specified on the command line.

usage section

It is also an absolute requirement that a scripting language provide usage information for the script. The usage section contains structured help text that documents the script usage, terminated by %:

#!/usr/bin/env superdoit_solo
usage
-----
USAGE $basename [--help | -h] [--debug | -D]

DESCRIPTION
  Simple script that returns the result of evaluating `3+4`.

OPTIONS
  -h, --help                 display usage message
  -D, --debug                bring up topaz debugger in the event of a script error

EXAMPLES
  $basename --help
  $basename -D
-----
%
doit
  3 + 4
%

The -h, --help, -D, and --debug options are pre-defined and require no declarations to be included. If you want to change the options used for help and debugging, then you can use the customptions section

method section

I'm sure that all of us have written workspaces that could have benefited from a local method or two and shell scripts are no different, superDoit makes it possible to add local methods to the script. The method section defines a Smalltalk method, terminated by %, that can be called from within the script:

#!/usr/bin/env superdoit_solo
usage
-----
USAGE $basename [--help | -h] [--debug | -D]

DESCRIPTION
  Simple script that demonstrates the definition and use of a script method.

EXAMPLES
  $basename --help
  $basename -D
-----
%
method
three
  ^ 3
%
doit
  ^ self three + 4
%

instvars section

If you are going to allow methods in scripts, then you need to allow instance variables for sharing common state. The instvars section, terminated by % defines script instance variables.

input section

The input section), terminated by % can be used to load GemStone .gs files into the image.

method: section

The method: section), terminated by % is used to define an instance method for a class that is already present in the image.

classmethod: section

The classmethod: section), terminated by % is used to define a class method for a class that is already present in the image.

customoptions section

The customptions section), terminated by % is used to override the default command line arguments: -h, --help, -D, and --debug.

projectshome section

The projectshome section, terminated by % is used to declare the value of the ROWAN_PROJECTS_HOME environment variable during the execution of the script.

specs section

The specs section, terminated by % contains an array of Rowan load specification STON objects used to load external projects into the image.

specurls section

The specurls section, terminated by % contains a list of spec urls that reference the location of a Rowan load specification STON object.

superDoit installation

.solo script installation

download 3.6.1 product tree

cd gemstone/products
../../dev/downloadGemStone.sh 3.6.1

download 3.6.1 solo extent

cd ../solo
curl  -L -O -s -S "https://github.com/dalehenrich/superDoit/releases/download/v0.1.0/3.6.1_extent0.solo.dbf.gz"
gunzip --stdout 3.6.1_extent0.solo.dbf.gz > extent0.solo.dbf
chmod -w extent0.solo.dbf
ln -s ../products/GemStone64Bit3.6.1-x86_64.Linux product

Add superDoit/bin to your PATH:
```
export PATH="$GS_HOME/shared/repos/superDoit/bin":$PATH
```
It is recommended that you add this expression to your .bashrc so that .solo scripts can be run without any special setup.

Examples

simplest executable .solo doit

#!/usr/bin/env superdoit_solo
doit
3+4
%

simple executable .solo doit with default help and debugging options

#!/usr/bin/env superdoit_solo
usage
USAGE $basename [--help | -h] [--debug | -D]

DESCRIPTION
  Evaluate the Smalltalk expression: 3+4 and return the result on stdout.

OPTIONS
 --help | -h
    Display usage message.
  --debug | -D
    Without this option, when an error occurs, the error message is written to
      stdout. If this option is supplied, the topaz debugger will come in the 
      event of an error.

EXAMPLES
  $basename --help
  $basename -h
  $basename --debug
  $basename -D
  $basename
%
doit
	3+4
%

executable .solo doit with methods and Rowan specs

#!/usr/bin/env superdoit_solo
specUrls
https://raw.github.com/dalehenrich/RowanSample9/spec_0002/rowan/specs/spec_0002.ston
%
specs
RwLoadSpecificationV2 {
	#specName : 'spec_0022',
	#projectName : 'RowanSample9',
	#gitUrl : 'git@github.com:dalehenrich/RowanSample9.git',
	...
}
%
usage
USAGE $basename [--help | -h] [--debug | -D]
... and more
%
method
x
^ self y
%
method
y
^ 3+4
%
doit
	self preDoitSpecLoad.	"load specs"
^ self x
%

How does superDoit work?

Rowan and superDoit

Note: As of December, 2021, Rowan v2.2 and Jadeite should be treated as beta/alpha releases:

GemStone upgrade of a Rowan extent is not supported.
The Rowan API is not complete.
Jadeite the Rowan IDE is not fully functional.

However, with that said:

Rowan v2.2. sports a GemStone port of FileSystem which greatly improves the File manipulation API for GemStone, a critical capability for scripting
Rowan v2.2 can be used for loading external projects into a topaz solo stone. For example, an OrderedDictionary .superDoitCI.solo script is used to run Github CI for a port of OrderedDictionary to the GemStone kernel using Rowan

Branch naming conventions

vX
vX.Y
vX.Y.Z or vX.Y.Z-id

vX

Stable production branch.

X is incremented whenever there is a breaking change. vX.Y and vX.Y.Z branches are merged into the VX branch, when development is complete on the feature or patch.

vX.Y

Stable feature/bugfix candidate branch.

Y is incremented whenever work on a new feature or bugfix is started. vX.Y branches are merged into the VX branch when development is complete.

Primary work takes place on a vX.Y.Z branch and the VX.Y.Z branch is merged into the VX.Y branch at stable points, so if you want to have early access to a feature or bugfix, it is relatively safe to use this branch in production.

vX.Y.Z

Unstable development branch.

Z is incremented whenever work on a new feature or bugfix is started. A pre-release may be used to further identify the purpose of the work.

Primary work takes place on this branch and cannot be depended upon to be stable.

Name		Name	Last commit message	Last commit date
Latest commit History 842 Commits
.github/workflows		.github/workflows
bin		bin
dev		dev
examples		examples
gemstone		gemstone
packages		packages
plugins		plugins
rowan		rowan
scriptBin		scriptBin
shunit_tests		shunit_tests
src		src
tests		tests
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md

License

dalehenrich/superDoit

Folders and files

Latest commit

History

Repository files navigation