Skip to content

Cookbook

Jump to bottom Edit New page
Sean DeNigris edited this page May 28, 2021 · 67 revisions

NB. This wiki is in the process of being migrated to GT. During this process, sections are removed as they are migrated, but when the migration is complete, they will all be restored here via export.

TOC

  1. Commands
  2. Contributing
  3. Fuel
  4. Docker
  5. Email Addresses
  6. Encryption
  7. (u)FFI
  8. Host Window Customization
  9. JSON APIs
  10. Libraries of Interest
  11. Markup (HTML/XML)
  12. Metamodel (i.e. Ring)
  13. PDFs
  14. Playground Pages
  15. Processes, External
  16. Underscore Assignment
  17. Visualization
  18. Web Stacks
  19. Wikis (Other of Mine)
  20. Zinc

Commands

Pharo uses the Commander library. See its README for an overview and here is an example usage.

Menu Items - How to Create

  1. Inherit from the most appropriate subclass of CmdCommand in the image. For example, if your command works on method(s), you will use SycMethodCommand.
  2. Use existing commands for inspiration, but two likely methods you'll implement are #execute and class-side #canBeExecutedInContext:
  • NB. Be conscious of situations where multiple items may be selected
  • Example: Magritte-GT's ClyTagInspectorExtensionCommand

Contributing

I.e. to External (i.e. non-personal) Projects

Warning: WIP. I am learning as I go. This section is very shaky and untested at the moment; just a collection of things I need to test out.

Setup

When contributing code to shared/public projects, there are three possible starting points (in ascending order of complexity):

1. Project Not Loaded i.e. not already in your image

Example: Any random Pharo-based project on GH

Solution

  • Fork (if you haven't already)
  • Load your fork via Metacello API; it might be good to specify a release tag as a commitish if the project uses them (but release tags are not synced to forks)

2. Project Loaded, But Iceberg Repo Missing Local

Example: Pharo itself

  • Problem: Because the image already knows what commitish you have, you can clone your fork and it will figure out a path from there to the checked out commitish
  • Solution: Re-Clone repo from your fork. Iceberg -> Repair repository -> clone again

3. Project Loaded, But No Iceberg Repo

Example: Iceberg

Problem

Since there is no Iceberg repo, you don't know what commitish you have loaded

Solution

  1. Add repo for upstream (not fork as above)
  2. Determine commitish loaded (e.g. for Iceberg, look in BaselineOfIDE>>#loadIceberg
  3. Tell Iceberg that the commit above represents what is currently loaded:
    1. It seems you can now just "Adopt Commit" from within the tool
    2. If that doesn't work, the "old" way was:
      1. Checkout commitish e.g. for release, Iceberg -> Repository -> Right click on committish on left pane -> Checkout tag
      2. When merging, DO NOT LOAD PACKAGES...

Once you have an Iceberg repository with a commitish loaded (even if the head is detached), all paths converge to follow these steps:

  1. Backstop branch (optional): Iceberg -> Repair repository -> create new branch pointing to image commit
  2. Issue branch: In Iceberg, if issue GH -> New branch for issue; otherwise just new branch
  3. Commit - cherry pick if needed
  4. Push to fork - FYI it's normal to show lots of commits here
  5. PR - may have to choose correct base, but can always be changed later based on feedback from maintainers

Questions

  • What happens if you just load your out-of-date fork in #3 and then follow Marcus's steps for Pharo

References

Fuel

Migration

One hurdle is that you can not serialize/materialize between Fuel versions. Thus, if you are migrating from Pharo 5 with Fuel version 1.9.3 to Pharo 6 or 7 with Fuel version 1.9.4, you have two initial options: upgrade Fuel in the source image or downgrade in the target image (or both) until you find a version that works in both images. If the above fails, for example because too many Pharo versions have been released (e.g. porting from Pharo 3 to Pharo 6), you may be able to migrate in steps across several versions (if your project loads in all of them!), but it would probably be easier to use a different library, like STON.

Upgrading/Downgrading

Installation

Varies by Pharo platform. See here for options.

Pharo Default Fuel Versions

Pharo Version Fuel Default Known Fuel Loadable
3 1.9.3 1.9.4
4-6 1.9.4 ?
7 1.9.4 3.x
8 3.0.0 ?

Handling Class Renames (per Mariano)

materializer
	migrateClassNamed: pointClassName
	toClass: pointClass.

Blocks (and Use by SortedCollections)

The problem with serializing blocks is that they drag in their outerContext, even if the block is clean and doesn't (from the user perspective) need any information from it. If this method's bytecodes have changed, an error will be signaled. For example, (as I reported on the Pharo ML):

CompiledMethod(Object)>>errorSubscriptBounds: 
CompiledMethod(Object)>>at: 
InstructionStream>>interpretNextV3PlusClosureInstructionFor: 
OpalEncoderForV3PlusClosures class>>interpretNextInstructionFor:in: 
...

The issue is likely that you tried to serialize a block (often the #sortBlock of a SortedCollection). As Mariano explains, "clean" blocks (which do not refer to objects outside their scope) are serializable, but with an important caveat:

Fuel does check if a closure is clean or not. If clean, it avoids serializing the whole stack. The way it does this is to simply clean the outerContext of the block to NOT have a sender. See methods CompiledMethod >> fuelAccept: 
So..a clean closure would avoid serializing the sender and the sender of the sender and ... the whole stack. But... at the very least it does need to serialize the outerContext of that closure. That outerContext (even if the block is clean) has a reference to the method in which the closure was compiled... Do you  see how even a clean closure will still serialize ONE compiled method? (#deleteMe in this case). That is the method that could have changed bytecodes (remember that by default, Fuel does NOT fully serialize the methods if they are installed in classes...it will simply serialize the necessary information for looking it up at runtime) and that's where Fuel could bark even for clean closures.

Where the trouble comes is if the serialized method bytecodes have changed (e.g. when upgrading from 32-bit to 64-bit).

Mariano then describes a workaround:

The idea is: always keep the string version instVars and sixx/fuel ignore the block version. block version getter compiles from string if nil. I think you might have understood I use the same instVar and that sometimes I set a string and sometimes I set a closure. If that's the case, then yes, I ended up with 2 different instVars...

Example: Complete with pseudocode:

FaAction class>>#fuelIgnoredInstanceVariableNames
	^ #('actionBlock')


FaPersistentObject subclass: #FaAction
	instanceVariableNames: 'action actionBlock'
	classVariableNames: ''
	package: 'FA-Core'


FaAction>>#actionBlock
	^ actionBlock
		ifNil: [ action isEmptyOrNil
				ifTrue: [ nil ]
				ifFalse: [actionBlock := self compile: action ] ]

Recovering Un-materialize-able Blocks/Methods

If you find yourself in the situation that you have serialized methods which you then can't materialize, maybe this exchange from the Fuel ML will be a starting point to recover your data:

Is there any way to hook in during materialization and say, if the block is clean, recompile it instead of doing the usual?

No, there's no way to hook in there but you could modify the methods beforehand, either manually of by compiling from the command line, for example. Without having tested it, you would do something like this:

Modify FLGlobalCompiledMethodCluster>>materializeInstanceWith: so it doesn't raise an error when the byte codes have changed. That will make it possible to load the methods and associated blocks. Now it gets trickier as you might have to adjust the size and content of the block closure if the compiled method has changed. You can do that in FLVariableObjectCluster>>materializeInstanceWith: to modify the size of the block and in FLVariableObjectCluster>>materializeReferencesVariablePartOf:with: to change the contents. To know what you have to put in you'll have to look at the differences between the old and new methods and the byte codes they generate. You can, of course, just replace the materialized block with one that you compile on the fly too.

Remember to never change what is read from the decoder, even when you don't need the result, as you'll end up a wrong offset.

After having said all that, I belive your problem should be fixed by simply removing the FLMethodChaned error because what you'll get then is just a BlockClosure whose outer context is the compiled method that was found on the class, regardless of what that looks like. Now, this may still break your debugger but you can inspect the graph without problems.

As far as I remember there are no replacement hooks on Materialization (only on serialization). In any case, how do you know the serialized block/method would have the source code? You know, most of the times the CompiledMethod just has a pointer to sources/changes file....unless it encodes the source in it (it was able to do that some time ago, but it was not the default).

In the longer term, I’m wondering if this process could/should be the default for clean blocks? Or maybe Fuel could replace them with FullBlockClosures (would that even work/help)? It seems to me that for clean blocks - and maybe most blocks even if not clean - the user doesn’t care about the outer context, which is just an implementation detail, as long as referenced literals are copied. That said, I’m definitely not an expert in this area :) First, here we're actually talking about method contexts, not (only) compiled methods.

The thing is we don't know anything about the context. We don't know that entry 5 corresponds to the variable named 'myvar' and the position of that variable may have changed (or it may have been deleted) so we would risk assigning the value to the wrong variable. So to achieve this (which I'm not sure is possible) we would need to record meta information on the variables. Actually, we would probably have to build an AST because we don't know that two compilers will generate the exact same byte code, e.g., due to optimizations (inlining, for example). Then, at least we could probably keep the variables that still exist and fill the others with nil.

In summary, having to deal with closures is cumbersome :). For clean blocks, ignoring the fact that

Misc

Docker

Email Addresses

There is no domain object, but there is the ability to parse a string list of addresses into a collection of strings representing the individual addresses, like so:

MailAddressParser addressesIn: aString

Encryption

Libraries

  • PasswordCrypt - A Pharo Smalltalk FFI to sha-crypt for SHA-256/512-based password hashing.

Blog Posts

ML Threads

uFFI

Determining a library's dependencies

From this GH issue:

  • Linux: ldd file.so
  • MacOs: otool -L libvlc.dylib

Pointers

Research:

Host Window Customization

  • Change Title: DisplayScreen hostWindowTitle: 'Hi there'
  • Change Icon (Linux-only): DisplayScreen hostWindowIcon: (FileLocator imageDirectory / 'icon1.ico') fullName.. For other platforms, you get the cryptic message: Error: File existing but not found by the VM sorry, try another location.... Vincent Blondeau explained in the Discord Pharo development channel on 10/20/2019 that he implemented in Linux because that's where he works.

JSON APIs

Libraries of Interest

Logging

Object Graph Traversal

Cool (Misc.)

Markup (HTML/XML)

HTML

Creation/Generation Strategies

Smalltalk DSL

The advantage of this method is that all work is done in a Smalltalk image, with all the normal advantages. For example, ZnHtmlOutputStream comes standard with Pharo:

ZnHtmlOutputStream streamContents: [ :s |
	s tag: 'body' attributes: {'width'. '10'} do: [
		s tag: 'div' attributes: {'style'. 'padding:0px'} with: 'my text' ] ]

Template

The advantage is that you can use standard web dev tools (e.g. Dreamweaver). This is probably more important when you have a team where different members have coding vs. web design responsibilities

Import

There are several libraries, including:

  • Soup, which is good for HTML that you don't control because it deals gracefully with malformed markup.
  • XMLParser and friends.
  • Here's a Squeak-Dev ML thread discussing some options and their tradeoffs.

Remove Extraneous Whitespace (i.e. empty nodes)

  • XMLParser: Send #removeAllFormattingNodes. See ML for rationale on why these nodes exist in the first place.

Pipeline Idea

Can we write simple glue code (e.g. a visitor) to generate ZnHtmlOutputStream DSL from an HTML model (e.g. Soup)?

Pillar & Microdown

Pillar is an extensible rich text model, which includes a markdown syntax. See here for more info. Microdown is a subset of Markdown which is parsable into a Pillar model and can be used for rich text class comments.

Metamodel

One can use Ring to work with code that would not load cleanly into a Pharo image (e.g. missing classes). Pavel shared this script on Discord #general 12/2/19 @ 1:41pm EDT:

repoName := 'pharo'.
repo := IceRepository registry detect: [ :e | e name = repoName ] ifNone: [ nil ].
commit := repo headCommit.
commit packageNames.
env := RGEnvironment new.
commit packageNames do: [ :packageName |
    | snap |
    snap := commit snapshotFor: (RPackage named: packageName).
    snap importInto: env asPackageNamed: packageName ] displayingProgress: 'loading'.

env clean.
env browse.

And here's an example of rewriting the Ring model:

class := RGClass named: 'MyClass'.
class package: (class environment ensurePackageNamed: 'MyPackage').
class compile: 'myMethod
    self assert: 1 = 2' classified: 'accessing'.

method := class >> #myMethod.
parseTree := method parseTree.
r := RBParseTreeRewriter new.
r replace: '`@receiver assert: `@arg = `@arg2' with: '`@receiver assert: `@arg equals: `@arg2'.
(r executeTree: parseTree) ifTrue: [ class compile: parseTree formattedCode classified: method protocol].
class environment browse.

PDFs

External Libraries [1]:

  1. http://forum.world.st/LiteratureResearcher-where-graphs-PDFs-and-BibTex-happily-live-together-tp5005883p5012110.html

Playground Pages

  • To name a page, double-click on the tab label
  • The only way to access is currently via Spotter (per Doru)

Processes, External

OSProcess

Interesting ML Threads:

PIDs

Sometimes you need to know the pid of a long-running process. Let's say we want to run Jenkins. The problem with the standard idiom:

aString := '/usr/bin/java -jar "/path/to/jenkins.war'.
p := PipeableOSProcess command: aString.

is that p is a shell process that launched java, not the java process itself.

Solutions

ProxyPipeline (via David T. Lewis)

This looks like the first choice for most situations

CommandShell itself uses a ProxyPipeline to represent one or more process proxies from a command line, so you can use that class directly to evaluate a command line:

ProxyPipeline command: '/bin/sleep 10'

Total Control

For example, if you need/want to do your own argument parsing, see PipeableOSProcess - The Long Way

Other (Worse) Solutions

Ugly Workaround using Shell Magic
  p := PipeableOSProcess command: '/usr/bin/java -jar "/path/to/jenkins.war" & { echo $! ; }'. 
  p output lines first.
Using exec (via Norbert Hartl)

You should be able to exec your command. An exec /usr/bin/java -jar "/path/to/jenkins.war" (if it works) will replace the shell with the execution of the java process.

The downside, per David T. Lewis, is:

it may a bit confusing because the process proxy will think that it is running a shell even though that shell has done an exec to replace itself with the /usr/bin/java executable.

Using CommandShell (via David T. Lewis)

One way to avoid this confusion is to do the unix shell processing in Smalltalk rather than relying on an external unix shell. That is what CommandShell does, so you could run the /bin/sleep program like this and let CommandShell do all the work that would otherwise be done by /bin/sh. CommandShell command: '/bin/sleep 10' But this is still not quite right, because CommandShell will also open a user interface window, and that is not what you want.

UnixProcess (via Yanni Chiu)
process := UnixProcess 
	forkJob: job 
	arguments: args 
	environment: env 
	descriptors: nil. 
	^ process pid

Note that no "sh" is run, so you may need to set some environment variables yourself.

PipeableOSProcess - The Long Way

This gives you total control (at the obvious cost of LOC!). Reasons you may want to use this:

  • If you need/want to do your own argument parsing (e.g. a path with "can't" in it can get complicated to escape for both Smalltalk and the shell (i.e., can\''t)
  • If you need/want the pid of a (esp. long-running) child process. See PIDs 
	env := CommandShell new environment.
	pwd := '/'.
	args := Array
		with: '-jar'
		with: '/path/with/another space/jenkins.war'.
	desc := Array with: nil with: nil with: nil.
	p := PipeableOSProcess
		new: '/usr/bin/java'
		arguments: args
		environment: env
		descriptors: desc
		workingDir: pwd
		errorPipelineStream: nil
  1. http://forum.world.st/Running-in-background-with-OSProcess-tp4615534.html

Underscore Assignment

Pharo 2 or earlier must be used because it seems this capability was removed in Pharo 3.0. Here is an example:

packagesToTransform := #('Functions' 'MorphicWrappers')
env := RBBrowserEnvironment new forPackageNames: packagesToTransform.
rule := RBUnderscoreAssignmentRule new.
RBSmalllintChecker runRule: rule onEnvironment: env.
change := RBCompositeRefactoryChange new.
“rule changes inspect”.
change changes: rule changes.
change execute

This was originally explained in the Pharo CollaborActive book, which has moved to here. The original URL, http://book.pharo-project.org, doesn't seem to exist or be navigable via the internet archive

Visualization

Roassal - Updating in Moose 7

Moose 7 loads Roassal packages directly instead of using the baseline, so it is not straightforward to update. There are a few extra steps (described in code comments below):

"Delete existing Roassal2 package"

Metacello new
    baseline: 'Roassal2';
    repository: 'github://ObjectProfile/Roassal2/src';
    load.

"Checkout local master to sync working copy"

Web Stacks

Essentials:

GLORP, ApplicationSecurity, Gettext, integrating with Java, UnifiedFFI, lambda in the cloud, remote access, monit/supervisord, CI/CD with Jenkins and Smalltalk CI, loading tons of dependencies, and how to write a CLI with a CommandLineHandler for solution management. - Phil Back, Discord #offtopic 2/12/18 1:17pm

Bootstrap

Login

Pattern from Michael Gorsuch

I have been building my sites with a subclass of WATask as the root component. I implement the workflow in it's 'go' method.

The first task in my workflow is to build a new Welcome component (subclass of WAComponent) and #call it.

The Welcome component has two child components that it displays during #renderContentOn. One child is the registration form, the other is the login form. When these components were initialized, I defined onAnswer for each one, so that when they #answer, it sends it back to Welcome, which will then #answer the original #call from my workflow.

Both of those subcomponents validate the input, and if all is well, they #answer with a User object. If something went wrong, the stuff the message into an instance variable called 'errors' that is always checked when the components are rendered.

When one of the forms suceeds, and the user object is sent back with #answer, I then store it in my session (I subclassed WASession and added an instance variable called 'user').

Pier?

Pier has a security system which is based upon an internal list, and there is an extension to it that enables authentication from some external table. (I think the package is called Pier-ExternalValidation)

JS

Resources

Wikis (Other of Mine)

Zinc

Non-ASCII URLs

Characters like diacritics are technically not allowed in URLs and must be encoded. Unfortunately, because Zinc's parser (e.g. via #asUrl) enforces this, one can't safely parse arbitrary URLs one finds, say, in a browser. Various workarounds have been discussed. The most general has been included in my Pharo-Enhancements project, which was adapted from code posted by Sven on the ML.

Alternatively, passing the URL through FileSystem first (via Sven) may result in an incorrectly parsed URL, even though it may "work" for some uses:

'http://myhost/path/with/umlaut/äöü.txt' asFileReference asUrl.

aZnUrl + a Compound Path String

The two easiest options are aString addedToZnUrl: url or the convenience method url / aString, both of which return a copy of the URL (see this Pharo Users ML thread for motivation)

Add a custom footer
Add a custom sidebar
Clone this wiki locally