Skip to content

Contributing

Jump to bottom Edit New page
Chris Watkins edited this page Jun 9, 2022 · 123 revisions

NOTE: If anything on this page doesn't work for you, please file a bug!

(Related: Oil Native Quick Start / Where Contributors Have Problems / Oil Dev Cheat Sheet / Oil Dev Tips)

Requirements

You should use Linux to make changes to OSH and test them. One reason for this is that the spec tests run against four other shells (bash/dash/zsh/mksh), and we don't have a way of building the right versions of these shells on other OSes.

When I'm on a Mac, I use Ubuntu on Virtualbox to develop.

  • Note that OSH does work on OS X for end users, but development is a different story.
  • 3/2020 update: there's a shell.nix in the root of the repo that may work on OS X. Ping us on Zulip if you need help with it. (Developing Oil With Nix)

Quick Start

Note that this is the dev build, which is very different than the release build, due to Oil's use of code generation! (In particular, don't run ./configure when developing Oil.)

git clone https://github.com/oilshell/oil.git    # or use your own fork

cd oil                    # or enter the base directory

git submodule update --init --recursive # get dependencies

build/dev.sh ubuntu-deps  # Python 2 dev headers, etc.
                          # Uses sudo apt-get, so it's for Ubuntu/Debian.
                          # may need python2-dev instead of python-dev
build/dev.sh minimal      # build help, native/libc.c Python extension

bin/osh                   # sanity check to make sure it works. 
                          # Make sure you have Python 2 installed.
                          # Ctrl-D to quit.

test/unit.sh unit frontend/lexer_test.py     # Run an individual unit test
test/unit.sh minimal      # run most unit tests, takes about a second or two

test/spec.sh smoke        # run a single file; there are more but this is a good start

After you've run these tests, you're ready to work on Oil! Details below.

Running Tests

There are three kinds of tests: unit tests, spec tests, and "wild tests".

Unit Tests

You can run the minimal tests after a minimal build:

$ test/unit.sh minimal

You can run a single test like so:

$ test/unit.sh one osh/word_parse_test.py

Running the full suite of tests requires a full dev build:

$ build/dev.sh all
$ test/unit.sh all

Notes

  • Unit tests are written in Python.
  • unit.sh is a simple wrapper that sets PYTHONPATH.

Spec Tests

Spec Tests test Oil against specific versions of existing shells via the test/sh_spec.py framework, and require a bit more setup. Ping us on Zulip if you have problems.

First, set up the spec tests as described in test/spec-bin.sh and Spec Tests:

$ test/spec-bin.sh download     # Get the right version of every tarball                                                                                     
$ test/spec-bin.sh extract-all  # Extract source                                                                                                             
$ test/spec-bin.sh build-all    # Compile                                                                                                                    
$ test/spec-bin.sh copy-all     # Put them in _deps/spec-bin                                                                                                 
$ test/spec-bin.sh test-all     # Run a small smoke test

Now you can run the spec tests.

$ test/spec.sh osh-all     # all OSH tests in parallel
$ test/spec.sh oil-all     # all Oil tests in parallel
$ test/spec.sh smoke       # a single file -- look at the list of functions

Output is in _tmp/spec.

Cheat Sheet

See Oil Dev Cheat Sheet for a quick summary of these commands.

Other Test Suites

There are many other test suites which are reported on:

  1. The /release/$VERSION/quality.html page on every release
  2. The "Soil" continuous build at http://travis-ci.oilshell.org/github-jobs/

For example, you can run

test/parse-errors.sh all

To see a big demonstration of parse errors. I use this to supplement the spec tests when changing the syntax of the language.

Running build/dev.sh all

I left a few things out of build/dev.sh minimal to make it easier to get started:

  1. It doesn't generate a lexer with re2c.
  2. It doesn't generate help text, which depends on CommonMark.

To install re2c:

soil/deps-tar.sh layer-re2c

# smoke test:
$ ../oil_DEPS/re2c/re2c --help

To install CommonMark (cmark) and smoke test generating HTML from Markdown:

soil/deps-tar.sh layer-cmark

# smoke test
doctools/cmark.sh demo-ours

Three Kinds of Builds

(1) What I've documented above, i.e. running bin/osh, can be called the developer build. This builds native/libc.c and uses the system Python interpreter /usr/bin/env python2 to run Oil.

(2) The "production" build is split into two steps: (a) git repo -> release tarball, and (b) release tarball -> Oil app bundle.

2a. Release tarball. This step is architecture-independent and occurs on the developer machine. This process may change, but it's roughly as follows:

# First we need to build a raw Python interpreter, in order to dynamically discover Python dependencies.
build/dev.sh ubuntu-deps        # or equivalent for other distros
build/dev.sh yajl-release       # build the git submodule (TODO: perhaps remove this)
soil/deps-tar.sh layer-cmark
soil/deps-tar.sh layer-re2c
soil/deps-tar.sh layer-cpython
build/dev.sh all
devtools/release.sh quick-oil-tarball          # build _release/oil.tar with the Makefile and build/*.{sh,py}
build/test.sh oil-tar           # test that you can build the tarball

Notes:

  • This step is likely to be Linux-centric for awhile. The end-user build, step 2b, should not be Linux-centric. Ideally it would only require a POSIX environment.
  • The full release process is documented at the top of devtools/release.sh. Those comments are kept up-to-date.
  • After you've done the steps above, you can also try devtools/release.sh quick-oil-tarball to quickly make tarballs for testing on other machines. (It doesn't run any tests, so make sure the code works on Linux first.)

2b. Oil app bundle build. This step is architecture-dependent and occurs on the machine of the system packager or end user. To perform this step, follow the instructions in the INSTALL.txt file at the root of the tarball.

Tips for Developing

  • Spec Tests are very important! Make sure you know how to run them quickly and maintain a fast, iterative development style.
    • Where Do I Put Spec Tests? If I'm going to change the behavior of the source builtin, I do grep source spec/*.test.sh and that leads to the spec test that should verify the new behavior.
  • Debugging Completion Scripts -- Use --debug-file!
  • Debugging OSH / OVM / CPython itself
    • make _bin/oil.ovm-dbg and misc/bin.sh which creates _bin/osh-dbg may be helpful in debugging... although it's not something I do very often!
  • make clean is meant for the end user source tarball and will just remove the binaries in _bin. make clean-repo is more aggressive and removes the "pre-build" in _devbuild.
  • Make sure to build/dev.sh minimal (or all) after you git pull ! The Python extension modules (.so files) that OSH uses may be out of date, and they have to match the Python code.

Pull Requests

Most people have used Github pull requests to send patches. I will use Github's system to make comments. After addressing them, please:

  1. push the commits so I can see them
  2. reply with a message, like "all done", or "I disagree because ..."

If only push commits, I won't look at the PR again, because I'll assume it's in progress.

Code Style

Python code follows Google's Python style guide, which is essentially PEP 8, but with

  • 2 space indents
  • CapWords for function names

There are notable exceptions in the opy/ dir, because I didn't write that code.

The general rule is to follow existing style. (TODO: Should we use an auto-formatter?)

  • For shell code, also follow the existing style. It has 2 space indents, funcs-like-this (unless POSIX shell is required), and vars_like_this.
  • C code generally follows Python style (except 2 space indents?)
  • C++ is formated with clang-format -- see test/lint.sh

OPy Compiler

See opy/README.md. (Note: this will be obsolete when oil-native is done.)

Add a custom footer
Add a custom sidebar
Clone this wiki locally