Skip to content

Dev notes

Jump to bottom Edit New page
Charles Comstock edited this page Dec 12, 2023 · 43 revisions

Random useful notes for developing quil itself.

Release process

  1. Run lein do check, cljfmt check and verify that there are no reflection warnings and that everything is formatted.
  2. Run "automated" tests: lein test. Watch out for exceptions in console. If you want to run tests one by one manually proceeding to the next test - use MANUAL=true lein test.
  3. Run manual Clojure tests: lein test :manual.
  4. Run manual ClojureScript tests: lein with-profile cljs-testing do cljsbuild once tests, ring server. Run both "automated" and manual tests. In Chrome, Firefox and ideally IE.
  5. Update version in project.clj.
  6. Update RELEASE-NOTES.md, filling all changes which went to the release.
  7. Update Quil version in README.md.
  8. Change version, commit, create git tag with new version.
  9. Upload to clojars.
  10. Update quil.info website.
  11. Update templates.
  12. Make an announcement on clj-processing and possibly clojure google groups.
  13. Update Quil version in third-party sites/pages. Check list below.

Announcement template

Subject:
[ANN] Quil $VERSION Release
Body:
Happy to announce Quil 2.2.5 release.
Quil is a Clojure/ClojureScript library for creating interactive drawings and animations.

The release available on clojars: https://clojars.org/quil. List of changes:

  • Change 1
  • Change 2

Documentation on http://quil.info has been updated as well.

Happy hacking!
$YOUR_NAME

Places to update Quil version

How to update quil when new version of Processing is released

Unfortunately Processing doesn't use maven so we can't just update versions in project.clj and we have to upload Processing jars to clojars ourselves.

Uploading Processing jars

1 - Download new version of Processing, let it be 1.2.3.
2 - Create temp folder and copy 3 Processing jars there:

  • cp core/library/core.jar temp/processing-core-1.2.3.jar
  • cp modes/java/libraries/pdf/library/pdf.jar temp/processing-pdf-1.2.3.jar
  • cp modes/java/libraries/dxf/library/dxf.jar temp/processing-dxf-1.2.3.jar
  • cp modes/java/libraries/svg/library/svg.jar temp/processing-svg-1.2.3.jar

3 - Create temp/pom.xml like this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>quil</groupId>
  <artifactId>processing-core</artifactId>
  <packaging>jar</packaging>
  <version>1.2.3</version>

  <licenses>
    <license>
      <name>GNU Lesser General Public License</name>
      <url>https://www.gnu.org/licenses/lgpl.html</url>
    </license>
  </licenses>
</project>

Check that <version>1.2.3</version> is correct version of Processing.
4 - Upload all jars one by one to clojars using following command:

lein deploy clojars quil/processing-core 1.2.3 processing-core-1.2.3.jar pom.xml

Be sure you modified <artifactId> entry in pom.xml before uploading pdf.jar and dxf.jar. More on pushing manually to clojars: https://github.com/ato/clojars-web/wiki/Pushing#ssh
5 - Check if jars were uploaded successfully:

6 - Modify project.clj to use new jars:

:dependencies [...
               [quil/processing-core "1.2.3"]
               [quil/processing-pdf "1.2.3"]
               [quil/processing-dxf "1.2.3"]
               ...]

Updating jogl and gluegen-rt jars

It is very likely that new version of Processing will use new jogl and gluegen-rt libs. Jogl/gluegen-rt guys use maven so in theory we could just use their maven libs. But no... Unfortunately uberjars built from this libs won't work correctly because of native libs loading. There is a special page describing how to create jars containing native libs in it: https://jogamp.org/wiki/index.php/JogAmp_JAR_File_Handling. We decided to go with Fat-Jar option. Fat jar is a single jar that contains all native libs using following structure (example):

fat.jar
  natives
    linux-amd64
      lib1.so
      lib2.so 
    linux-i586
      lib1.so
      lib2.so
    windows-amd64
      lib1.dll
      lib2.dll

Undoubtedly it increases size of jar. But in case of uberjar it is not so mmm... crucial because whole clojure library is included in uberjar which is pretty heavy. One always can remove unnecessary native libraries from uberjar afterwards. So we want to create our own Fat-Jar for jogl and gluegen-rt libraries and upload it to clojars just like we did with Processing jars. Here are what we should do:

1 - Figure out what versions of jogl and gluegen-rt are used in processing. Go to core/library/ and check gluegen-rt.jar/meta-inf/manifest.mf, it should contain version used. Let it be 4.5.6 for example.

2 - Use following script to build Fat-Jar: https://gist.github.com/nbeloglazov/11332612. Run following commands which will create 2 jars:

(fat-jar 'org.jogamp.jogl/jogl-all-main "4.5.6" "jogl-all-fat")
(fat-jar 'org.jogamp.gluegen-rt/gluegen-rt-main "4.5.6" "gluegen-rt-fat")

Basically script does following: download all dependencies of provided artifacts (first and second arguments of fat-jar) and pack them to single jar (third argument), putting them to appropriate subfolder (e.g. natives/linux-amd64) depending on classifier of dependency.

3-6 - Repeat steps 3-6 of "Uploading Processing jars" part.

Congratulations! Finally you can have fun fixing all the issues caused by updating to the latest Processing!

Automated image tests

Quil has tests that run each snippets, takes a screenshot and compares with expected output image. The expected output is stored in the repo under the dev-resources/snippet-snapshots directory. It supports both clj and cljs. During test run a screenshot of the sketch is taken and compared with reference image. If difference exceeds threshold then snippet considered failed and snippet-name-difference.png is stored in the folder where screenshots are kept.

Ensure that you have imagemagick, geckodriver, and chromedriver installed before running tests. Imagemagick required for both clj and cljs while geckodriver or chromedriver is needed only for cljs. Currently CLJS tests are using chromedriver to account for a screenshot size issue with geckodriver.

To see all differences for failed snippets try:

$IMAGEVIEWER path/to/screenshots/folder/*difference*

where $IMAGEVIEWER is your image viewer. For example on ubuntu it can be eog.

Running clj

In clj simply run regular tests like lein test. It will automatically compare images and print to console names of failed tests if diff exceeds threshold. An example snapshot difference failure would look like:

lein test :only quil.snippet/quil_snippets_output_save-frame

FAIL in (quil_snippets_output_save-frame) (test_util.clj:73)
Image differences in "save-frame", see: dev-resources/snippet-snapshots/clj/normal/save-frame-difference.png
dev-resources/snippet-snapshots/clj/normal/save-frame-actual.png PNG 500x500 500x500+0+0 8-bit sRGB 5870B 0.000u 0:00.000
dev-resources/snippet-snapshots/clj/normal/save-frame-expected.png PNG 500x500 500x500+0+0 8-bit sRGB 3929B 0.000u 0:00.000

expected: (<= result threshold)
  actual: (not (<= 0.0974062 0.03))

The first line shows how to re-run just that test (possibly with the MANUAL=true environment variable to interact with the sketch while it's running). The "Image differences" line reports the diff image to inspect to see why it failed. The next two lines are output from identify which helps verify the sizes are equivalent. Finally we see can the see the threshold for comparison that was exceeded.

This can be adjusted on a per snippet basis using the :accepted-diff-threshold key. Snippets can also skip the snapshot test entirely by setting :skip-image-diff? to true. Image diffs can be unreliable so feel free to either increase the threshold or skip the test entirely if it's not deterministic, or hard to keep it deterministic across different machines or architectures.

Running cljs

Before running cljs test you need to compile cljs snippets and start test page server:

lein with-profile cljs-testing do cljsbuild once tests, ring server

After that you can run the test that using webdriver goes through all snippets, takes screenshots and compares them:

lein test :manual :only quil.snippet/all-cljs-snippets-produce-expected-output

Updating screenshots

To update screenshots (save existing screenshots as foo-expected.png) - prepend UPDATE_SCREENSHOTS=true to the lein command:

# all CLJ tests
UPDATE_SCREENSHOTS=true lein test 
# or to just update on test, in this example the output.save-frame test
UPDATE_SCREENSHOTS=true lein test :only quil.snippet/quil_snippets_output_save-frame
# cljs tests
UPDATE SCREENSHOTS=true lein test :manual :only quil.snippet/all-cljs-snippets-produce-expected-output

using lein?

  1. add this to your project.clj

    :main "name-of-your-project.core"

  2. add this to your projectname/src/core.clj

    (defn -main [& args])

and then you can run lein run !

Add a custom sidebar
Clone this wiki locally