Preparing the Models Library for release

Here are the steps that should be followed to prepare the Models Library for a new NetLogo release. Most of these steps should be done by the Models Czar. The final steps starting with Resave models in the library with the latest NetLogo version will be done by the dev team member doing the release.

Setting up your local git repository

We are going to start from scratch, only assuming that you have a Unix terminal and git working (and a basic knowledge of how to use both).

First, let's get a copy of NetLogo from GitHub. Move to the directory under which you want to clone NetLogo and run:

$ git clone -b hexy https://github.com/NetLogo/NetLogo.git nl
$ cd nl

This will put the NetLogo code in the nl directory. In the steps that follow, you will have to run some commands from the nl directory and other commands from the models directory under it. To make clear which location you have to run them from, we will try to prefix commands with either a nl$ or a nl/models$ prompt.

Don't forget the -b hexy part of the git clone command. This will tell git to get the hexy branch

At this point, your models directory will be created, but empty. To download the content from the models repo, do:

nl$ git submodule update --init

This will initialize and update all the NetLogo submodules, including the models library but also the bundled extensions, Behviorsearch, and the Mathematica-Link.

At this point, you should try to build NetLogo:

nl$ ./sbt all

Hopefully, all will go well. If it doesn't, the Building page is your best bet for help.

Now since we are going to presumably update some models, we need to make sure that you are able to push to remote models repo. If you do:

nl$ cd models
nl/models$ git remote show origin

You should see the following lines in the output:

  Fetch URL: https://github.com/NetLogo/models.git
  Push  URL: https://github.com/NetLogo/models.git

Note that all of this is useless if you don't have an GitHub account with write access to this repo. Someone from the devel team should be able to help you if that is not the case.

Now one last thing before we can start updating models:

nl/models$ git branch

...should tell you:

* (no branch)
  hexy

We're not on a branch! The reason for this is related to the way git submodules work. A submodule doesn't point to a branch: it points to a particular commit. So your git head is currently at whichever commit the NetLogo hexy branch points the models submodule to. This is fine if you plan to leave the model library alone, but that is not our case. We need to be on the models' hexy branch:

nl/models$ git checkout hexy

Again, make sure you are in the models folder. You do not want to check out the hexy branch of the main NetLogo repo. (Note that both the all branches of NetLogo currently use the master branch of the models library.)

You should also generally be careful that the branch checked out in models submodule is not changed without you realizing it. Running nightly.sh on the main repo, for example, runs git submodule update --init, which checks out whichever commit the submodule points to. So if you run nightly.sh at any point, master won't be checkout anymore in models (and you'll need to check it out again). Running the extensions task in sbt (or anything that depends on it, including all) also has the side effect of running git submodule update --init.

If you have made it this far without being too confused, we are actually ready to start doing stuff!

Adding or updating models in the library

This is the step where you actually update the models library. You may add brand new models, overwrite existing ones with new versions or just move some files around.

To verify the content of a change before committing it, it is often useful to run:

nl/models$ git diff

You should do a separate git commit for every change that you make, and every commit should be linked to a GitHub issue in the models repo. To link a commit to an issue, just include the issue number (e.g., #42) in the commit message. These issues don't have to be too detailed, but they should be associated with a NetLogo release milestone, as they are what the dev team will use to build the release notes for a new NetLogo version.

Push to GitHub

The next check is to push your changes to GitHub and to let automated tests tell you if there is anything wrong with them.

We used to have a bunch of scripts that needed to be run manually, but this is now taken care of using GitHub Actions. To see the results of the most recent tests on the models repo, go to:

https://github.com/NetLogo/models/actions

Actions will also send you an email if you push changes that fail the tests.

Error messages from the tests should be fairly explicit, but if anything is unclear, your best bet is probably to look at the test code:

https://github.com/NetLogo/models/tree/master/src/test/scala/org/nlogo/models

Feel free to add comments to the code to clarify things if needed.

The case of URLs

One of our tests checks that URLs in the info tab are valid. While this is a very useful test to have, it sometimes fails because a web page is only temporary unavailable or a server takes too long to respond. If you suspect that this is the case, try restarting the build later to see if it still fails before changing the URL.

Make sure each model has a suitable preview image

Some models have automatically generated previews, others are manually generated. The automatically generated ones are listed in .gitignore; the manual ones are checked into version control. We have an automated test to make sure that this is the case.

To generate a preview image for a model, you will need to use sbt. In addition to building NetLogo, sbt is used to run various tasks related to that process. Generating preview images is one of those tasks.

To run sbt, go the main repo's root directory and do:

nl$ ./sbt

This will take you to the sbt command prompt. You can get out of sbt at any time by typing exit.

If you now type tasks, you will get a list of all the tasks that sbt supports. Those tasks are defined in a bunch of .scala files in the project/ directory, should you ever want to look at them. The two tasks we are interested in at the moment are allPreviews and preview. As you might guess, allPreviews generates preview images for all models in the library, while preview takes the name of a model as an argument and generates the preview image for that specific model.

Running allPreviews will take a while, but might be worth it if you have a bunch of new models. When you are in the refining stage (i.e., tweaking a preview for a specific model), preview is your friend:

> preview "models/Sample Models/Biology/Wolf Sheep Predation.nlogo"

If you find a model (among those you modified or added) that does not have a suitable preview image, there are two ways to go about it:

• Tweak the model's preview commands to get a usable image. You can do that in the Preview Commands Editor (as of hexy) or directly in the model file: see Unofficial features. Don't forget to commit your changes.

• Make an image yourself, add it to the repo, and in the model's preview commands (again, directly in the model file) put need-to-manually-make-preview-for-this-model. This should used only as a last resort because:

  • Preview commands are useful for testing;
  • Manual previews add bulk to the repo and risk getting out of sync with the model.

A final step: if there are new models that will have automatically generated previews (whether you have tweaked the preview commands or not), you need to add the .png files to models/.gitignore, so that they don't get saved in the repo. If you go to the models directory and do git status, you'll see those that git would be adding: just copy the paths of those that should not be there to .gitignore. If you re-run git status after that, they should not appear anymore.

Resave models in the library with the latest NetLogo version

This step requires that the NetLogo version you just cloned be identified as the version we are about to release, not as the previous one. You can check that by running NetLogo (nl$ ./sbt run) and looking at the About item in the Help menu or by running:

nl$ head -1 netlogo-gui/resources/system/version.txt

If the version number is not updated yet, ask someone from the dev team.

The original motivation for this step was that the version number saved in the model file appears on the Modeling Commons website and it looked weird to have most model from the NetLogo Model Library appear to have been saved with various older earlier versions of NetLogo.

This step is handled by a ModelResaver object in the org.nlogo.tools package. To run it, fire up sbt:

nl$ ./sbt

This will take you to the sbt command prompt, indicated by a >. Now run the ModelResaver:

> netlogo/test:runMain org.nlogo.tools.ModelResaver

It opens and saves all models in the library, excluding 3D models. To resave the 3D models, you will need to run it again separately, starting sbt with ./sbt -Dorg.nlogo.is3d=true.

It's possible that you see some error message during the process (e.g., "MIDI is not available") but those should not interfere with the process. If some models truly fail, you should get a list of FAILED MODELS: at the end.

Assuming the model repo was clean when you started (it should be), you can check what the script actually did to your model files by going back to the models repo and running:

nl/models$ git diff

If all looks well, just commit everything:

nl/models$ git add -A
nl/models$ git commit -m "update model version numbers"

Resaving all the models like that with require regenerating all the model checksums, which is the next step in the process anyway.

Update the models checksums

You can also find a bit of info about checksums in the Model checksums section of Unofficial features. Actually, you should read that before you go on. Checksums are built by running a models' preview commands and taking the SHA hashes of the resulting world (via export-world) and view image. All models that were changed need to have their checksums updated, and since we updated version numbers in models in the previous step, that's all of them.

Note there are two steps, since the NetLogo GUI project will generate test/checksums.txt and the headless project will generate test/checksums-headless.txt.

From the main repo, run the below to wipe and regenerate all checksums:

echo "" > test/checksums.txt; echo "" > test/checksums3d.txt; echo "" > test/checksums-headless.txt
./sbt "netlogo/allChecksums; headless/allChecksums; threed; netlogo/allChecksums"

Note that allChecksums does not automatically delete checksums for models that are no longer present in the library, so if you have moved a model, the old checksums will linger on, which will cause problems when running tests. You could delete old checksums by hand, but the quick and dirty way is to wipe out checksums.txt (and test/checksums-headless.txt, you can echo "" > test/checksums.txt) and let allChecksums regenerate the whole thing. Clearing the file each time the library is updated is best practice, as models are regularly moved around. We want to keep them in the same order in the checksum files to minimize apparently unrelated changes when someone does decide to wipe the file for a new model.

sbt takes care of updating the test/checksums.txt, test/checksums-headless.txt, and test/checksums3d.txt files. It's best to compare this list to the changed models in the library to make sure only intended checksum changes are included. In the models repo, try a command like git diff --name-only HEAD 6.2.0 to get the change list. The only things left for you to do is to make sure the new checksums pass:

./sbt "netlogo/testOnly *TestChecksums"
./sbt "headless/testOnly *TestChecksums"
./sbt "threed; netlogo/testOnly *TestChecksums"

A note about previews and checksums for 3D models

Note that the graphics checksums stored in checksums3d.txt are actually 2D checksums, because NetLogo cannot, at the moment, headlessly generate 3D previews. While this may seem a bit strange, it is still potentially useful for catching regressions.

Updating the main repo

There are three things that need to be done in the main repo:

Updating the models submodule

As we mentioned earlier, the models repo acts as a git submodule in the main 5.x repo. This means that the main repo points to a particular commit in the models repo. We need to update it up to point to the latest commit.

git should already be aware that some changes have been made in the submodule. If you go to the main repo and do git status you should see:

# modified: models (new commits)

To commit these changes, simply do:

nl$ git add models
nl$ git commit -m "update models submodule"

You can see which models commit the main repo now points to by doing:

nl$ git ls-tree HEAD models

This should be the same commit as the current HEAD of the models repo, which you can look at by doing:

nl$ cd models
nl/models$ git log -1

Updating Model Cross-references

Cross-referencing models is done by updating changes in crossReference.conf within the models library repository. The developer responsible for building the release may be asked to review this file. See the models library wiki for more information on updating this file.

Updating the credits

If any new names should be added to the credits in resources/system/about.txt, add them.

Paying special attention to models used for tests

In the test directory of the models repo, you will find a few files with the .txt extension. These files define some NetLogo language tests using models from the library. If any of the models used in these tests are modified, you should make sure that the tests still pass. You can do that by running:

nl$ ./sbt "test-only *TestModels"

Note that these tests can be brittle. The tests using Artificial Neural Net - Multilayer.nlogo, for example, rely on the behavior produced using a specific random seed. If the model is changed ever so slightly, a test might fail. In case of a failure, you will need to figure out if it's a "real" failure (i.e., due to a bug that was introduced somewhere) or if it's just a matter of, say, finding a new random seed that works for this specific test.

Doing a final check

At this point you should make sure that you don't have any un-committed changes in either the main or the models repo. This would also be a good time to push your changes to GitHub if you have not already done so.

Further instructions for CCLers are here (but you can also just ask someone from the dev team).

Tagging the model library for release

Once a release has been produced by the dev team, the models repository should be tagged with the version number of the release so that it's easy to see which commit was used in the release. You can see the already existing tags by running:

nl/models$ git tag

To add a new tag, use the following command, replacing, of course, 5.0.6 by the right version number:

nl/models$ git tag -a v5.0.6

This tags your local repo, but the new tag won't be pushed to GitHub until you do:

nl/models$ git push --tags

Final step

Have you learned anything new during the process? Was there any misleading information or mistakes on this page? PLEASE UPDATE THIS PAGE!