Regression Tests after Dependency Upgrade

[xsc]

lein-ancient should handle the case where a newer version of an artifact causes incompatibilities in a project depending on it. The following are ways to either automatically ensure the stability/consistency of a project or to offer the possibility to revert a stable state.

See here for the comment that elaborated on potential issues with lein-ancient's current behaviour.

Do not overwrite the main profile

When upgrading dependencies, collect them in a new profile instead of overwriting the existing dependency vectors. This enables running tests using Leiningen's with-profile feature, letting the user decide whether to "activate" the new dependencies or to delete said profile (either by hand or automatically using lein-ancient).

Insertion of a new profile and automatic removal of it might cause issues with a project file's formatting.

Create a backup of the project file

The new artifacts are "live" immediately, reverting would be as simple as replacing the new project.clj with the backup one. Tests can be run without having to use with-profile, formatting issues should not arise.

This poses the question how to handle a requested upgrade if a backup file exists. A simple prompt à la "An existing backup file was detected. Do you want to continue?" might be enough to address this.

Automatically run Regression Tests

After an upgrade, existing tests are run immediately to determine incompatibilities. If everything passes, the backup project file is deleted/the main profile is replaced, otherwise changes will be undone. The possibility to run the tests after each upgraded dependency could facilitate the search for the cause of an incompatibility.

It has to be determined which test mechanism should be run (e.g. clojure.test, midje, speclj, ...) which poses the need for either a command line hint in the right direction (provided by the user) or usage of things like TravisCI's .travis.yml, containing the right call to use.

Fin

I'd really like to hear opinions or suggestions on this. lein-ancient shouldn't cause more problems than it solves.

[xsc]

Regression Testing is now done by specifying a test-ancient alias in your project.clj, e.g.:

...
  :aliases { "test-ancient" ["with-profile" "dev:dev,1.4:dev,1.6" "midje"] }
...

Backup files are created and restored when necessary.