Clone this wiki locally
Umple Architecture: Testing
This entry gives highlights of how testing is organized in Umple. Umple contributors who wish to better understand how the tests work in Umple should work their way through the files and directories described here. Please visit the Architecture page for a top-level view.
Test-Driven Development (TDD)
Umple tests are automated. The sections below will describe the types of tests, and how to integrate new tests.
When making a change, the process is:
1) Write tests that you know currently fail. For example, a test of a new feature, or some code that should compile but does not.
2) Verify that the tests fail in the build process.
3) Write the code to make the tests pass (developer test using
ant -Dmyenv=local -f build.umple.xml umpleSelf compile packageMainJar)
4) Verify that the tests now pass (do a full build using
5) Commit your changes locally (
git add path/to/changed/file; git commit)
6) Create a Pull Request, see CONTRIBUTING
You can find more information about the relationship between test-driven development (TDD) and Umple in the slide show resource located here, including an example straight from the Umple code.
All tests must pass 100% automatically. You can commit tests after step (2) if you comment them out first. However, you must remember to then immediately uncomment the tests.
Note that we also do a small amount of manual sanity checking using UmpleOnline and the command line compiler, but this is really just to make sure that nothing has been overlooked.
Tests are always run using ant; ant scripts are in the build directory. See HowToRunAutomateTestSuitesForUmple for information about running tests.
Most tests use JUnit. To learn more about JUnit, see http://www.vogella.com/articles/JUnit/article.html
Quality assurance directory
After you have run Umple tests (including a full build), you will see an automatically generated website with all the test results. Go to
dist/qa/cruise.umple/index.html in your local working directory. You can navigate this to find failing tests. On the build server, it is located at: http://qa.umple.org
Key directories with tests
- Main directory with tests of the compiler and various other components: cruise.umple/test/cruise/umple/ Here you will find tests for the compiler (subdivided in to directories for the main parts of the compiler, state machines, tracing, utilities, etc.) You will also find tests for the builder and the user manual generator. Tests for the command line programs are directly in this directory.
- cruise.umple/test/cruise/umple/compiler/ contains many .ump files that are compiled, but thee key junit files used for testing have the .java suffix. The .ump files have a numerical prefix to help keep them in order.
- Testbed Umple files that are compiled to each of the key generated languages: testbed/src/ The next items are the actual tests.
- Testbed tests used to run Umple-generated Java programs: testbed/test/cruise/
- Testbed tests for Ruby and Php
- Tests for the Eclipse plugin
Types of compiler testing
There are several levels of testing in the Umple compiler. There is also testing of all the non-compiler components. This section focuses on the compiler testing.
1. Parser Tests
These verify that a construct parses correctly. If you are changing the grammar you will need to add additional tests. If one of these tests fail it indicates that the a construct is not parsing correctly. Since we want to maintain backward compatibility, it is important to think carefully before altering existing tests.
Key file: UmpleParserTest.java
2. Metamodel Tests
These check that the metamodel is being populated correctly by various Umple constructs. The tests call methods on the metamodel classes. Add or modify tests if you are modifying the metamodel.
Example junit files (there are many others):
3. Template Tests
These verify that the generated output in languages like Java is as expected. These tests are a little tricky as certain changes can result in changing a large number of files. They use assertions of the form assertUmpleTemplateFor. Add one of these if you are adding new generated code. Modify or add if you are modifying generated code. Make sure you deal with all supported generated languages.
- CodeInjectionTest.java. Note that these have an associated .ump file that is compiled, and then associated .txt files in each language directory.
Key file with assertions that are used:
To edit a template test you update the relevant .txt file so that it has the expected output (i.e. generated Java, PhP, Ruby, etc). A common way to do this is to simply compile the input file and then paste the content over the output file. However, warning: Sometimes you have to do this over an over again for dozens of files as you debug your contribution. If you are making a change that will require a large number of changes to template test files, then it is acceptable to temporarily deactivate certain template tests until you have your feature working correctly. But it is then critical to update and reactivate the tests before committing.
4. Testbed Tests
These verify that compiled code that generates languages such as Java, can then compile under the language compiler and execute correctly. Create complete little programs that implement any new feature.
5. Tests in the User Manual
The user manual has sample code between @@example and @@endexample tags. Each new feature should be documented here. Some examples have main programs. Currently these examples are not run automatically at each build, but it is intended that this will happen in the near future. See Deploying for information on modifying the user manual files.
The files used to create the user manual are in build/reference. The numbering scheme is used to sort the files in the same order they will appear in the user manual. the order.group file is used to arrange the manual into categories.
6. Sample systems loaded into UmpleOnline
There is a directory of simple systems displayed in UmpleOnline. These are also compiled during testing. See ModifyingUmpleOnlineExamples.
7. Testing by compilation of the Umple compiler itself
Key to Umple is compiling Umple itself. A full build of Umple will first compile the newly changed code using the existing compiler to create a new compiler, and then recompile the changed code under the new compiler. This is essentially an extremely large test case. As Umple evolves we attempt to incorporate new Umple features into the Umple compiler itself.