Fetching contributors…
Cannot retrieve contributors at this time
497 lines (471 sloc) 33.3 KB
OK - refactor BuildTest & SuiteConfiguration
OK - extract reusable components from BuildTest to an external library
OK - jumi: extract reusable components from TestEnvironment
OK - jumi: extract reusable components from BuildTest
OK - use improved parameterized runner; remove hack in fi.jumi.test.util.JarFileUtils#checkAllClasses
OK - create new project, groupId: fi.luontola.buildtest
OK - create a stripped history of /end-to-end-tests/src/test/java/ that contains only:
- BuildTest, TestEnvironment (and any new classes extracted from them)
- AsmMatchers, AsmUtils, CompositeMatcher, JarFileUtils, XmlUtils
OK - merge the stripped history to the new project, reorder packages
OK - tests for non-trivial helpers: JAR, Maven
OK - downgrade to Java 6
OK - update the BuildTest in jumi and jumi-actors to use that library (at first in a feature branch, because Maven Central Sync will take time to activate)
OK - rename SuiteConfiguration.classPath to classpath
OK - deprecate current methods
OK - create new methods
OK - create a process for removing the deprecated methods after a particular number of days or releases
OK - BuildTest: maintain a list of all deprecated APIs and a date/version when they were deprecated, and when to remove them
OK - release the buildtest library
OK - merge feature branch, use the released buildtest library
!! - communicate using memory-mapped files (see how IPC is done here)
* benefits:
- more secure than sockets (uses file system permissions)
- should be faster (benchmark latency and throughput)
- automatically saved to file for later use
- no need to close socket connections
- no need for a networking library
* disadvantages:
- how to manage multiple writers? one file per writer?
- probably requires polling to get informed about updates
* need to find out how to do synchronization
- create a util like Fences?
- lazySet/StoreStore barrier good for throughput in single-writer scenarios (as long as doing writes continually):
- beware of word tearing, MMFs guarantee nothing:
- list of possible optimizations:
- MappedByteBuffer slower than ByteBuffer:
OK - spike communicating over memory-mapped files
OK - create IpcBuffer which wraps multiple MappedByteBuffers; combine the good parts of:
Mina 3:
+ lookup using singly linked list -> let's use a doubly linked list with O(1) in both directions, and field "current" instead of "head & tail"
- accesses primitives using 1-byte operations
+ only 1 public and 2 private classes
- class is final, cannot customize easily
- has bugs and low coverage (just reported and
- no putString
- doesn't add ByteBuffers automatically
Netty 4:
- lookup using binary search
+ accesses primitives using smart divide-and-conquer
- tons of classes, do many more things than is necessary (e.g. detects Unsafe and Java version; see PlatformDependent) and increase class loader overhead
- has low coverage
- no putString
- doesn't add ByteBuffers automatically
+ naming: get/set for absolute, read/write for relative
OK - absolute get/set, relative read/write
OK - byte
OK - generic scaffolding for random data; on failure show the seed that was used
OK - short (currently not needed by our listeners)
OK - char
OK - int
OK - long (currently not needed by our listeners)
OK - use Integer.SIZE etc. in absolute tests
OK - TestableRandom: on failure, print all generated values and events (including "reset"), maybe also allow logging custom messages
OK - make it based on multiple ByteBuffers
OK - create AllocatedByteBufferSequence
OK - interface ByteBufferSequence.get(int index):ByteBuffer for dynamically creating the MMFs (mockable in tests)
OK - same index returns same backing data
OK - we can assume sequential access to new indeces; IpcBuffer's position depends on every preceding segment's size
OK - expand buffer automatically
OK - maintain our own position
OK - convert positions to buffer's relative positions
OK - test well traversing forwards and backwards
OK - use lambdas to remove duplication in all get/set methods
OK - escape analysis did not remove the allocations; undo the lambda refactoring
OK - test well the splitting of data between buffers
OK - generic tests with initial buffer size varying from 0 to sizeInBytes
SKIP - consider using parameterized tests for the above
OK - also check that there is no gap in buffers; it should be possible to dumbly concatenate them all and read the same data
OK - create MMF based MappedByteBufferSequence
OK - extract contract test
OK - use MMFs
OK - multiple instances/processes with same path should access same data
OK - doesn't need to cache the mapped buffers, but may re-map if called multiple times with same index
OK - IpcBuffer: avoid looking up adjacent buffers repeatedly (opening a MMF is relatively slow)
OK - new FileSegmenter(initialPath, initialSize, maxSize).pathOf(index) & sizeOf(index)
OK - when MMF already exists, use its file size (i.e. map whole file) instead of whatever FileSegmenter tells by default
OK - retry if opening the file fails at first
OK - see about making fi.jumi.core.ipc.MappedByteBufferSequence#tryMapFile thread safe
OK - hand-written serialization of SuiteListener and CommandListener events (rename to LauncherListener?)
protocol draft:
header: "Jumi" (4 bytes), protocol-version (int), interface (string), interface-version (int), events...
events: status (byte), event-type (byte), event-data
status: empty (0), exists (1), end-of-file (2)
* stuff that depends on interface & interface-version: event-type, event-data
OK - roundtrip tests
OK - string operations (only relative read/write, because strings are variable size)
OK - fully test serializing StackTrace
OK - refactor: create StackTrace.Builder, use it also in StackTrace.copyOf, keep the constructor private
OK - make StackTrace.copyOf aware of StackTrace, to not rewrap it? else it might obstruct getExceptionClass (test: chain two copyOf() calls)
SKIP - test equality using Apache Commons EqualsBuilder; Mockito might already provide a matcher which uses it
OK - test deep structural equality using a custom matcher; the existing tools weren't enough
OK - test: simple case
OK - test: multiple chained causes
OK - test: multiple suppressions
OK - make deepEqualTo compare Collections using iteration
OK - refactor: unify cause and suppressed
OK - test serializing strings
OK - empty string
OK - random string of random length
OK - all non-printable characters, i.e. 00-1F
OK - java.lang.Throwable#getMessage may be null
OK - add test for StackTrace serialization
OK - create writeNullableString or change writeString? use length of -1 as marker, or write a single byte marker?
OK - use a fixed-size buffer for the round-trip tests, to avoid PIT timeouting (removing a writeString() causes it to try read huge strings)
OK - spike whether that solved the timeout by mutating manually
OK - rename StubByteBufferSequence to FixedByteBufferSequence
OK - make FixedByteBufferSequence pass the contract
OK - test that accessing unavailable indexes throws IllegalArgumentException with clear message
OK - use FixedByteBufferSequence with one big enough buffer in the serializer's tests
OK - there are still a couple of PIT timeouts in writeStackTrace
OK - use FixedByteBufferSequence for StackTrace tests or add totalCapacityLimit to AllocatedByteBufferSequence
OK - handle all (4) failure conditions when reading data that has wrong header fields
OK - concurrency tests
OK - test: concurrent producer-consumer pair and some 100 events
OK - make header work: write protocol-version after the rest of the header (i.e. same way as status) to avoid race conditions?
OK - make events work: between writing an event and the event's status there must be a happens-before relationship
OK - refactor: separate the protocol and SuiteListener specific stuff
OK - remove duplication of writing the status: take event objects as input (implement MessageSender), so that we can surround them with the status writes
OK - extract deserialization of events to its own method (excluding status reads)
OK - move the event reading and writing to its own class (SuiteListenerEncoding)
OK - String read/write to its own class
OK - extract Encoding interface
OK - accessors for interface name and version
OK - make SuiteListenerEncoding#deserialize non-static
OK - rename source/target IpcBuffers to just buffer, keep it in a instance field
OK - rename SuiteEventSerializer to IpcProtocol
OK - split SuiteEventSerializerTest to match the new classes
OK - rename packages: ipc -> ipc.buffer, serialization -> ipc
OK - refactor: support deserializing one event at a time
OK - create a poll method (no sleeps nor loops) that returns NONE, SOME or END based on whether the stream had events
OK - move the wait loop to a utility method
OK - refactor: create a wait loop timer utility that uses progressively longer sleeps
OK - make sure that all the memory barrers are in place
OK - public APIs for read-only & write-only access
OK - make IpcProtocol implement restricted interfaces IpcReader and IpcWriter
OK - create factory methods IpcChannel.reader/writer(Path)
OK - add a read-only flag to MappedByteBufferSequence (or factory methods readWrite/readOnly), in which case will open/map files read-only
OK - the writer should be the first to touch a new segment, so that the writer would determine its size
OK - concurrency test: writer uses 1 byte segments, reader 2 byte segments; afterwards all segments should be 1 byte (can we do it also for the first segment?)
OK - append a null byte before writing the previous event's status?
SKIP - should read-only MappedByteBufferSequence not be able to create new segments?
OK - refactor fi.jumi.core.util.Resilient#tryRepeatedly to use only IOException, no generic exceptions
SKIP - IpcBuffer: test how it handles it if some segments are of size 0 (e.g. if there was a concurrency race in creating the file)
-> MappedByteBufferSequence will throw an exception
OK - TestId: create a method for getting the path as int[]
OK - make fi.jumi.api.drivers.TestId#getPath public
OK - write tests directly for getPath
OK - refactor SuiteListenerEncoding#writeTestId to use it
!! - communicate suite results and commands over MMFs instead of a socket
OK - tell the other side that where to put the MMFs
OK - DaemonConfiguration: daemon specific --daemon-dir
OK - DirBasedSteward: create a new daemon dir (JUMI_HOME/daemons/*)
OK - ProcessStartingDaemonSummoner: get the daemon dir from Steward, tell it to daemon
OK - remove the setDaemonDir spike code ProcessStartingDaemonSummoner#connectToDaemon
OK - make the daemon poll receive commands from daemon dir
OK - create CommandListenerEncoding
OK - example usage & serializing all events
OK - remove duplication: CommandListenerEncoding & SuiteListenerEncoding
OK - remove duplication: CommandListenerEncodingTest & SuiteListenerEncodingTest
OK - new packages:
ipc.encoding (*Encoding) (IpcChannel, IpcProtocol, IpcReader, IpcWriter, PollResult, *WaitStrategy)
ipc.api (CommandListener)
ipc.watcher/monitor/endpoint/coordinator/supervisor or just the top-level ipc package
OK - make the daemon monitor the daemon dir for new command MMFs (daemon-dir/commands/*), create IpcCommandMonitor as replacement for DaemonNetworkEndpoint
OK - util for noticing new files
OK - read commands from every new command file (daemon-dir/commands/*/request)
OK - read commands until IPC channel closed (though typically only one command per file)
OK - something to manage the daemon dir, e.g. creating command dirs
OK - write result to a corresponding file (daemon-dir/commands/*/reply)
OK - include the suite results file path in the reply (daemon-dir/suites/*/suite)
OK - write suite results to a MMF
OK - write suite results header before sending the reply
OK - close and forget the command MMFs after use
OK - close the suite results MMF after the suite is finished
OK - start it in Main, alongside DaemonNetworkEndpoint (using a different SuiteFactory instance?)
!! - switch to communicating over MMFs
- rename ProcessStartingDaemonSummoner to NetworkedProcessStartingDaemonSummoner
- rename RemoteSuiteLauncher to NetworkedRemoteSuiteLauncher
- create a new (non-actor) ProcessStartingDaemonSummoner which creates the daemon dir, starts the daemon process, and returns a CommandListener to it
- create a new RemoteSuiteLauncher
- send the runTests command
- handle timeout if onSuiteStarted doesn't happen
- handle timeout if process dies
- create a heartbeat MMF on the launcher side (daemon-dir/heartbeat), includes currentTimeMillis (in binary)
- daemon updates the heartbeat every 100 ms or so; create daemon thread in main
- monitor that heartbeat on the launcher side, give an internal error if the process dies in the middle of a suite
- use the new MMF-based classes instead of the old networked classes
- remove NetworkedRemoteSuiteLauncher
- remove NetworkedProcessStartingDaemonSummoner
- remove DaemonNetworkEndpoint
- remove and all that depends on it
- enable fi.jumi.test.DaemonProcessTest#classes_showing_up_in_actor_logs_have_custom_toString_methods
- possibly subclass SuiteListenerToEvent (IPC writer?) and give it a custom toString
- after implementing, consider refactoring the following places:
- DirBasedSteward: remove old daemon dirs after they are no more needed (keep last X dirs, at least Y days)
- refactor: do not convert to a Path at SuiteEventSerializer#readTestFile and then back to String in TestFile#fromPath, but expose a raw String constructor or factory?
- save v1 events to file for testing backward compatibility
- save the current example usage
- save all the variations of exceptions (hand-written stack trace elements) and test them for deep equality
- ensure platform compatibility of MMF communication (detect and degrade gracefully)
- lack of file change events
- DirectoryObserverTest: on OS X 10.9.4 it takes 10 seconds for the WatchService to notice changes; file system format: Mac OS Extended (Journaled, Encrypted)
* no native support for OS X
FAIL - SensitivityWatchEventModifier.HIGH only lowers the delay to 2 seconds, which is too much
- use a faster poller (50-100ms) as a failsafe?
- custom support for Mac?
- inability to map files into memory
- on VirtualBox it's not possible to map files that reside on a directory shared from Windows; network drives probably won't support mapping files either
- fall back to using RandomAccessFile/FileChannel and FileLock
- write whole events to file under a file lock
- buffer modifications to internal byte buffers
- flush the buffer regularly but not after every small message
- keep track of which buffers are dirty and need flushing
- send the actor asynchronously a flush event after writing an event, so that it would be flushed only after all currently queued events have been written to the buffer; noop if nothing to flush
- CI: run all tests in restricted environments to tests each of the above mentioned degradations
OK - BuildTest.all_classes_must_be_annotated_with_JSR305_concurrent_annotations does not report all classes that are missing annotations, but only the first one that is found
OK - Eclipse's JUnit integration adds to classpath entries that are not valid file paths on Windows
- detect when running on Windows and a path starts with "/C:/" (regex "/[A-Z]:/"), drop the first "/"
- detect with
- detect by checking the FileSystem's roots?
FileSystems.getDefault().getRootDirectories() - check if first of them starts with "/"
- optimize PIT timings; too many TIMED_OUT mutations, makes the build slow
FAIL - tried to lower timeoutConstant; didn't help on the slow CI server, and only marginal improvement on fast desktop (total build time from 3:30 to 3:00)
- add a sanity check that will detect infinite loops and will throw an exception
- investigate individual TIMED_OUT mutations; in what kind of an infinite loop do they get stuck? can we detect it and break?
- safety threshold to readNullableString and everywhere else where an array length is read?
- write daemon output to log file
- use ProcessBuilder.redirectOutput to write the log file (else the process would hang when the launcher JVM exits)
- write log to JUMI_HOME/daemons/*/console.log
- default JUMI_HOME to ~/.jumi
- update e2e tests to read the output from the log file
- get rid of fi.jumi.launcher.remote.ProcessStartingDaemonSummoner#copyInBackground
- don't create background threads in the launcher automatically (the tests should create them explicitly, if any)
- also remove the hack in fi.jumi.test.ReleasingResourcesTest#launcher_stops_the_threads_it_started
- update ErrorHandlingTest#gives_an_error_if_starting_the_daemon_process_failed
- show the path to the daemon log file in the error message
- consider showing the log file path for all internal errors? or leave it for later, for another mechanism to tell stats about the process?
- end-to-end tests: persistent daemon process
* should drive better design on launcher side
- connect to existing process
- cancel idleTimeout if somebody connects again (unit test only?)
- if daemon dies unexpectedly (System.exit/halt, JVM crash etc.), report it and restart next time
- how to show unfinished test runs in the TextUI? (both JVM crash unfinishing all tests, and buggy testing framework unfinishing some tests)
- if daemon disconnects due to network problem, reconnect and resume where we were left? (should not happen with MMFs?)
- should not exit the JVM (due to idle timeout) while tests are running
- have a second look at whether the launcher-side design improved
- make it reliable
- release
- tidy up release notes
- bump version
- release
- measure speedup, but don't yet publish (wait for class loader caching before doing a full comparison)
- run tests in a loop on a weakly ordered, multi-core CPU, to look for cache coherency issues
- where to find a multi-core ARM for a test drive? and are the newer ARMs anyhow cache coherent?
RPI 2 is a quad core ARMv7:
- are there Itaniums (IA64) on any cloud for rent?
- Azul's Vega should have a relaxed memory model; ask for co-operation, also to ensure scalability?
"Azul hands out 'academic' accounts all the time."
- class loader caching
- release
- tidy up release notes
- bump version
- release
- create a performance comparison (e.g. with Dimdwarf's tests, or some publicly known open source project with fast tests; Joda-Time?)
- promote widely
- plugin system for custom DriverFinders (keep it minimal, nearly walking skeleton level, until we have more plugins)
- hook into fi.jumi.core.drivers.DriverFinderFactory
- move JUnit compatibility into its own module
- create jumi-junit.jar
- extract it automatically to ~/.jumi/plugins, the same way as the daemon jar
- tell to daemon as system property what plugins to load: e.g. -Djumi.plugin.path=C:/plugin.jar;C:/another.jar (separated by path.separator)
- load each plugin in a separete plugin classloader, with access to test classloader and to jumi-core
- at first make the test classloader the parent of the plugin classloader (it'll work as long as we don't hide jumi-core from the test classloader, after which the plugin classloader might need two parents or use context classloader)
- refactor JUnitCompatibilityDriverFinder et al to avoid reflection
- refactor tests that use JUnitCompatibilityDriver.class.getName() instead of instanceOf checks (in DriverFinderFactoryTest)
- investigate maven-shade-plugin; can we now turn on minimizing? it used to remove the JUnitCompatibilityDriver from jumi-daemon.jar
- experiment:
- how much is daemon jar minimized? if it minimizes only a little, it might not be worth it
- otherwise: write a test that the daemon jar contains everything from the jumi-core
- ignoring tests
- ignoring without running a test
- ignoring after test has started
- JUnit @Ignore
- update the test in fi.jumi.test.JUnitCompatibilityTest
- implement fi.jumi.core.junit.JUnitRunListenerAdapter#testIgnored (JUnitCompatibilityDriverTest)
- JUnit assumptions
- update the test in fi.jumi.test.JUnitCompatibilityTest
- implement fi.jumi.core.junit.JUnitRunListenerAdapter#testAssumptionFailure (JUnitCompatibilityDriverTest)
- release
- tidy up release notes
- bump version
- release
- promote
- make option 2 explicit by adding a test for it; check for "nihao: ??" and "umlauts: ???" on US-ASCII
> - when default charset is ISO-8859-1 (fi.jumi.test.StandardOutputTest#compensates_for_the_default_charset_of_the_daemon_process), Unicode chars are lost already when they are captured (nihao becomes ??); should we do something about it?
> (1) We could have an internal PrintStream which uses UTF-8/16 and capture losslessly from it, after which it's routed to a PrintStream which uses the actual encoding (might interfere with writing binary data to stdout).
> (2) Or then keep it as is. That would be the way that System.out works normally. Principle of least surprise?
- misc refactoring (choose some, defer others)
- SuiteMother: twoInterleavedRuns seems to be missing onTestFound for the root test
- EventBuilder: nextRunId could use the RunIdSequence
- upgrade to Java 8
- check that most developers have upgraded to Java 8, or wait for Java 7 end-of-life (April 2015, as estimated as of May 2014)
- search the code for "Java 8" TODO comments and upgrade those to use Java 8 APIs
- look for other places that could benefit
- for loops -> Stream
- StringBuilder usage -> StringJoiner
- update documentation of Jumi and dependent projects to reflect the new minimum requirement
- PIT shows incorrect coverage for TestId; stopped working near commit cccff66d4541a0811509a3dadf41084abca278a4, after upgrading to JDK 8
- prepend TestFile as context information to the message in fi.jumi.core.runs.RunEventNormalizer#onInternalError?
- create a custom taglet (as independent project) for cross-references from public API's javadocs to tests: e.g. {@ref com.example.FooTest#someTestName}
- DaemonConfiguration: refactor the code for parsing command line parameters (try to minimize shotgun surgery, e.g. commit 1ea1b44a186ef00b32b2ae0ad05f88387e422fcb)
fi.jumi.core.config.DaemonConfiguration#JUMI_HOME and other constants
- thread-safety-checker: when an inner class is missing the annotation, use the annotation of the enclosing class or default to @NotThreadSafe when the enclosing class has any concurrency annotation
- replace SuiteRunnerIntegrationHelper with TestBench?
- consider implementing SynchronizedPrintStream using something else than CGLIB (ASM? Tapestry's Plastic?) to make the JAR smaller
- check whether we still have many ASM versions we have on classpath (caliper depends on ASM 3), block "asm:*" artifacts with maven-enforcer-plugin?
- make JUnit print a thread dump when a test times out due to @Test(timeout)
- improve JUnit's test timeouts to show a stack trace of the timed out test, to help debugging sporadic timeouts in RunningTestsTest
- evaluate whether some of the concurrent tests could be made simpler by using MultithreadedTC
- BuildTest: make sure that jumi-daemon.jar contains everything from jumi-api.jar no matter what
- test the test by adding an unused class to jumi-api
- add includes if necessary:
- fix Maven's pomPropertiesFile element, get rid of the current workaround of a separate
<archive> element:
<pomPropertiesFile> element:
- there might be unnecessary eventizers in fi.jumi.daemon.Main and /jumi-core/pom.xml
- create EventizerUsageStatistics (extends EventizerProvider), print usage statistics when program exits
- organize the project so that it's possible to run unit tests separate from end-to-end tests
- put end-to-end tests in their own package,
- or use JUnit categories, maybe with a custom runner?
- or use ClasspathSuite?
- evaluate JarJar or ProGuard, maybe it could minimize the jars better than the shade plugin by removing also unused methods (uses newer version of ProGuard)
1. run shade plugin to embed all dependencies and create the dependency-reduced POM
2. run proguard to minimize the JAR at method level and preverify (in and out files must be different?)
3. replace the main artifact
- consider logging with MinLog
or create our own clone, since it's just one file (and )
- actors: extract AbstractMessageLogger from PrintStreamMessageLogger
- create logger specific implementation of AbstractMessageLogger
- customize MinLog's output to use microsecond accuracy, similar to PrintStreamMessageLogger
- consider using
- minor release
- tidy up release notes
- release
- promote
- deployment pipeline
publish [M]
announce [M]
- create jumi-integration and add it to the pipeline
- integration tests for Specsy
- platform for running the tests
- run an untrusted go-agent in a restricted environment
- add it to the pipeline
- push-button releasing the staging repository
- run smoke tests before promoting
- if smoke tests pass, promote automatically from OSSRH to Central
- consider migrating to nexus-staging-maven-plugin,
- automate updating the web site
- update the version number of latest release to the web site
- update javadocs to the web site
- web site improvements
- site for jumi-actors-maven-plugin
- aggregated javadoc of all modules?
- improving TextUI
(- if multiple readers are needed: Streamer class for receiving events from UI thread)
- different colors for out (default), err (red), metadata e.g. test names (blue), binary output (yellow BG)?
- JCons (uses JNI on Windows, so maybe support only Unix)
- could also JLine do it? (uses a native binary)
- misc links
- ANSI escape codes work in IDEA?
- write performance tests
- test: suite with a hundred test classes, each with a hundred tests, some of which also do printing; measure total time of running the suite
- experiment with different queue implementations, for example ConcurrentLinkedQueue
- try following the single writer principle:
- for example: separate queue for each test worker (implies parallelism = locally sequential, except for slow tests?, use a work stealing executor?) and one thread which loops through them and writes to the coordinator actor queue
- "Error creating shaded jar", probably an issue with NTFS's pending delete and the shade plugin
- make a bug request, maybe also a patch
- investigate warnings on build:
- happens when jumi-core-0.1.0-SNAPSHOT.jar (or daemon JAR) exists from the previous build
[WARNING] We have a duplicate net/orfjackal/jumi/core/INTERNAL/org/apache/commons/io/input/TailerListener.class in C:\DEVEL\Jumi\jumi\jumi-core\target\jumi-core-0.1.0-SNAPSHOT.jar
[WARNING] We have a duplicate net/orfjackal/jumi/core/INTERNAL/org/apache/commons/io/input/TailerListenerAdapter.class in C:\DEVEL\Jumi\jumi\jumi-core\target\jumi-core-0.1.0-SNAPSHOT.jar
[WARNING] We have a duplicate net/orfjackal/jumi/core/INTERNAL/org/apache/commons/io/input/TeeInputStream.class in C:\DEVEL\Jumi\jumi\jumi-core\target\jumi-core-0.1.0-SNAPSHOT.jar
[WARNING] We have a duplicate net/orfjackal/jumi/core/INTERNAL/org/apache/commons/io/input/XmlStreamReader.class in C:\DEVEL\Jumi\jumi\jumi-core\target\jumi-core-0.1.0-SNAPSHOT.jar
- Pronouncing Jumi: ['jumi]