From a95dd60df9a1a09faf0d243b723e664298bdf8de Mon Sep 17 00:00:00 2001 From: Jeremiah D Jordan Date: Mon, 18 Sep 2023 11:31:24 -0500 Subject: [PATCH] Squashed commit of the following: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit commit f9e589098e89417cd41ef627b3e1c7371986e3e1 Merge: b0dbc8bd57 d32eed9d68 Author: Jeremiah D Jordan Date: Mon Sep 18 11:28:48 2023 -0500 Merge remote-tracking branch 'datastax/vsearch' into cndb-7632-vsearch-rebase-20230915 commit b0dbc8bd57fde3dd473d4aae85a3063f2156106d Author: Jeremiah D Jordan Date: Mon Sep 18 10:12:30 2023 -0500 Update expected string for error message change. commit f2cafdac9c3e5bf838c10e0078895d56e3271370 Author: qannap <130002578+qannap@users.noreply.github.com> Date: Thu Sep 14 08:41:07 2023 -0700 Optimize partition-aware queries to use bloom filter (#729) Add boom filter field in QueryViewBuilder. Here is the link to the issue: [Vector-72](https://datastax.jira.com/browse/VECTOR-72). commit 9765b23b5b748286391b8e374bab24a31cb5d934 Author: Jonathan Ellis Date: Thu Sep 14 09:47:19 2023 -0500 Vsearch 18 (#741): optimize MergePostingList.advance for the case where it is called repeatedly commit 66c00e094942de72f2f1fdb780ac00239e0aa284 Author: Mike Adamson Date: Thu Sep 14 14:33:41 2023 +0100 Remove smile-nlp and add internal Glove implementation for testing (#743) commit b13098e0cd5a9f961066c0059953b525f5bfa787 Author: Jakub Żytka Date: Thu Sep 14 13:28:01 2023 +0200 CNDB-7363: extend read tracking API to notify about post-filtered, reconciliated partitions and rows (#692) * CNDB-7363: extend read tracking API to notify about post-filtered, reconciliated partitions and rows * RowFilter::getExpressionsPreOrder will now return all row filter's expressions in pre-order commit 2bca65cfe4ba864a8a6719e0e28ddf76e83c17e0 Author: Michael Marshall Date: Sun Sep 10 01:01:47 2023 -0500 Fix JsonTest and AnalyzerViewTest errors (#730) commit a0dde9f89e02ef40d9aa87514eb7b5becfdce8e8 Author: Michael Marshall Date: Fri Sep 8 19:34:09 2023 -0500 Reject SAI creation for invalid combinations of options (#736) commit abd5399f9c1b6feb0f5ba71b279ff5b76d0f92ee Author: Jonathan Ellis Date: Fri Sep 8 11:52:53 2023 -0500 Fix exceptions when executing complex queries (#735) * fix ReorderingRangeIterator.performSkipTo * fix NPE commit acc39a8281cf9163057c00ec9413219b210cc262 Author: Michael Marshall Date: Fri Sep 8 11:04:58 2023 -0500 Analyzer cleanup: add comments and fix docs (#733) commit 4989dff0f150bb65535fa598e53290bca1f3ce0c Author: Michael Marshall Date: Thu Sep 7 08:51:09 2023 -0500 Fix RowAwarePrimaryKey#hashCode for deferred keys (#725) * Fix RowAwarePrimaryKey#hashCode for deferred keys * Use recommendation from code review commit 2268c1c806632d62cf1a29b815740207a0e2939c Author: Jonathan Ellis Date: Wed Sep 6 13:38:39 2023 -0500 optimize MergePostingList.advance -- we don't care what order we evaluate the merged lists in, since we're going to rebuild the PQ anyway, so use a fast iterator instead of a slow poll() loop commit 6bb72c3272fbc8d12427fb0d9f00916b6f6ee9c0 Author: Jonathan Ellis Date: Wed Sep 6 13:22:07 2023 -0500 PKM is not threadsafe, need to allocate a new one for every request commit f1d7c66b54f7b95e12698d7a6ed00a1e8ac958d6 Author: Jonathan Ellis Date: Wed Sep 6 13:10:11 2023 -0500 r/m slightly gratuitous IOException from PKM.Factory signature commit 73e96df69a1861d5a8bda04ab69252a480a78c62 Author: Jonathan Ellis Date: Wed Sep 6 11:35:40 2023 -0500 fix updateTestWithPredicate by removing broken assert (did not account for deleted rows) commit 36327a93b8c1651acd7af1d8b687dd92e08021e3 Author: Jonathan Ellis Date: Wed Sep 6 11:34:44 2023 -0500 add failing updateTestWithPredicate commit 80eb443b65637dc1782cb3a745d4d56431f787d8 Author: Jonathan Ellis Date: Wed Sep 6 11:05:32 2023 -0500 comment upsertTest commit 3f5fbc72547aa790989d3c7bce8b92fd4bb2fb2c Author: Marianne Manaog <91789965+marianne-manaog@users.noreply.github.com> Date: Wed Sep 6 19:42:15 2023 +0100 Feature/Optimize SAI.getPostQueryOrdering as SAI.postQuerySort (#710) * Optimize sorting in postQuerySort method * Remove need for prepareFor method following sort optimization in postQuerySort * Add testOrderResults * Simplify creation of resultSet in the test testOrderResults and keep rows as final in ResultSet.java * Apply DSU pattern to SAI.postQuerySort method and leverage it in SelectStatement.orderResults method * Use Pair.create instead of new Pair * Put back comment on ANN support only * Optimise undecorate step by using List instead of ByteBuffer in the Pair * Add licence to StorageAttachedIndexTest.java * Move StorageAttachedIndexTest.java up a directory * Simplify map to create listPairsVectorsScores commit 23555ab43101cd6010fe9cc320e7429779faa678 Author: qannap <130002578+qannap@users.noreply.github.com> Date: Fri Sep 1 15:29:22 2023 -0700 Add row count field to columnFamilyStore (#713) * Add row count field to columnFamilyStore * Add test to row count field * delete used import * delete used function and separate test * Add row count field to columnFamilyStore * Add test t0 row count field * delete used import * delete used function and separate test * resolve version * return to updated version * restore other tests as vsearch branch --------- Co-authored-by: Michael Marshall commit 1eae389fadecfc6637207933fd6ae1739355dc00 Author: Michael Marshall Date: Thu Aug 31 17:59:53 2023 -0500 Fix NPE in LuceneAnalyzer#end (#721) When testing the analyzer with an integration test, I discovered an NPE from a part of the code that wasn't tested. Here is the code: https://github.com/datastax/cassandra/blob/9cabec504d9f70ddfb345122dc4defed56a40f85/src/java/org/apache/cassandra/index/sai/plan/StorageAttachedIndexSearcher.java#L91-L100 We build the Analyzer to use one of its methods, but not to analyze any text, hence the NPE. Here is the NPE: ``` ERROR [Native-Transport-Requests-1] 2023-08-31 14:35:38,695 QueryMessage.java:121 - Unexpected error during query java.lang.NullPointerException: null at org.apache.cassandra.index.sai.analyzer.LuceneAnalyzer.end(LuceneAnalyzer.java:110) at org.apache.cassandra.index.sai.plan.StorageAttachedIndexSearcher.filterReplicaFilteringProtection(StorageAttachedIndexSearcher.java:99) at org.apache.cassandra.service.reads.DataResolver.lambda$preCountFilterForReplicaFilteringProtection$6(DataResolver.java:300) at org.apache.cassandra.service.reads.DataResolver.resolveInternal(DataResolver.java:341) at org.apache.cassandra.service.reads.DataResolver.resolveWithReadRepair(DataResolver.java:244) at org.apache.cassandra.service.reads.DataResolver.resolveWithReplicaFilteringProtection(DataResolver.java:284) at org.apache.cassandra.service.reads.DataResolver.resolve(DataResolver.java:130) at org.apache.cassandra.service.reads.range.SingleRangeResponse.waitForResponse(SingleRangeResponse.java:59) at org.apache.cassandra.service.reads.range.SingleRangeResponse.computeNext(SingleRangeResponse.java:65) at org.apache.cassandra.service.reads.range.SingleRangeResponse.computeNext(SingleRangeResponse.java:31) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:93) at org.apache.cassandra.service.reads.range.RangeCommandIterator.computeNext(RangeCommandIterator.java:123) at org.apache.cassandra.service.reads.range.RangeCommandIterator.computeNext(RangeCommandIterator.java:45) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:93) at org.apache.cassandra.db.partitions.PartitionIterators$1.hasNext(PartitionIterators.java:154) at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:93) at org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:892) at org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:518) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:495) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:341) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:102) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:290) at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:394) at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:108) at org.apache.cassandra.transport.Message$Request.execute(Message.java:242) at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:111) at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:131) at org.apache.cassandra.transport.Dispatcher.lambda$dispatch$0(Dispatcher.java:78) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165) at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137) at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ``` I am still new to query execution, but I am guessing the unit tests skip this filtering code. I added an integration test that fails without the fix. commit ea97cc414ef3e68ad30e8c666ecf04b719352901 Author: Michael Marshall Date: Wed Aug 30 09:10:14 2023 -0500 Fix failing KDTreeIndexSearcherTest (#717) #680 included a change that makes 5 of the KDTreeIndexSearcherTest tests fail. This PR essentially reverts back to the old behavior, and all of the failing tests pass. commit cee55b1b2139627b331d9f7225c522ca5bee299b Author: Michael Marshall Date: Tue Aug 29 21:56:24 2023 -0500 Remove ability to configure unique query_analyzer (#712) commit c248f905cc13833f9857b9d750a60091afc5c38a Author: Michael Marshall Date: Tue Aug 29 21:55:18 2023 -0500 Fix failing distributed SAI tests using : operator (#715) commit 367e84adf0663e8baab3f9195f3eb81412640b8c Author: Michael Marshall Date: Tue Aug 29 21:55:03 2023 -0500 Test compound predicate queries for : operator (#716) commit 0ea0d95f52ac18522a0cb5a09c1c9ee00069380a Author: Michael Marshall Date: Tue Aug 29 11:10:02 2023 -0500 Refactor SAI analyzer configuration; add built in analyzers (#711) The current `OPTIONS` configuration for the `index_analyzer` does not correctly model the available configuration options. In order to make the configuration more directly match the single tokenizer, and the lists of filters and charFilters, I propose we update the expected JSON schema. Further, we add some default analyzers to simplify the configuration of generic analyzers. commit 67ef0f3794cbd2b9ff41b632205dd087a992e674 Author: Michael Marshall Date: Mon Aug 28 11:58:30 2023 -0500 Return more specific exception on incorrect usage of : operator (#705) * When the `:` operator is used for indexes that do not support it, the current error is a recommendation to `ALLOW FILTERING` or an `AssertionError`. Instead, we should match the `LIKE` behavior and reject queries that attempt to use the `:` operator on columns that are not properly indexed. The `LIKE` conditional is slightly modified to simplify the logic for rejecting unsupported queries. * When the `:` operator is used for `DELETE` and `UPDATE` queries, return a helpful error message. * Update the `toString` method on the `ANALYZER_MATCHES` enum. The `toString` method is used to generate error messages, and it is confusing to see an error message with `: ''` when it really should just be `:`. commit c26cbfa2298d422813e02b439f4db6494bb64a84 Author: Jakub Żytka Date: Mon Aug 28 14:12:15 2023 +0200 Additional testcases for read query tracking with different data models (#698) * Additional testcases for read query tracking with different data models commit bf143a2b3fc97979e5e083f8eac7aca017cd618d Author: Michael Marshall Date: Thu Aug 24 13:28:47 2023 -0500 Ignore default analyzer settings when classifying an SAI as analyzed (#706) ### Problem When creating an SAI, the `AbstractAnalyzer` creates an analyzer in cases where it should not because the logic is too naive. ### Details Users can create SAI indexes with a `NonTokenizingAnalyzer` using the following queries where `?` can be replaced with `true` or `false`: ``` CREATE CUSTOM INDEX ON table(some-column) USING 'StorageAttachedIndex' WITH OPTIONS = { 'normalize' : ? } CREATE CUSTOM INDEX ON table(some-column) USING 'StorageAttachedIndex' WITH OPTIONS = { 'case_sensitive' : ? } CREATE CUSTOM INDEX ON table(some-column) USING 'StorageAttachedIndex' WITH OPTIONS = { 'ascii' : ? } ``` The options are known as "non tokenizing analyzers" because they "analyze" values but do not "tokenize" them. Practically speaking, these analyzers are projections where each input has one and only one output. Lucene calls projections "filters". The `StorageAttachedIndex` default is for normalize = false, case_sensitive = true, and ascii = false. When a default value is passed for one of the options, the `AbstractAnalyzer` incorrectly classifies the index as "analyzed". Instead, the `AbstractAnalyzer` should return the `NoOpAnalyzer` and the index should not be classified as "analyzed". ### Solution * Only construct the `NonTokenizingAnalyzer` when `normalize`, `case_sensitive`, or `ascii` are configured with non default values. * Do not consider an SAI as analyzed if `normalize`, `case_sensitive`, or `ascii` are configured with default values. * Updated some tests to use `=` on the non-analyzed SAI commit 7a6c4755f150ad647aae2bc806d8b91c423db14b Author: Michael Marshall Date: Tue Aug 22 15:57:08 2023 -0500 Add ANALYZER_MATCHES operation; Disallow = on analyzed SAI columns (#702) * Adds the `:` operator, a.k.a the ANALYZER_MATCHES operator. This operator only works on SAI indexes that are analyzed where analyzed means the values stored in the index are derivative values from the raw value in the column. The new operator uses the enum value `100` to prevent future collisions with the upstream CQL implementation. * Disambiguate the `=` operator so that it cannot perform matches on an `SAI` column that is indexed. The primary motivation for this change is to prevent surprising matches. If a query is attempted using `=` where the target column only has analyzed indexes, the query must include `ALLOW FILTERING`. * Updated many tests. commit 77cb7f265e40600288fe90fffd74ef6e428f87fe Author: Jonathan Ellis Date: Fri Aug 18 10:48:54 2023 -0500 comment commit 2dcdd9fd7464a6e26185bfa4ac3cd44b02574fb1 Author: Michael Marshall Date: Thu Aug 17 16:55:51 2023 -0500 Fail SAI index creation when using analyzer on column in primary key (#699) * add testStandardAnalyzer * debugging wip * Fail index creation when using analyzer on column in primary key * Remove unnecessary debug logging * cleanup * Reject all attempts to add analyzer to index on pk columns * Rename noop analyzer test and add explanation --------- Co-authored-by: Jonathan Ellis When the target column for an SAI is part of the primary key and is analyzed by the LuceneAnalyzer, queries do not return correct results. Until the underlying feature is solved, do not allow these kinds of indexes to be created. This change is covered by a test to verify index creation and search works correctly for the `NoopAnalyzer` and another to verify index creation is rejected for the `LuceneAnalyzer`. commit 4f5585be198a949b0264eb64d0403e2adda9ab52 Author: Jonathan Ellis Date: Thu Aug 17 17:41:18 2023 -0500 default vector cache size per segment bumped to 4MB commit 9464bfe3e2bbd958f4f1d60e799019582a9836cb Author: Marianne Lyne Manaog Date: Thu Jul 13 20:32:36 2023 +0100 Deserialize ByteBuffer into a query vector once by moving the deserialization up the call stack commit bc2ef7ff721493b20b1bb76dd2ae37b43d365071 Author: Shaunak Das Date: Wed Aug 16 13:27:12 2023 -0500 VECTOR-79: Simplify population of VectorCache (#695) * use BFS at higher levels so we can avoid a separate cachedNodes set * try to naively cache as much of level 0 as possible * add VectorCacheTest * cleanup and revert to intended behavior of not caching L0 --------- Co-authored-by: Jonathan Ellis commit 1c58e431e55d8b327314ddb82b30bf8eced59269 Author: Shaunak Das Date: Tue Aug 15 11:24:30 2023 -0500 Avoiding sorting node ordinals for level 0 (#697) * avoiding sorting node ordinals for level 0 commit 16eeaf89a8ac39fee616f7a71a3f9a4b4170521e Author: Jonathan Ellis Date: Tue Aug 15 08:47:48 2023 -0500 extract OnDiskVectors to top level class so we can use it in debugging tools (#693) commit 2271a230761a416fbd5c016d47684b388be93989 Author: Jonathan Ellis Date: Wed Aug 9 09:59:45 2023 -0500 VECTOR-77 fix reading vectors past the first 2GB mmap region commit 5705daa97f848454d8d29ab08234296906b3dbc4 Author: Andrés de la Peña Date: Thu Aug 10 18:06:28 2023 +0100 VECTOR-76: Add back vector size guardrail (#694) patch by Andrés de la Peña; reviewed by Brandon Williams and Maxwell Guo for CASSANDRA-18730 commit cb3bedccc255476300528183e8ba30a4e390f57f Author: Jakub Żytka Date: Tue Aug 8 16:52:16 2023 +0200 CNDB-7390: fix static rows not being tracked by ReadTrackingTransformation (#691) commit a8740fb8dae267fb8ddef1a7f5c77e948cd2a0ff Author: Andrés de la Peña Date: Fri Jul 28 15:42:42 2023 +0100 Add system property to reject non-float vectors, true by default (#689) commit 5a4e439dc73b479d79bd427472d382a53b6fe680 Author: Jonathan Ellis Date: Fri Jul 21 13:10:44 2023 +0200 r/m failing emptyIndexTest until we can address it (VECTOR-59) commit eec8b7b387cd041fbd54aef8f0090773962a9e49 Author: Piotr Kołaczkowski Date: Tue Jul 18 17:36:43 2023 +0200 VECTOR-69: Fix LWT test failure caused by null key bounds (#686) * VECTOR-69: Fix LWT test failure caused by null key bounds --------- Co-authored-by: Jakub Zytka Co-authored-by: Jonathan Ellis commit 033ae1a9567dfef88d0214dd34716f2468c5d4e3 Author: Andrés de la Peña Date: Tue Jul 18 15:23:30 2023 +0100 VECTOR-68 Vector dimensions are not being validated correctly (#679) commit e22ff6977f986a4148f931bad4b04645bf93d2e9 Author: Jonathan Ellis Date: Tue Jul 18 06:01:04 2023 +0200 set sai_hnsw_allow_customm_parameters=true for IndexWriterConfigTest commit ccaba92656e78e63765da5c34f562f2c3b582bd0 Author: Mike Adamson Date: Tue Jul 11 16:12:31 2023 +0100 VECTOR-55: Review REVIEWME comments: (#680) * VECTOR-55: Review REVIEWME comments: - Refactor search methods to only return long variant - Remove SSTableQueryContext in favour of QueryContext - Add checking to SegmentMetadata.toSegmentRowId method commit 338c8507de197570f4879019e0e1fc8e1f77025e Author: Jeremiah D Jordan Date: Mon Jul 10 21:46:56 2023 -0500 Fix IndexError when querying a list/set/map column. commit 1b4d6de7e0a051b95c0a9b62c8bc167e23ec65ed Author: Jonathan Ellis Date: Sun Jul 9 21:03:51 2023 +0200 add test that fails before decompose fix and passes after commit 664911ac7de336d3fcffd90083e6a220069bd4e2 Author: Jonathan Ellis Date: Sun Jul 9 13:44:28 2023 +0200 re-use TypeUtil.decomposeVector commit e17493c95ac247ca1da7caa3ccd58be866958d6a Author: Jonathan Ellis Date: Sun Jul 9 13:28:56 2023 +0200 fix casting in deserialize commit d169ff68f6576ce5b150818cb0a52a12abf6f2e0 Author: Jonathan Ellis Date: Sun Jul 9 13:30:02 2023 +0200 fix NPE when using ReadExecutionController.empty commit aa899f55f77df5b157e3e6afa930c377e4cacc57 Author: Zhao Yang Date: Fri Jul 7 10:12:28 2023 +0800 assert no single partition trace for SAI request commit 277251e1aad15e2a5aedaaaae3170d574d224fe5 Author: Jonathan Ellis Date: Thu Jun 29 12:08:05 2023 -0500 - add Tracing.traceSinglePartitions to avoid cluttering range and index queries with noise - update QueryViewBuilder to include a count of sstables per index accessed commit 4682a417fdaaefb135e57b0ebe174c1ff49b7e94 Author: Jonathan Ellis Date: Thu Jul 6 12:13:20 2023 -0500 simplify commit 52d19a1ef5605aebdb87385786dd323057267164 Author: Jeremiah D Jordan Date: Wed Aug 30 14:17:12 2023 -0500 Squashed commit of the following: commit 4c4c1ff802c7667e1289a4e730cc34bf6b3bd007 Author: Jonathan Ellis Date: Wed Jul 5 12:03:22 2023 -0500 add test for creating index after data is added commit 0acaae364c19ed7556b66b5576762ecc131af5fd Author: Jeremiah D Jordan Date: Wed Jul 5 09:23:01 2023 -0500 Revert "enable segment compaction by default for vector indexes -- we should prioritize read performance" This reverts commit a4fa0728337ebf236ad03f6ad708cf51c3dc08c5. commit e3ddb6426de4e89b5fe61ed742b69257865c2d3c Author: Jonathan Ellis Date: Wed Jun 28 10:54:49 2023 -0500 VECTOR-64 set sstable_growth to 0.5 by default commit 3ca75046e6802ce5836a654459b01948c83d1d7b Merge: a4fa072833 10a2a31ae8 Author: Jonathan Ellis Date: Wed Jun 28 10:54:18 2023 -0500 merge ds-trunk commit a4fa0728337ebf236ad03f6ad708cf51c3dc08c5 Author: Jonathan Ellis Date: Wed Jun 28 10:24:41 2023 -0500 enable segment compaction by default for vector indexes -- we should prioritize read performance commit 08153b38549d700505d064c25d7a2070c2ff07aa Merge: 6a063b4f84 1a45fdc5f9 Author: Jonathan Ellis Date: Wed Jun 28 10:18:38 2023 -0500 Merge branch 'vsearch' of github.com:datastax/cassandra into vsearch commit 6a063b4f8458f119d4b99bd6b78cbc8832f27f13 Author: Jonathan Ellis Date: Wed Jun 28 10:13:51 2023 -0500 VECTOR-63 add cassandra.sai.hnsw.allow_custom_parameters defaulting to false to control setting maximum_node_connections and construction_beam_width, which can cause memory usage to explode if used incorrectly commit 1a45fdc5f92e9c82dba6e9e56a5e341806070b13 Author: Jeremiah D Jordan Date: Mon Jun 26 15:57:10 2023 -0500 Add the option to configure the file_cache_size_in_mb in a -D like it is in 6.8-cndb. commit d2e45dab5b57f20fbdd6250d6dc186c83d783ed3 Author: Jonathan Ellis Date: Thu Jun 22 16:13:00 2023 -0500 VECTOR-62 don't flush a graph that only contains deleted vectors commit 4a9afe205c20b1f863e89d4e02138557d225c3fc Author: Jonathan Ellis Date: Thu Jun 22 16:11:11 2023 -0500 r/m obsolete comment commit a925dc22696bcbebcad17a460441e8b84ff97660 Merge: 7c7d160dd9 2b22984fe1 Author: Jeremiah D Jordan Date: Thu Jun 22 09:03:08 2023 -0500 Merge remote-tracking branch 'datastax/ds-trunk' into vsearch commit 7c7d160dd90624b0e365287ef0a2f0b7ef94da53 Merge: a14b387372 7a5e374cea Author: Jonathan Ellis Date: Tue Jun 20 21:42:51 2023 -0500 Merge branch 'vsearch' of github.com:datastax/cassandra into vsearch commit a14b387372ef3f220ece68b019acaafaf0d56f42 Author: Jonathan Ellis Date: Tue Jun 20 17:52:13 2023 -0500 move new options to jvm17-server.options commit dc67582585a90628e026a693497793b7d8a298ed Author: Jonathan Ellis Date: Tue Jun 20 17:47:27 2023 -0500 copy jvm11 options to jvm17 commit b6717410f925486a858c4c9414e417eed50950d7 Author: Jonathan Ellis Date: Tue Jun 20 17:30:48 2023 -0500 enable simd commit 92b9e6e2f136254fb1f8e50732809ea529617f48 Author: Jonathan Ellis Date: Mon Jun 19 15:31:09 2023 -0500 make it run under jdk 20 commit 3e985953637819ac8504955552ce0feba29bdd40 Author: Jonathan Ellis Date: Tue Jun 20 17:13:56 2023 -0500 update to lucene with simd commit 7a5e374cea785678670444121ad36be6170dd59b Merge: 5e5ae4de82 e9ddd5f0d9 Author: Jeremiah D Jordan Date: Tue Jun 20 15:00:24 2023 -0500 Merge remote-tracking branch 'datastax/ds-trunk' into vsearch commit 5e5ae4de824a7e838bbdfdbabc92e8b256911e3f Author: Jonathan Ellis Date: Mon Jun 19 10:57:18 2023 -0500 we weren't calculating length correctly (supposed to match ordinals range) but it didn't matter b/c length() is not called on the search path where these classes are used. make this explicit by throwing in length implementation. commit c0d4055a0bc6914d0220fbd4912d6355e9f6a1d9 Merge: f2c9a7cb59 c605f8c9f0 Author: Jonathan Ellis Date: Mon Jun 19 10:20:02 2023 -0500 Merge branch 'vsearch' of github.com:datastax/cassandra into vsearch commit f2c9a7cb59b113a349992011b739079106274c1e Author: Jonathan Ellis Date: Mon Jun 19 10:19:34 2023 -0500 VECTOR-61 remove deletedOrdinals collection that was not threadsafe wrt add and remove operations; create postingsByOrdinal map that allows Bits operations to look up the postings, instead commit 9744a13b405cbeb1b68894bb08cc24796aedd8f7 Author: Jonathan Ellis Date: Mon Jun 19 09:13:24 2023 -0500 add LVT.testMultiplePostings (works fine) commit c605f8c9f037e724544b21fe2aadcf0d8503d86f Author: Zhao Yang Date: Sat Jun 17 12:44:36 2023 +0800 skip replica-filtering-protection for ANN requests commit 71935bd07c9218ba70f1ffde7b6db1e88009e529 Author: Jonathan Ellis Date: Fri Jun 16 20:20:06 2023 -0500 forgot that maxBruteForceRows has to stay non-final for Byteman commit b264afc07727305ce88ab2f18b8a2643d2510bd3 Author: Jonathan Ellis Date: Fri Jun 16 20:10:19 2023 -0500 VECTOR-54 - validateIndexable also checks for very large vectors (that explode to NaN when we take the cosine) - check query vectors with the same criteria as vectors to index commit 66c31a6dcfcbf81b7657ac9c4119803e026d150a Author: Jonathan Ellis Date: Fri Jun 16 20:07:41 2023 -0500 if CFS.apply catches an InvalidRequestException, keep the same type when we re-throw so it gets back to the client as intended commit e0bdfdbe421a1cca30746604cda9ef5b992002bb Author: Jonathan Ellis Date: Fri Jun 16 17:33:37 2023 -0500 add [failing] emptyIndexTest commit 75cba96f90730b32ea4a39375c72aaff4ffd6221 Author: Jonathan Ellis Date: Thu Jun 15 16:52:46 2023 -0500 re-use bitsets across searches commit 2af2ce8a321f7fea268d81c393da7f8f6d886776 Author: Jonathan Ellis Date: Fri Jun 16 17:05:52 2023 -0500 more asserts that graph is in sane state when we write it commit a63492686d13e12b799acdf50c6f0c20dd680f21 Author: Jonathan Ellis Date: Fri Jun 16 17:13:23 2023 -0500 update lucene commit 4c6b43e3606b8c9eb5e4ec4cf0817dc9b061c555 Author: Jonathan Ellis Date: Fri Jun 16 16:24:33 2023 -0500 add debugging information when node is not found on level commit 81c8e730bd4ca314a81bc13e30b197603a8eb005 Author: Jonathan Ellis Date: Fri Jun 16 11:57:24 2023 -0500 add similarityWithAnn test commit 87c8da9a5dddd10215a84c07a09dbaaecaa522b8 Author: Jonathan Ellis Date: Wed Jun 14 18:26:02 2023 -0500 add tests to confirm that zero-length vectors are rejected commit 2510c1b987d846821e7095fbc2c368fa51d9a15d Merge: a3d9e33554 e86f91c568 Author: Jonathan Ellis Date: Mon Jun 12 17:09:22 2023 -0500 Merge branch 'vsearch' of github.com:datastax/cassandra into vsearch commit a3d9e3355470f6400328cd7ac1eb07ff06f8e00b Author: Jonathan Ellis Date: Mon Jun 12 17:08:43 2023 -0500 use ArrayList in CVV instead of HashMap since we know the keys are consecutive commit e86f91c568f8a82e83e61bd0dd768478a7584cfc Merge: 77c1bc11f3 8a7a6d9c4f Author: Jeremiah D Jordan Date: Mon Jun 12 09:25:06 2023 -0500 Merge remote-tracking branch 'datastax/ds-trunk' into vsearch commit 77c1bc11f31e4ff001c7eff1e6fe50812e136a45 Author: Jeremiah D Jordan Date: Mon Jun 12 09:24:56 2023 -0500 Revert "Revert this after CNDB-6974 is fixed." This reverts commit 8e04280312583a18085e7e7b9d31810790b039f4. commit cae30e9879c463d80de2a2b7e4a642979803c13a Author: Jeremiah D Jordan Date: Mon Jun 12 09:23:39 2023 -0500 Revert "Revert this after CNDB-6974 is fixed." This reverts commit 67a2b7eba3c957a993726d37706643979e92a3a3. commit 46d1a45de84f40311f193beba3ca5fc3fe7450d7 Author: Jonathan Ellis Date: Mon Jun 12 09:03:03 2023 -0500 replace our TODO entries with VSTODO to make them easier to find commit 415b3f29e8e7f1526e28655308a1937a1175e575 Merge: 2fd8b848be 481d29721a Author: Jonathan Ellis Date: Mon Jun 12 09:02:16 2023 -0500 Merge branch 'vsearch' of github.com:datastax/cassandra into vsearch commit 481d29721a7993e9babd44f792be55aca925d41a Author: Zhao Yang Date: Mon Jun 12 21:26:43 2023 +0800 Fix VectorMemtableIndex to handle max token and fix Segment#intersects (#670) * Fix VectorMemtableIndex to handle max token and min/max bound * Fix Segment#intersects to compare bound instead of token and add tests for range search * make brute force rows per query for VectorMemtableIndex * apply feedback on Segment#intersects * add comments to VectorMemtableIndex#search * Fix SegmentTest commit 2fd8b848be82e62999e64b57cb9800d03de8e953 Author: Jonathan Ellis Date: Mon Jun 12 07:51:18 2023 -0500 clean up a couple REVIEWME commit 3c066e07be6188c3f48c51c02cfc99b395e2a6bc Author: Zhao Yang Date: Sun Jun 11 14:26:42 2023 +0800 fix flaky VectorDistributedTest commit 6c719a752e90e031916dddf6cc7b7db23627bc38 Author: Jonathan Ellis Date: Sat Jun 10 10:48:01 2023 -0500 fix use of bruteForceRows -- should be larger of limit,maxBruteForceRows commit 0a2279d7f010bcb68053bb14ea2974295b21d1c4 Author: Jonathan Ellis Date: Sat Jun 10 10:46:34 2023 -0500 use maxBruteForceRows when deciding whether to skip ANN commit 1859f7355231ce7972e3d4b7af70e5d2967da516 Author: Jonathan Ellis Date: Sat Jun 10 10:27:03 2023 -0500 simplify partitionKeySearchTest using euclidean distance commit 4f40e1c2d9b1c30675eeaee7d800a8113cce97c0 Author: Jonathan Ellis Date: Sat Jun 10 10:09:24 2023 -0500 typo commit 7dbe85ac9a248aa6a0985839df1f40ceb2c08e6c Author: Jonathan Ellis Date: Sat Jun 10 10:08:23 2023 -0500 rename methods that return Bits but had bitset in their names commit 2861bfdba8f2638a1e3f9e81df9cad3878d0e544 Author: Jonathan Ellis Date: Sat Jun 10 10:06:51 2023 -0500 simplify skipANN commit fbfe4bcdd03a66f4dedfa91ce9d06206f89827cd Author: Zhao Yang Date: Sat Jun 10 16:36:56 2023 +0800 optimize VectorIndexSearcher#searchPosting - return empty posting if key range is not found in current sstable - return empty posting if all row ids are shadowed - skip ANN if matching row ids are less than limit commit 168911b70888a3dda0ddf4f615af1b7ff54f43e3 Author: Zhao Yang Date: Sat Jun 10 20:54:41 2023 +0800 fix data visibility issue during flush: make sure SSTableAddedNotification is sent before MemtableDiscardedNotification commit c5dd47e3ce1c7276703fa3f89a9932a3b2cd43b7 Author: Zhao Yang Date: Sat Jun 10 12:31:53 2023 +0800 fix primaryKeySearchTest and partitionKeySearchTest to use correct selector and expected results commit 2e636529c63d0d726868d6df9b3e11d15d3871d8 Author: Zhao Yang Date: Sun Jun 11 08:31:39 2023 +0800 Vector-48: index#update is not triggered by partition/range deletion,… (#665) * Vector-48: index#update is not triggered by partition/range deletion, we have to fix VectorMemtableIndex to include shadowed primary keys - during flush, append ordinals that are removed by partition/range deletion into deletedOrdinals set * add comments to VP * revert redundant variables * simplify --------- Co-authored-by: Jonathan Ellis commit daa2623b880e4b82d99204fe718db22e52ae3b42 Author: Jonathan Ellis Date: Sat Jun 10 13:54:38 2023 -0500 use mmapped builder in OnDiskHnswGraphTest commit 553778a22cfa8757daf3b3ec0e2cfa6fc7ba29cf Author: Jonathan Ellis Date: Sat Jun 10 08:34:41 2023 -0500 fix logic in partitionKeySearchTest, test still fails commit a7f5a2f824085691da80ef62e37a430e5637c31c Author: Jonathan Ellis Date: Fri Jun 9 17:57:34 2023 -0500 ignore invalid vectors during build against existing data, instead of failing the build commit 2af203646a2e57fca0a2521c31ba4269b0d8f326 Author: Jonathan Ellis Date: Fri Jun 9 17:43:47 2023 -0500 switch from ignoring zero vectors to throwing IRE commit 4c98fbb447a275cbf557ae0adf8f4f8a30fba17f Author: Jonathan Ellis Date: Fri Jun 9 17:15:57 2023 -0500 Revert "if there are no rows in the range to search, return EmptyPostingsList instead of giving the hnsw search an empty Bits (which is a pathological case for the search)" This reverts commit 6e5eab787a3add985dc30eca82c5dee2646f5564. commit da99598a5d11dfa4ecac3b1eeeb9d19a4bad21e5 Author: Jonathan Ellis Date: Fri Jun 9 17:09:30 2023 -0500 don't attempt to add zero vector to cosine indexes commit a77792447d33600f69dd0a75a280a2b6dac51389 Author: Jonathan Ellis Date: Fri Jun 9 16:43:34 2023 -0500 add failing partitionKeySearchTest commit b061dd1c77f30c67324da4f94ea8b5a22d50fba7 Author: Jonathan Ellis Date: Fri Jun 9 10:52:08 2023 -0500 inline the test ops commit 6e5eab787a3add985dc30eca82c5dee2646f5564 Author: Jonathan Ellis Date: Fri Jun 9 10:51:09 2023 -0500 if there are no rows in the range to search, return EmptyPostingsList instead of giving the hnsw search an empty Bits (which is a pathological case for the search) commit 518c055dfeabacf0fa0863e40bf814058f2ad981 Author: Jonathan Ellis Date: Fri Jun 9 10:28:57 2023 -0500 cleanup commit e922b03ffe5436dc1341dbe20ee7aaca9901058f Author: Jonathan Ellis Date: Fri Jun 9 10:27:53 2023 -0500 failing tests for primary key search commit 428e4b713e25a5820d02ca790c30e9f1477c0f44 Author: Jonathan Ellis Date: Fri Jun 9 10:21:59 2023 -0500 cleanup commit 5319bcdf3b694e861509b2a9f549f39bf51cd253 Author: Jonathan Ellis Date: Fri Jun 9 10:20:02 2023 -0500 move testInvalidColumnNameWithAnn to VectorInvalidQueryTest commit d8b64845e59c0bd3658a16df21dcc75564417e3b Author: Jonathan Ellis Date: Fri Jun 9 10:12:40 2023 -0500 upgrade lucene to reduce Integer boxing on build path commit a62356e49c0a32f61732b8525d2bb655f46dd767 Author: Jonathan Ellis Date: Fri Jun 9 10:11:55 2023 -0500 cleanup commit b754eb7f021d10d0ae8de83ebdf2ce2f19ce59d8 Author: Jonathan Ellis Date: Fri Jun 9 08:29:34 2023 -0500 replace CHM with NBHMLong to avoid boxing commit 53d232ad2bcddefd72b10542f703cf36728c05f5 Author: Piotr Kołaczkowski Date: Fri Jun 9 12:17:46 2023 +0200 STAR-550 Handle SAI AbortedOperationException AbortedOperationException is thrown by SAI when index search hits a timeout. Now instead of allowing this exception to bubble up to the top and be eventually logged as error, we catch and swallow it in the InboundSink after creating the query response. Additionally now we also we set a proper error code (TIMEOUT) in the response, so the client has a hint on what happened. commit 5e0c5eb3c54b58002162ce3e39d0891643e1f663 Author: Matt Fleming Date: Fri Jun 9 08:44:10 2023 +0100 CNDB-7037: Check that memtable exists before flushing and avoid IOOBE (#664) For offline services such as the compactor it's possible to not have live memtables. Account for this during flushing after removing indexes to avoid triggering an IndexOutOfBoundsException: Error happened while updating the schema java.lang.IndexOutOfBoundsException: Index: -1 at java.base/java.util.Collections$EmptyList.get(Collections.java:4483) at org.apache.cassandra.db.lifecycle.View.getCurrentMemtable(View.java:106) at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:1115) at org.apache.cassandra.db.SystemKeyspace.forceBlockingFlush(SystemKeyspace.java:552) at org.apache.cassandra.db.SystemKeyspace.setIndexRemoved(SystemKeyspace.java:576) at org.apache.cassandra.index.SecondaryIndexManager.markIndexRemoved(SecondaryIndexManager.java:837) at org.apache.cassandra.index.SecondaryIndexManager.removeIndex(SecondaryIndexManager.java:420) commit c241f7d41ee46c28f60d691611f32e071dac6684 Author: Jonathan Ellis Date: Thu Jun 8 19:19:09 2023 -0500 read neighbors using .intBuffer since we know that the HNSW searcher always reads all the neighbors commit 32f021a4f0623e57d82d70c54350eab6f0f57db7 Author: Jonathan Ellis Date: Thu Jun 8 18:08:30 2023 -0500 VECTOR-51 leave vectors as ByteBuffers during compaction to avoid 2x memory usage incurred by keeping them around as float[] as well commit 32d8cbbd19301228530d3d84f7a7e4120b4d4422 Author: Jonathan Ellis Date: Thu Jun 8 17:37:08 2023 -0500 r/m node cache from query metrics since there is nothing the operator can act on there commit 890df8e6c36c5c9e1314dbcc0a681415401120dd Author: Jonathan Ellis Date: Thu Jun 8 14:15:25 2023 -0500 add more information to exception when reading row offsets goes wrong commit 5c7fe0cfed1edb10f81b134f8949dd1b5abb2781 Author: Jonathan Ellis Date: Thu Jun 8 13:26:17 2023 -0500 revise ordinals cache as follows: // cache full levels including neighbors up to neighborsRamBudget, starting with the top level, // but always cache all levels above the bottom two levels -- this will be ~1% of the graph. // then on L1, cache at least the offsets // L0 we do not cache since we only need one extra seek (no bsearch) to read the neighbors offset commit e131014c5cba59624e0ed9fac142abe5340e2fc0 Author: Jonathan Ellis Date: Thu Jun 8 13:53:10 2023 -0500 add deletes test commit 7c5a880f007535d9f03b7c7d9d3e84dcc79ef3d7 Author: Jonathan Ellis Date: Thu Jun 8 13:34:42 2023 -0500 must use graph.size for bitset size -- not correct to use max rowid, since deleted ordinals will not have a rowid but will still be in the graph commit 0654fbb61aec86deae69edf83fb1cf653ed7df65 Author: Jonathan Ellis Date: Thu Jun 8 08:44:28 2023 -0500 info -> debug for validatePerIndexComponents commit b289db2362b928f29b6c5c6289e09b33f0be1e88 Author: Jonathan Ellis Date: Wed Jun 7 18:08:18 2023 -0500 update lucene commit 347dae644b0d395014d8faf2affd48a6b2546e96 Author: Jonathan Ellis Date: Wed Jun 7 18:23:41 2023 -0500 undo burntest logging config change commit 87a6a84f38d0a0423acd9a0dda67d386876681b5 Author: Jonathan Ellis Date: Wed Jun 7 15:20:06 2023 -0500 add debug logging commit 20013bc865c7fb2c34111c98362aaf997dc724dc Author: Jonathan Ellis Date: Wed Jun 7 14:50:39 2023 -0500 add a bit more information to exception when we fail in index construction from disk commit 86389ceae8188b8e427d8a528fada2f913f1f633 Author: Jonathan Ellis Date: Wed Jun 7 12:25:31 2023 -0500 reduce test vector count from "all of them" to 200 commit 20177a1c84ba34babc82d8c1aa4e3436321b5d9f Merge: 2e2fce09ee f2c697ac46 Author: Jonathan Ellis Date: Wed Jun 7 12:21:20 2023 -0500 Merge branch 'wip' into vsearch commit f2c697ac46348301260c5947ade4ebed7dee90ee Author: Jonathan Ellis Date: Wed Jun 7 12:16:28 2023 -0500 multipleSegmentsMultiplePostingsTest commit 59e4a1c53e1c19a7ebba4a0834229e61f9e4d3ce Author: Jonathan Ellis Date: Wed Jun 7 12:16:04 2023 -0500 cleanup commit 2e2fce09eeb5bef4780cecf7c7b5e492a810f67a Author: Zhao Yang Date: Wed Jun 7 23:33:00 2023 +0800 fix OnDiskOrdinalsMap to seek to segment offset before reading (#661) commit 9502f1040229f2be6619f7e4497dc25ca5126b39 Author: Jonathan Ellis Date: Wed Jun 7 10:04:38 2023 -0500 fix build commit 3535444b57366fe705e0fd8c1ca095e60ed2a706 Author: Jonathan Ellis Date: Wed Jun 7 09:19:41 2023 -0500 add write-only workload commit a112b0965cb298f293b6e384bafc92a1d5a1fb8f Author: Zhao Yang Date: Wed Jun 7 21:15:53 2023 +0800 VECTOR-44: improve in-memory partition-restricted query perf (#660) * VECTOR-44: improve in-memory partition-restricted query perf - using post-filter top-k processor instead of ANN: 14x improvement on partition-restricted query in LongVectorTest commit cf1ec3be50ddbe40ae99dfc34eea51b9fc5c2648 Author: Zhao Yang Date: Wed Jun 7 21:06:37 2023 +0800 Vector-45: fix bitsetForShadowedPrimaryKeys to skip shadowed primary keys outside of current sstable/segment (#659) * VECTOR-45: fix bitsetForShadowedPrimaryKeys to skip shadowed primary keys outside of current sstable/segment * return empty iterator if results are empty instead of ReorderingRangeIterator commit 87217b4f49935fbbd13c73f4d0aadb84836696a7 Author: Jonathan Ellis Date: Wed Jun 7 07:31:46 2023 -0500 don't make areL0ShardsEnabled final, it breaks mocks commit 2bb3a299df43561369544b662674872575975b7c Author: Matt Fleming Date: Wed Jun 7 10:56:08 2023 +0100 VECTOR-42: Check columns exist to avoid NPE during ANN expr binding (#657) Using an non-existent column with an ANN expression triggers an NPE like so, java.lang.NullPointerException: null at org.apache.cassandra.cql3.ArrayLiteral.forReceiver(ArrayLiteral.java:43) at org.apache.cassandra.cql3.ArrayLiteral.prepare(ArrayLiteral.java:54) at org.apache.cassandra.cql3.Ordering$Raw$Ann.bind(Ordering.java:178) at org.apache.cassandra.cql3.Ordering$Raw.bind(Ordering.java:139) Use TM.getExistingColumn() which throws InvalidRequestException if the column is undefined. commit f29d4528fc43f34889e560acafa810eee1b88ba9 Author: Jonathan Ellis Date: Tue Jun 6 21:18:47 2023 -0500 restore query timeouts commit 6e5734e52b41ab1e355d202ad0433440828fbb75 Author: Jonathan Ellis Date: Tue Jun 6 19:08:52 2023 -0500 looking at performance over time in LongVectorTest commit f96af8f0b9185163a49156bdab107801d2588bf0 Author: Jonathan Ellis Date: Tue Jun 6 16:00:23 2023 -0500 log level = info for burn tests commit 1247cf372927cc185be5c9767a06fdd15f102dbe Author: Jonathan Ellis Date: Tue Jun 6 17:48:00 2023 -0500 write the in-memory deleted ordinals to disk at the start of the postings component commit e319d71bd6b0d265f616c807dcaf4a9d9fc38aef Author: Jonathan Ellis Date: Tue Jun 6 16:36:27 2023 -0500 don't allocate unnecessary objects on the happy path of no tombstones commit 6d146a8460a17d146de756aab2ccbbf2abdb967a Author: Jonathan Ellis Date: Tue Jun 6 16:19:56 2023 -0500 VECTOR-23: force UCS to not shard L0 commit 79d6e177093312bc1b951d6740ad45ce8a6c5875 Author: Jonathan Ellis Date: Tue Jun 6 16:01:05 2023 -0500 mark AutoCloseable commit 5ec2f7d7f792bfa2dcf582ea4eed3fa8745fdaff Author: Jonathan Ellis Date: Mon Jun 5 14:37:15 2023 -0500 Support string literals as vectors Co-authored-by: Andrés de la Peña commit 73a53c37536835e462e3cf17c452027c7aa591ed Author: Jonathan Ellis Date: Tue Jun 6 12:25:39 2023 -0500 (take 3, this time with the code changes) fix race conditions across concurrent inserts + searches in memtable commit 51f4f419d31916a81088f171b158f1e002b5c800 Author: Jonathan Ellis Date: Tue Jun 6 12:25:39 2023 -0500 (take 2) fix race conditions across concurrent inserts + searches in memtable commit 94097a1f7cf1831c03e52b1f539bdc2e42fa038b Author: Jonathan Ellis Date: Tue Jun 6 12:35:32 2023 -0500 Revert "fix race conditions across concurrent inserts + searches in memtable" This reverts commit c4d41b492190fb2644f83bf4902acf71d1e4f891. commit c4d41b492190fb2644f83bf4902acf71d1e4f891 Author: Jonathan Ellis Date: Tue Jun 6 12:25:39 2023 -0500 fix race conditions across concurrent inserts + searches in memtable commit 22143a8c5db160ef0fa7687ff3e8cb227de160f0 Author: Jonathan Ellis Date: Tue Jun 6 10:28:30 2023 -0500 fix AOOB in bruteForceRows logic commit b5624f62c88378e03260399440edf4d335ccc3af Author: Jonathan Ellis Date: Tue Jun 6 10:12:53 2023 -0500 call deserializeFloatArray (which gives float[]) instead of deserialize (which gives List) commit 13ad40718737b54652c875d81ec20992b5828777 Author: Jonathan Ellis Date: Tue Jun 6 08:36:32 2023 -0500 create CassandraHnswGraphBuilder with concurrent and serial implementations, so that single-threaded compaction doesn't have to pay the concurrency overhead commit b83f4e4007e3d597963023b8ae32f4d0934b792d Author: Jonathan Ellis Date: Tue Jun 6 08:00:44 2023 -0500 add ASL header to injections.md to make CI happy commit 16074336b1322d4bade40eb71562ca50221399dc Merge: 227c6c13a3 1ff6992354 Author: Jonathan Ellis Date: Tue Jun 6 07:48:04 2023 -0500 Merge branch 'VECTOR-37' into vsearch commit 227c6c13a3cfe42484fc560d629d54a93f8265e0 Author: Jaroslaw Grabowski Date: Tue Jun 6 13:02:37 2023 +0200 CNDB-7007 return expired tables level from getLevels commit 312d07de8d297cdede39017354b678f2fb1b1006 Author: Jonathan Ellis Date: Tue Jun 6 07:31:26 2023 -0500 randomize our brute force threshold, which will get the actual index scans exercised more commit 0b897961869965629893c94250b7d4b1fb7c0f86 Author: Zhao Yang Date: Tue Jun 6 13:44:01 2023 +0800 Reference ANN sstable indexes in case of ann hybrid search (#653) * Reference ANN sstable indexes in case of ann hybrid search * simplify --------- Co-authored-by: Jonathan Ellis commit 1ff6992354e8411f9f7c48a76f0aadb4a00f2789 Author: Jonathan Ellis Date: Mon Jun 5 21:55:14 2023 -0500 per-query hnsw metrics commit d3d6ae107737020eb0ca3b00c117d5bc84027dc0 Author: Jonathan Ellis Date: Mon Jun 5 21:16:14 2023 -0500 reduce test size to prevent Jenkins OOM commit 7765ac73fd29c9664b6f8e946c067d6f1b0a9a03 Merge: 5a04dbcf67 c5cd09cebf Author: Jonathan Ellis Date: Mon Jun 5 21:15:44 2023 -0500 Merge branch 'vsearch' of github.com:datastax/cassandra into vsearch commit c5cd09cebf0c63f7bafcd2f4548395efe68b12cd Author: Zhao Yang Date: Tue Jun 6 09:38:42 2023 +0800 Skip warning log about receiving a range that is not owned by the current replica, see VECTOR-30 commit 5a04dbcf6738d7681732e28518d6eb37f74357b8 Author: Jonathan Ellis Date: Mon Jun 5 11:21:15 2023 -0500 add injections.md from bdp repo commit baf9cce301a39f9506bd60c5a75d2011150f2fea Author: Jonathan Ellis Date: Mon Jun 5 10:02:02 2023 -0500 VECTOR-36 fix looking up indexes and SSTableContext by SSTableReader, because the SSTR will be removed from internal structures once it's compacted commit e7fdc8454e750e2ee886c4ab27fb09abb46303fa Author: Jonathan Ellis Date: Mon Jun 5 09:53:51 2023 -0500 limitToTopResults should skip rows that are not in the current segment (or more preciesely, have no vectors that were indexed) commit 2f64695a90651d9e76eee0a3399032ea241c983c Author: Zhao Yang Date: Mon Jun 5 14:55:20 2023 +0800 Vector 6 take 2: reapply Vector-6 commit and fix NPE in SelectStatement (#648) * Revert "Revert VECTOR-6" This reverts commit 63a6ff609a02e4f7d9f5499f1af48ca5e58f63a8. * Vector-6 take 2: - fix NPE in SelectStatement by skipping reversed() for null ColumnComparator - fix getOrderingColumns() to return LinkedHashMap to preverse ordering columns' order - fix VectorLocalTest compilation - remove debug log in CassandraOnHeapHnsw commit 9da51b315d207417de4f3e005616e275c62c74cb Author: Zhao Yang Date: Mon Jun 5 12:38:21 2023 +0800 fix SAI test failures in vsearch branch (#649) - BatchlogEndpointFilterTest - rangeRestrictedTest - SegmentFlushTest - SegmentMergerTest - OperationTest - RangeIntersectionIteratorTest commit dc53d7db5f0f8fe1c780f6d2b7580794f7b6b5fb Author: Jonathan Ellis Date: Sat Jun 3 13:35:11 2023 -0500 create views for ordinals map so we don't have to open a new Reader for each method call commit c6385022ff2949c23e0dd0161a043cf4061f050c Author: Jonathan Ellis Date: Sat Jun 3 08:19:32 2023 -0500 add assert sstableContext != null commit f340e7590db578b71201326e82006f72dd491047 Author: Jonathan Ellis Date: Sat Jun 3 07:37:36 2023 -0500 on-disk Searcher bits should not need to be growable commit fd5cc7f989a47d03689dd29495b974697ad1338b Author: Jonathan Ellis Date: Sat Jun 3 07:37:18 2023 -0500 add asserts commit 6436e5f4ad49881a62c1442692edfedfe31968dc Author: Jonathan Ellis Date: Sat Jun 3 07:21:50 2023 -0500 cleanup commit 32c8e005740d3c7ec0b1008091728e12260aecdd Author: Jonathan Ellis Date: Fri Jun 2 13:57:18 2023 -0500 comment commit 091d96832ff9a68f6ea7857eafd2ec7ee0c09a0b Author: Jonathan Ellis Date: Fri Jun 2 13:32:24 2023 -0500 r/m unused search method returning iterators over PrimaryKey commit 4e78a4b429b0ba6690871df9a91c8ffd8d2e53fa Author: Jonathan Ellis Date: Fri Jun 2 11:25:30 2023 -0500 we can use the slightly more lightweight RangeConcatIterator instead of RangeUnionIterator when combining results from different Segments commit e7a9186bf3426ad025e809481e47db85a7bf7190 Author: Jonathan Ellis Date: Fri Jun 2 11:22:35 2023 -0500 r/m obsolete FIXME commit c76ee9ea0cab5b70c535de3f50b81cf333bf94a2 Author: Jonathan Ellis Date: Sat Jun 3 07:12:43 2023 -0500 add testAppendedGraphs commit d00bdbfb6acf9f08d2bdfa2ef06e833e567cf9eb Author: Jonathan Ellis Date: Sat Jun 3 06:57:37 2023 -0500 move write-to-File to test class commit b10fd1100daf4cde0291b3b4ef06f5ceb3ca650c Author: Jonathan Ellis Date: Sat Jun 3 06:46:01 2023 -0500 fix Ref leaks in test commit 0656e558bb51172df4e0cde1ebe92946d4dcec8b Author: Jonathan Ellis Date: Sat Jun 3 06:26:05 2023 -0500 fix tests to use View commit 92797ced03ed26f9a5cce398504d6f0de519b899 Author: Zhao Yang Date: Sat Jun 3 19:19:38 2023 +0800 VECTOR-35: fix vector on-disk writer to append segments (#647) commit 22f5c01860367586d0cc39def7d975a6d6bae784 Author: Jonathan Ellis Date: Fri Jun 2 18:54:41 2023 -0500 add "Vector indexes only support ANN queries" check commit bfc0b056dbacd0a1eca86020e901f545dca7670e Author: Jonathan Ellis Date: Fri Jun 2 17:34:24 2023 -0500 split invalid requests into separate test class commit f66768c35920101382c1975c9d0ba0e3301b7c61 Author: Jonathan Ellis Date: Fri Jun 2 17:28:56 2023 -0500 add specific error message when trying to do ANN without an index commit 63a6ff609a02e4f7d9f5499f1af48ca5e58f63a8 Author: Jonathan Ellis Date: Fri Jun 2 12:31:22 2023 -0500 Revert VECTOR-6 This reverts commits: a37009c187edeba68389d239dc1b9f40519b1187 5565690fbe1056d4c159ddbe233fa22c7695320a e7733bb8f858a16b082b8a5c64d0322db6f6271a commit 3d496662b259470b505d88212074c436660c27ad Merge: fab67bb134 4bdae7e362 Author: Jonathan Ellis Date: Fri Jun 2 11:05:53 2023 -0500 Merge branch 'vsearch' of github.com:datastax/cassandra into vsearch commit 4bdae7e36213f5112efce085696dd24aaa0adfad Author: Mike Adamson Date: Fri Jun 2 14:36:01 2023 +0100 Stabilise random tests using word2vec model vectors commit 2ddc3ec6b11f6810f6c4cbaf6b62bfe46d2d1b6c Merge: 8e04280312 1f9179002a Author: Jeremiah D Jordan Date: Fri Jun 2 10:05:15 2023 -0500 Merge remote-tracking branch 'datastax/ds-trunk' into vsearch commit 8e04280312583a18085e7e7b9d31810790b039f4 Author: Jeremiah D Jordan Date: Fri Jun 2 09:51:06 2023 -0500 Revert this after CNDB-6974 is fixed. commit fab67bb13499691f813c893cd39a3bbd406653d2 Author: Jonathan Ellis Date: Fri Jun 2 08:59:12 2023 -0500 vector cache defaulting to 1MB per segment commit 67a2b7eba3c957a993726d37706643979e92a3a3 Author: Jeremiah D Jordan Date: Thu Jun 1 17:49:35 2023 -0500 Revert this after CNDB-6974 is fixed. commit 4538956b31d8c40d2e9b603b3b0a3392343e1853 Merge: 7518680a4a af4c83aef4 Author: Jonathan Ellis Date: Thu Jun 1 16:17:14 2023 -0500 Merge branch 'ds-trunk' into vsearch commit 7518680a4a17cb2589cf06a9175befc10c9eab1a Author: Jonathan Ellis Date: Thu Jun 1 16:14:33 2023 -0500 re-use float[] across calls to vectorValue to avoid allocation overhead (credit to Jake for the idea) commit d20bd8aedeb6232358dd16f901e2708f217b8108 Author: Jonathan Ellis Date: Thu Jun 1 15:35:17 2023 -0500 optimize vectorValue() with direct access to the mmap-ed region commit 24572025af5b7893279b5b03c58555b682f3abdb Author: Jonathan Ellis Date: Thu Jun 1 14:33:55 2023 -0500 decomposeVector does not modify the underlying buffer, so no need to duplicate() here commit efef35d47a24222fae300d1cbd5a801e9ed79442 Author: Jonathan Ellis Date: Thu Jun 1 14:29:30 2023 -0500 specialize deserializeFloatArray for ByteBuffer and FloatBuffer. this saves about 3% total CPU on search workloads commit 63c27f5550d6c3e22d960b9a42b9097029e3b760 Author: Jonathan Ellis Date: Thu Jun 1 11:04:23 2023 -0500 default target size of 5GB commit 69c55c8d07c6e1d58b4829c153f5ebf8a4343669 Author: Jonathan Ellis Date: Thu Jun 1 07:20:38 2023 -0700 make the on-disk hnsw code threadsafe using FileHandle.createReader (#641) commit b194cae2e56133b946d0231768254064a47f01b6 Merge: 7ee6422084 23c2891e7a Author: Jonathan Ellis Date: Thu Jun 1 06:22:17 2023 -0500 Merge branch 'ds-trunk' into vsearch commit 7ee642208485c9e2ae62aa662dbbddc7b295bae1 Merge: c286ec0ee2 a37009c187 Author: Jonathan Ellis Date: Wed May 31 20:41:48 2023 -0500 Merge branch 'VECTOR-6' into vsearch commit c286ec0ee231ededded18e81357edebe72064c59 Author: Jonathan Ellis Date: Wed May 31 20:40:21 2023 -0500 add testLargeGraph commit a37009c187edeba68389d239dc1b9f40519b1187 Author: Zhao Yang Date: Thu Jun 1 08:50:07 2023 +0800 cleanup unused code commit 6246dbe3a970a0f4672701632003cdf576705c24 Author: Jonathan Ellis Date: Wed May 31 16:43:42 2023 -0500 comment commit c7420440c7d3c81eac96e716148d570ae3cceb1d Author: Jonathan Ellis Date: Wed May 31 16:43:37 2023 -0500 encapsulate shadowedPrimaryKey better commit a1a25c33b8bec9d78354dd2c1d992444db2bf76f Author: Jonathan Ellis Date: Wed May 31 15:57:52 2023 -0500 make hnsw cache size configurable, and make the default 128KB (roughly the size of the bloom filter for a 1GB sstable) commit 55747d2351e915645697fa8dec404c749540456f Author: Andrés de la Peña Date: Wed May 31 19:28:47 2023 +0100 Replace FunctionParameter.sameAsFirst by FunctionParameter.sameAs, allowing type inferrence in both directions for vector functions commit 5565690fbe1056d4c159ddbe233fa22c7695320a Author: Jonathan Ellis Date: Wed May 31 12:59:42 2023 -0500 cleanup and fix commit e7733bb8f858a16b082b8a5c64d0322db6f6271a Author: Zhao Yang Date: Wed May 24 16:01:26 2023 +0800 VECTOR-6: return vector results to client in ANN order instead of token order commit 771067d1475c94484da178236928d2bad78e00b1 Author: Mike Adamson Date: Wed May 31 18:33:55 2023 +0100 Fix annOrderingMustHaveLimit test commit 8dd76a73541e70da0da022dd463e6711a315f971 Author: Mike Adamson Date: Wed May 31 18:30:49 2023 +0100 Improve the validation of ORDER BY ANN OF (#639) * Improve the validation of ORDER BY ANN OF * Change to hasNonClusteredOrdering and improve limit message commit 1ef6f2e3418db19af23f0eae24cd344d72290455 Author: Jonathan Ellis Date: Fri May 26 18:34:25 2023 -0500 add vector similarity functions commit f4e3a39a8ece623647ba7891981e7c5ee10ed96b Author: Jonathan Ellis Date: Wed May 31 09:39:45 2023 -0500 pull in the smallest set of changes possible from 93e0ae9a to get FunctionParameter.sameAsFirst commit 0ef2614346c9426a966004df220221f74353de70 Author: Jonathan Ellis Date: Wed May 31 06:22:41 2023 -0700 Add support for updates + deletes (#636) commit 5da9fefe1e18f733bb8ffd0a35ef2e1ac3cbd975 Author: Jonathan Ellis Date: Tue May 30 16:32:02 2023 -0500 rename SegmentOrdering.reorderOneComponent to limitToTopResults to align with MemtableOrdering commit caf6eb2154ddf57c88d69c6246404b65e183057d Author: Mike Adamson Date: Tue May 30 22:41:29 2023 +0100 Fix V1SearchableIndex.reorderOneComponent to use segments (#635) commit 72cfa94e9389d0078d64aee60d26b8b01c6ca22f Author: Piotr Kołaczkowski Date: Mon May 29 20:17:44 2023 +0200 VECTOR-14: Move ANN OF expressions from WHERE to ORDER BY Before: SELECT * FROM table WHERE ANN OF ; After: SELECT * FROM table ORDER BY ANN OF ; commit 226266ef124fc220819d328ed8547c5d86626c4b Author: Mike Adamson Date: Tue May 30 18:58:16 2023 +0100 Implement getInstance(TypeParser) for VectorType. Fix losing data at startup. commit 2bbe88ad88e2243f12dc8df789ea9bdf208ff74b Merge: dd0ffb64f3 a3b8661746 Author: Jonathan Ellis Date: Tue May 30 12:37:58 2023 -0500 merge ds-trunk commit dd0ffb64f3e54e2ce1af75da52fe3ece66a49f3f Author: Jonathan Ellis Date: Tue May 30 11:40:20 2023 -0500 reduce startup log noise at info level commit 1ef17101def1aacbb84f3434633a9b706291aa4b Author: Jonathan Ellis Date: Tue May 30 11:34:26 2023 -0500 add partialUpdateTest commit f816046a17809b06ad754a16c0f089c0d26909f9 Author: Jonathan Ellis Date: Mon May 29 12:57:12 2023 -0500 fix recall computation in test commit 1c1442c36baafdf4e1641a9c0288dbd520cf7c0e Author: Jonathan Ellis Date: Mon May 29 12:34:02 2023 -0500 also apply tolerance for inexact results to searchWithKey, but only for size > 10 commit 39df5112c6a551b53649ad98164f2bde93cbdc10 Merge: d598206329 8ca4bba861 Author: Jonathan Ellis Date: Mon May 29 12:27:24 2023 -0500 merge ds-trunk commit 8ca4bba8617e8d0fa4f48de1c479de66e3a77dd3 Author: Jonathan Ellis Date: Mon May 29 12:19:32 2023 -0500 rename MemtableIndex -> TrieMemtableIndex to make merge to vsearch easier commit d5982063292b6a91b156da32454abacd36fa6980 Merge: ae96e0f423 1488a5f0b9 Author: Jonathan Ellis Date: Mon May 29 09:03:24 2023 -0500 Merge branch 'vsearch' of github.com:datastax/cassandra into vsearch commit ae96e0f42393eab3547e2314f1acdb1ff766f2a6 Author: Jonathan Ellis Date: Mon May 29 09:02:12 2023 -0500 check for results within a percentage (I went with 5%) of the expected; the A in ANN means we shouldn't expect to find 100% of matches unless the graph is tiny commit 1488a5f0b9c65473bb6a69c9e18c045d931a1fe3 Author: Zhao Yang Date: Mon May 29 09:20:28 2023 +0800 Vector-19: fix PrimaryKey min/max prefix ByteComparable and add reversed lookup for row id (#627) * Vector-19: fix PrimaryKey min/max prefix ByteComparable and add reversed lookup for row id commit 14c41f5f913dc7f7f4cfc64b932bb6b9df475f19 Author: Jonathan Ellis Date: Wed May 17 17:35:16 2023 -0500 use incremental bytes used estimate from hnsw to avoid recomputing full ramBytesUsed on every call to add commit f8d2aaad6e96693f532d745f71e0eaabb1ddf934 Author: Jeremiah D Jordan Date: Fri May 26 12:02:54 2023 -0500 Update cqlsh for new syntax commit f0a882ae1171d41bc9c6ccf224ce3be80f7128e8 Author: Mike Adamson Date: Fri May 26 16:24:38 2023 +0100 Only allow VectorType to accept float commit a29712b1365cdfac9a709e35325e85f7fa6c61c8 Author: Mike Adamson Date: Fri May 26 16:17:51 2023 +0100 Fix max term size for vectors at 16k is SSTableIndexWriter commit 363d3f4e35c1a4df0f257557309bb042fd3f3ec7 Author: Mike Adamson Date: Fri May 26 15:23:02 2023 +0100 Merge CASSANDRA-18504 vector grammar (#628) - Now uses vector to describe vector - This commit does not bring in the whole 18504 patch only the essential grammar and type parts of the patch commit 18c4e35d4ce14cda7a3c03398e16edf368ad9e6a Merge: 16158a5b48 b0f71ee2c3 Author: Jonathan Ellis Date: Thu May 25 16:24:41 2023 -0500 Merge branch 'VECTOR-3' into vsearch commit b0f71ee2c371d1003cf241c3aedd7437385bcecb Author: Jonathan Ellis Date: Thu May 25 15:51:27 2023 -0500 cleanup commit 2d840eae74c079229729767731d3719d12ca1931 Merge: c01cacf300 197a3207b1 Author: Jonathan Ellis Date: Thu May 25 15:47:39 2023 -0500 Merge branch 'vsearch' (early part) into VECTOR-3 commit 16158a5b48665f57bde649e87a09d078298932f5 Author: Jonathan Ellis Date: Thu May 25 15:45:51 2023 -0500 optimize ramBytesUsed commit 197a3207b16ec97bae4924883247e2cd6a2923bd Author: Jonathan Ellis Date: Thu May 25 13:34:19 2023 -0500 update lucene commit a7bfcc7a6f31090c3c8443d299dabb7acd437ccd Author: Jonathan Ellis Date: Thu May 25 08:05:41 2023 -0500 optimize ConcurrentVectorValues.write commit 456fb08af0b922922ea0e4f24def0184cf4b32c6 Author: Jonathan Ellis Date: Wed May 24 21:25:52 2023 -0500 fix NPE better commit c01cacf30025d181a2d86c4632042449396333bc Author: Zhao Yang Date: Thu May 25 08:53:38 2023 +0800 Return negative ordinal if row id is not found; add failing test for null vector commit 009fadf2ed889bcc6748f500b7fa4b1d43e910df Author: Jonathan Ellis Date: Wed May 24 18:00:23 2023 -0500 search() errors out when an empty graph is passed to it, so special-case that commit 8c9972d7453fad46f55e6fe5e7538442235e7fd6 Author: Jonathan Ellis Date: Wed May 24 17:46:50 2023 -0500 use serializer dimensions instead of trying to cache it from the first vector added, because we might need it before vectors are added (if someone tries to search an empty graph) commit 8423541c4997462b65fe8f1231b0baf15dab3297 Author: Jonathan Ellis Date: Wed May 24 17:42:30 2023 -0500 fix NPEs when nulls are inserted commit d4510e0f1934f03f66d89b7c585b3d7aac17dda2 Author: Mike Adamson Date: Wed May 24 15:54:38 2023 +0100 Rebuild HNSW graph on flush since we don't know ahead of time where UCS will want to split the range boundaries commit 6b925531011798285a6b25d70961643092e2febe Author: Mike Adamson Date: Mon May 22 14:58:22 2023 +0100 disable index segment compaction by default commit 3480a80e3303c13b8c04f718f83f87489315e381 Author: Jonathan Ellis Date: Wed May 24 14:46:03 2023 -0500 move chatty logs to debug level commit 05c5758244f6f6d4fe30fbcefbb13932f68f8891 Author: Jonathan Ellis Date: Wed May 24 13:59:01 2023 -0500 r/m fixme obsoleted by #624 commit b09db4e0425c92491e3e9c73ebdbdb84508075fd Author: Jonathan Ellis Date: Wed May 24 13:56:46 2023 -0500 pre-size the intersection builder commit 6d62d8c2546cda1f487c14dcde1303e14a18c804 Author: Jonathan Ellis Date: Wed May 24 13:45:14 2023 -0500 fix nested boolean expressions commit b8301a86d37513b67545883548c5670131281dc1 Author: Zhao Yang Date: Wed May 24 14:19:41 2023 +0800 Vector-3: support partition/range restricted query commit 8ba56fad62d21da228e3a61ed7d5971441572cf9 Author: Jonathan Ellis Date: Wed May 24 10:35:16 2023 -0500 update lucene version hash commit a4768c583eb1a029f629e5ebda274ab2fcce3130 Author: Jeremiah D Jordan Date: Tue May 23 10:45:02 2023 -0500 Revert the revert of "Use published lucene fork jar."" This reverts commit 7a8ff12860a97d451045c9dd1978a3599a0f4679. commit 063a176caeec0f1e298d044490aa29ecdd468811 Author: Zhao Yang Date: Wed May 24 22:03:32 2023 +0800 Union the results from multiple segments per index (#624) - Union the results from multiple segments per index - Revert intersection behavior to pick 2 most selective indexes if there is no ANN - Fix SSTableRowIdPostingList to return END_OF_STREAM if next row is END_OF_STREAM - Skip TermTree for vector index - Fixes: - VectorMemtableIndexTest#randomQueryTest - SegmentMergerTest - SingleNodeQueryFailureTest#testFailedRangeIteratorOnMultiIndexesQuery - SelectiveIntersectionTest - QueryTimeoutTest commit 643f34c2b654b80573ba92381e386b4286f7fb63 Author: Jonathan Ellis Date: Wed May 24 08:25:35 2023 -0500 switch all the hnsw internals to use float[] so we don't need to keep both representations around commit 7a8ff12860a97d451045c9dd1978a3599a0f4679 Author: Jonathan Ellis Date: Wed May 24 08:06:28 2023 -0500 Revert "Use published lucene fork jar." This reverts commit 53c6c90920dd2a48fc3334a6cddb215d3f4ebfcd. commit 8731eefe4122a5af0b3723a3b07884a201616e7a Merge: e50d5e955b 53c6c90920 Author: Jonathan Ellis Date: Wed May 24 08:05:18 2023 -0500 Merge branch 'vsearch' of github.com:datastax/cassandra into vsearch commit e50d5e955bfde6357a5bb892b2ea70d46e8cc77d Author: Jonathan Ellis Date: Wed May 24 08:02:52 2023 -0500 cache float[] from ByteBuffer commit 53c6c90920dd2a48fc3334a6cddb215d3f4ebfcd Author: Jeremiah D Jordan Date: Tue May 23 18:29:00 2023 -0500 Use published lucene fork jar. commit f2567cd2fb2a683e7c0fee23945baac3ceca6ad4 Author: Jonathan Ellis Date: Tue May 23 14:56:25 2023 -0500 attempting to add back IN support but with limited success, even "basicOrTest" gets parsed as child nodes commit 9e1f018f5c0d5832446fc6a0338aaf95c8323303 Author: Jonathan Ellis Date: Tue May 23 14:21:23 2023 -0500 reduce log level of some of the chattiest locations commit efaf501fe3158da42dcd874b06014acb127f5cd9 Author: Jonathan Ellis Date: Tue May 23 13:55:40 2023 -0500 clean up RangeIntersection Builder overloads commit e7127d9d32392347f7010de6267521742f831217 Author: Jonathan Ellis Date: Tue May 23 13:46:27 2023 -0500 complete generification of RangeIterator in src/ (but tests are still incomplete) commit c07ded6420aa25b9dfa2b3c0785178b08d286b26 Author: Jonathan Ellis Date: Tue May 23 13:04:16 2023 -0500 TermIterator is more or less replaced by CheckpointingIterator commit 14d03ae99c69c2419dc0cca3291b4f5eaefbc054 Author: Jonathan Ellis Date: Tue May 23 14:45:18 2023 -0500 update lucene jar commit a15b964d56dcba285c6871d354755d7aa489af1d Author: Jonathan Ellis Date: Tue May 23 12:48:41 2023 -0500 disk and memory recalls should be the same (since disk graph should be exactly equivalent to in-memory) commit f03680bdf8581fd13a2d772e8265e3c8a713d247 Merge: 868a6b8f11 fd1293dbaf Author: Jonathan Ellis Date: Tue May 23 12:08:46 2023 -0500 Merge branch 'vsearch' of github.com:datastax/cassandra into vsearch commit 868a6b8f11a25c20b7e59a93d65cc7e2b021a432 Author: Jonathan Ellis Date: Tue May 23 12:04:11 2023 -0500 use brute force if we expect to perform fewer comparisons that way than with a graph search commit 2ed32bf53acaf8180130bef1aa5cebd9b6d32d13 Author: Jonathan Ellis Date: Tue May 23 11:59:11 2023 -0500 refactor AnnKeyRangeIterator -> ReorderingRangeIterator commit d28bad9b2df3fc84b117f9ed0a04314ecb9c0a02 Author: Jonathan Ellis Date: Tue May 23 11:55:15 2023 -0500 rename reorderOneComponent to limitToTopResults commit fd1293dbaf82f054593b3150249ac00110b286cf Author: Jeremiah D Jordan Date: Tue May 23 10:45:02 2023 -0500 Update python driver to one with supports the Vector type commit 6603f7601671357928948dfb0921b0c731bb96d3 Author: Jonathan Ellis Date: Tue May 23 11:06:12 2023 -0500 extract ReorderingPostingList commit 94b4ff20ccc3a51e966b4798f5364a2d9ed4b966 Author: Jonathan Ellis Date: Tue May 23 10:29:33 2023 -0500 deserialize doesn't modify buffer so duplicate is unnecessary commit 840053d212985a109cb5242c19e8a5876fdec7b3 Author: Jonathan Ellis Date: Tue May 23 10:26:57 2023 -0500 use LongHeap instead of generic PQ commit aac90dc3e0fdb745e465f044f2624228e705be10 Author: Jonathan Ellis Date: Tue May 23 09:47:22 2023 -0500 r/m unused defer flag from toPrimaryKeyIterator / toSSTableRowIdsIterator commit 1fd71c1cb7f2e785237b36c6e42842fb326d2380 Author: Jonathan Ellis Date: Tue May 23 09:26:03 2023 -0500 remove unused method marked "to remove" commit b96c5a9efccf3226acf2f2a08ca149d0acc0b21d Merge: 54f41e32d3 ba8aa07dd0 Author: Jonathan Ellis Date: Tue May 23 09:24:44 2023 -0500 Merge remote-tracking branch 'zhao/vsearch-row-id-iterator-for-reordering' into vsearch commit 54f41e32d3359941ddf70d34fc8e68a842b9091d Author: Jonathan Ellis Date: Tue May 23 08:14:10 2023 -0500 update lucene to fix NPE in ram usage commit ba8aa07dd09f52a72e5dffccb950a21ec37a1c11 Author: Zhao Yang Date: Tue May 23 16:51:36 2023 +0800 intersect on row-ids before fetching primary key for multiple non-ann indexes or single index commit 2a69547167f8dc1b40e8a26b71cdf43a45779280 Author: Zhao Yang Date: Tue May 23 09:44:00 2023 +0800 Use sstable-row-id iterator to re-order ann index - add searchSSTableRowIds to searcher commit 04b9a606d9a1aaeeeab3ec0981a5f1300cc2a423 Author: Zhao Yang Date: Tue May 23 00:30:12 2023 +0800 Make RangeIterator generic commit 4ff716a7411d3ad80009d4fb97bf53af51dd5eba Author: Jonathan Ellis Date: Mon May 22 14:57:34 2023 -0500 r/m obsolete FIXMEs commit 175ad0edce9cef3fb29f230fb64fb1913328d101 Author: Zhao Yang Date: Tue May 23 00:30:12 2023 +0800 fix vector distributed test: use 1 token and ignore multi-ann-index test (#622) commit 78adf77d9e6b7f20ece71fe665a5d1aec51f79b6 Author: Mike Adamson Date: Mon May 22 14:26:48 2023 +0100 Copy lucene snapshot to build folder in build-resolver.xml commit 5172315dcb51e05a44af224ddfe944bca29005d0 Author: Jonathan Ellis Date: Mon May 22 08:17:41 2023 -0500 declare lucene dependency to come from lib/ commit d4ee45b8675f79b4a1e06f97d0ce618ab08f030a Author: Jonathan Ellis Date: Sun May 21 05:40:30 2023 -0500 add caching and test code commit 5102d96e2fb3460f1e541cc1b481e224d112eac1 Merge: 6afcfabe88 87d39a7b25 Author: Jonathan Ellis Date: Sat May 20 11:15:14 2023 -0500 merge cep-vsearch commit 6afcfabe883c5627c96eda8eb488c51c06e17c91 Merge: a993dbe49a fa85a191c5 Author: Jonathan Ellis Date: Sat May 20 09:15:13 2023 -0500 Merge commit 'fa85a191' into vsearch commit a993dbe49a75dde4b9d5528e0816b81ae427a1dd Author: Jonathan Ellis Date: Fri May 19 22:00:31 2023 -0700 FIXME not sure why version is messed up post merge, this hacks around it commit aadf168e47a4271cc3faeb62ec7044cbfe193893 Merge: c217120935 5bc4d4b42e Author: Jonathan Ellis Date: Sat May 20 08:52:48 2023 -0500 merge 5bc4d4b42e commit c2171209350cc307f3b5fcc73a6ec09423d203df Author: Jonathan Ellis Date: Fri May 19 22:00:31 2023 -0700 FIXME not sure why version is messed up post merge, this hacks around it commit 2e44491d3db6a881324126490e6e1511d6082f39 Merge: 68f85f7009 0ff4566080 Author: Jonathan Ellis Date: Sat May 20 08:15:16 2023 -0500 merge 0ff4566080 commit 68f85f700995b274ec0d64654ed2692c7fc8bf61 Merge: c8843779be 2ae60e7411 Author: Jonathan Ellis Date: Sat May 20 07:50:16 2023 -0500 Merge commit '2ae60e74' into vsearch commit c8843779be9fe0c3edd047754bb08f633cff5646 Merge: 3cb57548bb 93616d080c Author: Jonathan Ellis Date: Sat May 20 07:43:18 2023 -0500 Merge commit '93616d080c539e880e37cd22fe5f27396a7a2594' into vsearch commit 87d39a7b252b3f3cc1f4aae0f149085f6bc83abc Author: Jonathan Ellis Date: Fri May 19 17:24:55 2023 -0700 fix intersection count, and add comments to KRI api commit 4fdc58fced9ca350333675cb19a42169bebeaccf Author: Jonathan Ellis Date: Fri May 19 15:27:34 2023 -0700 test recall commit 26fb1006430513a37f267a4dfb81ef473516ac7b Author: Jonathan Ellis Date: Fri May 19 14:19:08 2023 -0700 add tests commit 1cbf78bbf71fff2d597ddd199fd3a758313c7bf4 Author: Jonathan Ellis Date: Fri May 19 14:02:36 2023 -0700 comments commit 65e41490f041535f1a48a726032e28f6c8febcff Author: Jonathan Ellis Date: Fri May 19 13:57:04 2023 -0700 rm AnnResult commit 06b10a28c3f6f19c76b056526cdabb4c9aff1528 Author: Jonathan Ellis Date: Fri May 19 13:54:08 2023 -0700 cleanup commented-out code commit e33236ab8b9da7a025f4c59406c52ecbc3b5ae4f Author: Jonathan Ellis Date: Fri May 19 09:52:50 2023 -0700 convert suspicious one-off binary search in OnDiskHnswGraph to use DiskBinarySearch commit fbd30bce13e87c63b129db26c5c123ae9bf8f9ed Author: Jonathan Ellis Date: Fri May 19 09:31:16 2023 -0700 perform disk-based binary search for ordinals commit 77bb1b5cfcc743224bdb01d21563365caa184c21 Author: Jonathan Ellis Date: Fri May 19 08:22:36 2023 -0700 move OnDiskOrdinalsMap to its own class, and move write() code into it commit 8a1fd9f0a38f4c16b72d5bc7b2aeb3ee07bacc7a Author: Jonathan Ellis Date: Thu May 18 21:35:28 2023 -0700 add ordinal to row mapping commit db32d9c7b077133a213ac3a0da161f163c4310a1 Author: Jonathan Ellis Date: Wed May 17 20:43:49 2023 -0500 mostly-working reordering commit 62ba004d79aa7511f519ebb56fa8af2cb424e21e Author: Jonathan Ellis Date: Wed May 17 20:13:10 2023 -0500 turn QueryView inside out; perform intersections first for each sstable, then union the results commit d65039a6ccd8364e8eb8ef31933cdf8dca298437 Author: Jonathan Ellis Date: Wed May 17 18:37:07 2023 -0500 update MIM to give individual iterators back commit a190f1e917b93b6cabd15347dec55b1683807b1f Author: Jonathan Ellis Date: Wed May 17 18:59:13 2023 -0500 refactor CheckpointingIterator to just take an iterator to wrap, and index references to close if something goes wrong commit 307fe1c20b400e17048a356f73822ef51f66f341 Author: Jonathan Ellis Date: Wed May 17 18:47:20 2023 -0500 rename IndexSearchResultIterator -> CheckpointingIterator commit 690751b216afbe3e77ed0f89ce04d16e5c6e985d Merge: 4b6d0f2fb5 fa85a191c5 Author: Jonathan Ellis Date: Wed May 17 10:17:49 2023 -0500 Merge branch 'cep-7-sai' into cep-vsearch commit 4b6d0f2fb531f6832d2b24465b0982683e741dfe Author: Zhao Yang Date: Wed May 17 16:45:42 2023 +0800 Add vector top-k filter at replica side before returning to coordinator (#618) * added QueryPlan#postIndexQueryProcessor to filter top-k at replica side before sending response to coordinator commit 5bc4d4b42e77180df050bdbd9f5fd18fbb37332c Author: Jonathan Ellis Date: Tue May 16 07:38:45 2023 -0500 failing test was failing b/c test was wrong commit f37e40f5c5d99aeaafb33f41be0bd10bff6b5756 Merge: ae1bffa49b 1c41a095e1 Author: Jonathan Ellis Date: Mon May 15 21:24:47 2023 -0500 merge commit ae1bffa49b4c8757a3d9110ae065b8cb70d8e416 Merge: bbc4f72b8d 381e04aaa1 Author: Jonathan Ellis Date: Mon May 15 21:22:16 2023 -0500 Merge commit '381e04aaa132cf7a255efad3f792852d9ec1729d' into cep-vsearch commit 1c41a095e130bce4c7c215650f9ef3f817ae635e Author: Jonathan Ellis Date: Mon May 15 21:11:47 2023 -0500 fix computeNext to return endOfData when done commit b845395ad7e5f4670597cd171f28b98c941d03a7 Author: Jonathan Ellis Date: Mon May 15 21:11:15 2023 -0500 cleanup commit 58df11dad894aa5fb36bfbdaaf4c7dc1987fcc94 Author: Jonathan Ellis Date: Mon May 15 20:59:12 2023 -0500 failing tests commit ea5977620ebb07d220ad51800c9b90f470494079 Author: Jonathan Ellis Date: Mon May 15 20:58:52 2023 -0500 FIXME temporarily remove checkstyle commit 715790922379ad7b5dbdea5d3358f24542762889 Author: Jonathan Ellis Date: Mon May 15 20:49:10 2023 -0500 cleanup commit 19e47d363c856f81bda3323be02bfe18f60b51c0 Author: Jonathan Ellis Date: Mon May 15 17:55:52 2023 -0500 update NeighborSet usage commit 920abafce00c94bbc7d6b0c86ba78a2a9d3f9540 Author: Jonathan Ellis Date: Mon May 15 17:53:43 2023 -0500 cleanup commit d58db09004427506efe03e81364cadf44fc914c4 Author: Jonathan Ellis Date: Mon May 15 15:46:06 2023 -0500 pull in latest hnsw optimizations commit bbc4f72b8d7d99db003103103c90d78db4ee04d1 Author: Jonathan Ellis Date: Mon May 15 10:02:40 2023 -0500 use IndexOutputWriter in other components commit 381e04aaa132cf7a255efad3f792852d9ec1729d Author: Jonathan Ellis Date: Mon May 15 09:32:25 2023 -0500 switch other components to also use IndexOutputWriter commit 5951690f5472502bcb093179e8905212c1fcf815 Author: Mike Adamson Date: Mon May 15 14:48:16 2023 +0100 Fix CassandraHnswGraphWriter to use correct output writer (#616) commit bc5b956607fa77d53fcab9ad338776af69610feb Author: Zhao Yang Date: Mon May 15 17:35:42 2023 +0800 Distributed vector search (#613) * distributed vector search - query all replicas selected by consistency level at once with full request range - filter top-K results at coordinator in QueryPlan#postProcessor - skip short-read-protection, read repair and replica filtering protection because replica response will be top-k - fail ANN query without limit or limit exceeding MAX_TOP_K - make vector search max_top_k configurable and default to 1k commit 6d8c94535354f162f60099deecb0523f9f71bb99 Author: Jonathan Ellis Date: Sun May 14 10:01:08 2023 -0500 fix imports for checkstyle commit 68c20dc9ceeaf0c2169718aa4a1dce9c0f92f003 Author: Jonathan Ellis Date: Sat May 13 19:23:58 2023 -0500 add the index and ordinal mapping to the set of components that the system knows about commit ba7efaac142c8b3181881f86020f659837c90463 Author: Jonathan Ellis Date: Sat May 13 17:57:22 2023 -0500 cleanup commit 5ae06fe31840e5ba1899086f3b8eb8194f825db7 Author: Jonathan Ellis Date: Sat May 13 17:23:30 2023 -0500 update to latest lucene snapshot that addresses all known concurrency bugs commit f996edd63f66cf7b688f8e014b31f15450dd1d3d Author: Jonathan Ellis Date: Fri May 12 16:08:40 2023 -0500 fix for cqlsh by Bret McGuire commit fa85a191c5e0bd508da584412648308888769cb9 Author: Andrés de la Peña Date: Mon Apr 24 16:39:26 2023 +0100 Allow CQL queries on multiple indexes without ALLOW FILTERING patch by Andrés de la Peña; reviewed by Berenguer Blasi for CASSANDRA-18217 commit e2f3d2150ab2bc5e038f2661b8e2337d3b5cd4bf Author: Jonathan Ellis Date: Thu May 11 14:34:22 2023 -0500 imports commit 68e59bc715de6d1f69d69530310655a598a54200 Author: Mike Adamson Date: Thu May 11 17:20:52 2023 +0100 Use correct bind types for vector commit c57119254b7e065433804e36ed6ead71d53984ea Author: Jonathan Ellis Date: Wed May 10 23:06:19 2023 -0500 avoid calling neighborSet.size() in write path commit f91e9ea40fa4bfc07c0bb280b7c479d473956da3 Author: Jonathan Ellis Date: Wed May 10 23:05:11 2023 -0500 add more asserts. countNeighbors is failing commit 1bd6789304a60762f0ef497c2137dd96f806a262 Author: Jonathan Ellis Date: Wed May 10 22:40:01 2023 -0500 lock out updates to the graph while we're writing it to disk commit da7cb2984f96c06a5890ed2da21c42d946d7c754 Author: Jonathan Ellis Date: Wed May 10 22:06:58 2023 -0500 imports commit 3676a86b821c6c05d9c5df33f4e4fbc9eeb1f591 Author: Jonathan Ellis Date: Wed May 10 22:03:14 2023 -0500 compact by building graph in memory, like it did before commit fa1ebd013bc03def3d211932fe2a98d1bb4442d9 Author: Jonathan Ellis Date: Wed May 10 16:15:58 2023 -0500 rewrite save-to-disk without Lucene code, to support multiple rows having the same vector value bonus: we don't need to rebuild an index that we already have in memory commit 3b328324d7c3afcda34a95c4b9fcbb9fe017384c Author: Jonathan Ellis Date: Wed May 10 17:00:02 2023 -0500 upgrade to latest lucene snapshot commit 1f4e768e9ec3327a56d143fa17c86f88cfbc3674 Author: Jonathan Ellis Date: Wed May 10 16:17:50 2023 -0500 Revert use of ResumableSearcher, I'm giving up on maintaining that until Lucene is done requesting changes to the Concurrent code commit 14521723ea4b9c511f2efde12d7d7bd53360e0ea Author: Jonathan Ellis Date: Wed May 10 13:52:34 2023 -0500 add back writeCount.increment commit 7d0db43e22b93dc8990bf87c76db1488c67f7d15 Author: Jonathan Ellis Date: Wed May 10 13:44:37 2023 -0500 r/m vestigial keys commit c7455e912e2ea0113f034cf70ac174feaaab0c5b Author: Mick Semb Wever Date: Wed May 10 07:03:02 2023 +0200 use lib/lucene-core-9.7.0-SNAPSHOT.jar to build commit 7dffaf40f1bea86e63b014c876b1b3022ab734c2 Author: Jonathan Ellis Date: Wed May 10 11:46:50 2023 -0500 add CassandraHnswGraph that can map a vector to multiple keys commit 8ead56d1f933948de4b887b1c8200829ca3b946a Author: Jonathan Ellis Date: Wed May 10 11:07:58 2023 -0500 r/m unused bitset commit 7b18e3eff7bcaf2cb59d8cb4c592d03216c4af25 Author: Jonathan Ellis Date: Tue May 9 22:19:32 2023 -0500 Revert "add systemPath to lib jar, if this is part of the solution it's incomplete" This reverts commit c07b820395ed1ddf15b946c6dc7e74c1a222401f. commit c07b820395ed1ddf15b946c6dc7e74c1a222401f Author: Jonathan Ellis Date: Tue May 9 21:17:43 2023 -0500 add systemPath to lib jar, if this is part of the solution it's incomplete commit 6e0072df48c8b0377533afe919c1459cb367196b Author: Jonathan Ellis Date: Tue May 9 15:27:11 2023 -0500 make ByteBufferVectorValues threadsafe commit 0ffae2d4c707c14443a98ec352b67cd150bd233a Author: Jonathan Ellis Date: Tue May 9 13:58:40 2023 -0500 protect lists with readwritelock commit 7e13b8003eb7755e565660d1769ebd7dd99bb9c6 Author: Jonathan Ellis Date: Tue May 9 09:04:57 2023 -0500 validate that floats are finite commit 86fc9a3b9fc7079f8203df8ef083e61b1492d298 Author: Jonathan Ellis Date: Mon May 8 09:28:02 2023 -0500 fix imports commit 352d409acdfba5fc5211ef5292116b08d79edd50 Author: Jonathan Ellis Date: Fri May 5 16:24:38 2023 -0500 use ResumableSearcher commit 26a5693f78bdd32cc1978129b456bdc92552b24f Author: Jonathan Ellis Date: Fri May 5 12:32:25 2023 -0500 use lucene snapshot for concurrent hnsw build (not yet in search) commit 3a20fcb836725725db1da9f72ff2f958388e5a05 Author: Mike Adamson Date: Fri May 5 14:33:11 2023 +0100 Use SparseFixedBitSet and don't increase limit between batches commit 0ff4566080e0c31c89e1458f0784f12a65bfd0b7 Author: Mike Adamson Date: Thu May 4 14:58:28 2023 +0100 Batch search support for filtering queries commit 24e18dd147543b1489d7489bf1624d2a22db8f09 Author: Mike Adamson Date: Tue May 2 18:12:39 2023 +0100 Fix checkstyle failures commit 804d6deb255e857ef76f53e38a9c3562a1991d30 Author: Mike Adamson Date: Tue May 2 16:04:19 2023 +0100 Fix VectorType serialization commit 4fcf0441e6e00c22522281165857933337c4e18f Author: Mike Adamson Date: Tue May 2 15:01:42 2023 +0100 Checkstyle fixes commit 826787f39f2a175765606fe4e3223014ef8f95a2 Author: Mike Adamson Date: Tue May 2 14:55:27 2023 +0100 Fix serialization in Vectors commit 2ae60e74119037ad402afae62c338d32fb534da5 Author: Mike Adamson Date: Fri Apr 28 14:06:37 2023 +0100 Allow bind variable use with vector type (#611) commit 4240b4601eb7923f862b9f7d3f792f316fb2cfe8 Author: Mike Adamson Date: Tue May 2 14:50:00 2023 +0100 Additional changes for CEP commit 64271b95ba06aa540b5d50eec9183e9adf729191 Author: Mike Adamson Date: Thu Apr 27 13:21:18 2023 +0100 Grammar and options (#608) * Renamed grammar entities and added validation * Add index options for vectors * Fix some broken tests commit 8c6c8ff214ba1a12736481bd1886c5fb9d802ccf Author: Zhao Yang Date: Tue Apr 25 21:26:54 2023 +0800 fix TopDocsPostingList#advance and AnnTest (#607) * use dimension from VectorType * close KnnVectorsReader * skip per-column vector index file validation because lucene has different naming and manage its own validation * add more comments about lucene file names commit 9223bc26b3e0e9acc1b65462af4e4f91b113f328 Author: Jonathan Ellis Date: Mon Apr 24 14:43:40 2023 -0500 cleanup warnings commit e5332693fd301b4ab38bff63cbbbf2c19228c7ca Author: Mike Adamson Date: Mon Apr 24 14:43:52 2023 +0100 Allow dense float32[n] in the parser - This changes the AbstractType from DenseFloat32Type to VectorType that takes the number of dimensions declared in the CQL. Note: We should use IndexContext.isVector as the test for vector usage rather than instanceof on the type. commit bd76af052da7330474f40768f564936d775ba320 Author: Zhao Yang Date: Sun Apr 23 17:18:15 2023 +0800 Add vector flush and compaction Note: * vector files use different extension from SAI pattern * each vector segment doesn't support more than 2.1B vectors * both flush and compaction use VectorIndexWriter which doesn't support flushing into multiple segments commit 93616d080c539e880e37cd22fe5f27396a7a2594 Author: Mike Adamson Date: Fri Apr 21 16:18:57 2023 +0100 Fix RowFilter for ann during network messaging commit b6ea4f22ee0df2224f229875b8802d93d7a3e6f1 Author: Jonathan Ellis Date: Fri Apr 21 08:03:24 2023 -0500 remove topK from Expression commit 174f9ceb184040975929a4771ddb2762fad1be7f Author: Mike Adamson Date: Fri Apr 21 13:09:01 2023 +0100 Add AnnTest for end-to-end testing commit 2b72013d2ee0d8bab7dac4d7ac057d9e4bfe409c Author: Jonathan Ellis Date: Fri Apr 21 01:10:32 2023 -0500 r/m solved TODO commit 11e091ab63a261612e81770ea1f1c15068b8ae5a Author: Zhao Yang Date: Fri Apr 21 10:03:35 2023 +0800 Optimize key filter with KeyRangeFilteringBits commit ab2c443726a148bf55d07c933add5bc93fcc8535 Author: Jonathan Ellis Date: Thu Apr 20 22:42:01 2023 -0700 progress! index creation attempt is accepted (and results in bytes-read-exceeds-buffers-size error) commit 3e737a890117ee2a2030df3b1f5a51cd02fb5722 Author: Jonathan Ellis Date: Thu Apr 20 21:54:20 2023 -0700 rename to DenseFloat32Type for consistency w/ cql commit 99451100de515068e4b20ef07e63de23499495aa Author: Jonathan Ellis Date: Thu Apr 20 21:53:32 2023 -0700 leverage list literals for dense float32 literals commit 2d1a107a7388c5aebe3d99f9363774bd5499c545 Author: Jonathan Ellis Date: Thu Apr 20 20:25:28 2023 -0700 add ANN to parser and lexer commit 6b4c830952df496ff052bce1d5c414f671c2bfd1 Author: Zhao Yang Date: Fri Apr 21 10:03:35 2023 +0800 Filter keys inside HnswGraphSearcher if keyRange is not full ring. commit 61cadb27c65a444ea075297c6ab69f800d3a4262 Author: Jonathan Ellis Date: Thu Apr 20 13:54:53 2023 -0700 add VectorMemtableIndexTest commit 4af473690dce885907eec9c9cd0ab305f62068a4 Author: Jonathan Ellis Date: Thu Apr 20 11:48:24 2023 -0700 implement VectorMemtableIndex on Lucene's HnswGraphBuilder commit 4b139dc9ed20d67525a5cd32cc9b3d7991681a39 Author: Mike Adamson Date: Thu Apr 20 11:53:25 2023 +0100 Wire VectorMemtableIndex into indexing chain commit 14a6ac2a5c1cd245f5e96906ed26e845eff79f72 Author: Jonathan Ellis Date: Wed Apr 19 17:40:35 2023 -0700 r/m broken hnsw indexwriter, for now commit cb9001a003bd744572f9f208e3f68ef0468272ff Author: Jonathan Ellis Date: Wed Apr 19 17:35:46 2023 -0700 leave Items as ByteBuffer to avoid 2x memory usage commit 72da03df380a838871852fec2b1ec656d888c67d Author: Jonathan Ellis Date: Wed Apr 19 17:20:12 2023 -0700 adding hnswlib commit 2e1b198588a3866511fff06c482003914a00737c Author: Jonathan Ellis Date: Wed Apr 19 16:54:50 2023 -0700 MemtableIndex as interface with TrieMemtableIndex implementing it commit 1220cf6f40eaadce2ea03d83137a9b574946b264 Author: Jonathan Ellis Date: Wed Apr 19 10:58:11 2023 -0700 rewrite Memtable vectorvalues to just convert into a list of float[] up front commit 41e8efa7b6fdbf0e4fe57fb8091ac8144068cf78 Author: Jonathan Ellis Date: Tue Apr 18 20:26:50 2023 -0500 fumbling in the dark commit e4baedcb092a265ed8a545e522d9385ee4a9a681 Author: Jonathan Ellis Date: Tue Apr 18 12:05:45 2023 -0500 add Float32DenseVectorType commit 35711280b65f7a0c5029f1f7261cbe076850811f Author: Jonathan Ellis Date: Wed Apr 19 13:13:21 2023 -0700 stub commit 2c38afef81e3c3eeccb89664314f95cf1f924af1 Author: Jonathan Ellis Date: Wed Apr 19 10:58:11 2023 -0700 rewrite Memtable vectorvalues to just convert into a list of float[] up front commit 8687f4cd5fab209983aff4005b45e872649e9355 Author: Jonathan Ellis Date: Tue Apr 18 20:26:50 2023 -0500 fumbling in the dark commit 7c2afb9f69392816e1f9fb6701066714df73eb00 Author: Jonathan Ellis Date: Tue Apr 18 12:05:45 2023 -0500 add Float32DenseVectorType commit 28d440cbb10d80589ac3c2dfade05425c703238a Author: Mike Adamson Date: Tue Apr 18 10:59:25 2023 +0100 Change SAICodecUtils to use BE ints commit 39fbb3e7ef5172b7dce2aa7325f2f0e69f16defe Author: Jonathan Ellis Date: Mon Apr 17 23:15:29 2023 -0500 move oldlucene package to sai.disk commit 6dbfd1d7eae435944329d1e59f09978e2bbea690 Author: Jonathan Ellis Date: Sat Apr 15 13:40:40 2023 -0500 remove RAMIndexOutput and GrowableByteArrayDataOutput looked at removing ByteArrayIndexInput but it's a much bigger change commit f5e4df1615545a815ef00a825552a70335731545 Author: Jonathan Ellis Date: Mon Apr 17 15:04:37 2023 -0500 replace RAMOutputStream with ResettableByteBuffersIndexOutput commit 6bd00d1dbc2ade6b71aae4fcbe456713e0d2a251 Author: Mike Adamson Date: Tue May 2 10:55:34 2023 +0100 Remove DirectReaders commit 19c46c3423c8a7413ee8157778dfcf181d2e32a5 Author: Jonathan Ellis Date: Mon Apr 17 08:48:44 2023 -0500 special case bitsPerValue == 0 commit 07743459e33ff0fe9e3628a70e54f0ab3ed1f50a Author: Mike Adamson Date: Mon Apr 17 12:17:38 2023 +0100 Remove use of DirectReaders in favour of DirectReader commit 8ba1014c8515c54972f53c9833bd3e922cc2bffa Author: Jonathan Ellis Date: Fri Apr 14 10:21:12 2023 -0500 convert DirectReadersTest to SaiRandomizedTest commit d94cc91c483d478206a00a9deed97794fb54e5cc Author: Jonathan Ellis Date: Fri Apr 14 09:04:23 2023 -0500 fix LE int read in ByteSliceReader commit 133b3005a99a0aeb345752dfd79f1134b477a914 Author: Jonathan Ellis Date: Thu Apr 13 21:43:57 2023 -0500 convert DirectReaders to little-endian commit 29c29eed241562ecd0b72049f6acb6072c309d0a Author: Jonathan Ellis Date: Thu Apr 13 21:43:46 2023 -0500 add debug logging commit b63272ec247658e615185341eac52cf43d88c85c Author: Jonathan Ellis Date: Thu Apr 13 16:23:26 2023 -0500 switch our Lucene DataInput subclasses to little-endian commit 9d49945b382578b1756ff3ec0430e4510ff0d666 Author: Jonathan Ellis Date: Thu Apr 13 13:21:02 2023 -0500 add test-sai target commit a247f2dc8e4a42b3b2f9d8fbace297ac96765ae3 Author: Jonathan Ellis Date: Thu Apr 13 09:40:18 2023 -0500 upgrade to Lucene 9.5 (one FIXME in commented-out test code) commit 3cb57548bbcacf854ee38f19f3a1066d6290e349 Author: Mike Adamson Date: Fri Apr 28 14:06:37 2023 +0100 Allow bind variable use with vector type (#611) commit a31330ecfff1f9493603c3814326d2b25b193a69 Author: Mike Adamson Date: Thu Apr 27 13:21:18 2023 +0100 Grammar and options (#608) * Renamed grammar entities and added validation * Add index options for vectors * Fix some broken tests commit f1dd0515622cc3b4753c0a226524b88318af8a57 Author: Zhao Yang Date: Tue Apr 25 21:26:54 2023 +0800 fix TopDocsPostingList#advance and AnnTest (#607) * use dimension from VectorType * close KnnVectorsReader * skip per-column vector index file validation because lucene has different naming and manage its own validation * add more comments about lucene file names commit 80fe7e01150da8678161b58b6dde8a31f7ba6dc1 Author: Jonathan Ellis Date: Mon Apr 24 14:43:40 2023 -0500 cleanup warnings commit 10dd40716adcfcee5f66c4a23be5565fa7c78d2f Merge: 2cf2e6eab5 a3c0c8e7ac Author: Jonathan Ellis Date: Mon Apr 24 10:17:03 2023 -0500 Merge pull request #7 from mike-tr-adamson/grammar_changes Allow dense float32[n] in the parser commit a3c0c8e7acb029e36c0c61d6e1f2094b2731326a Author: Mike Adamson Date: Mon Apr 24 14:43:52 2023 +0100 Allow dense float32[n] in the parser - This changes the AbstractType from DenseFloat32Type to VectorType that takes the number of dimensions declared in the CQL. Note: We should use IndexContext.isVector as the test for vector usage rather than instanceof on the type. commit 2cf2e6eab52c29630735ef056d0f50bbdde65b74 Merge: 1c5c44933d 126035c92b Author: Jonathan Ellis Date: Sun Apr 23 18:23:42 2023 -0500 Merge pull request #6 from jasonstack/vsearch-zy Add vector flush and compaction commit 126035c92b82c50d24189ca5502d14365f9d55a4 Author: Zhao Yang Date: Sun Apr 23 17:18:15 2023 +0800 Add vector flush and compaction Note: * vector files use different extension from SAI pattern * each vector segment doesn't support more than 2.1B vectors * both flush and compaction use VectorIndexWriter which doesn't support flushing into multiple segments commit 1c5c44933d7b9472fef392313d11b6ac9ee5a7d3 Merge: 45b2a20949 3cabec315e Author: Jonathan Ellis Date: Fri Apr 21 08:37:01 2023 -0700 Merge pull request #5 from mike-tr-adamson/rowfilter Fix RowFilter for ann during network messaging commit 3cabec315ec5d7be8dcb1bd6f9722051bfedb2b2 Author: Mike Adamson Date: Fri Apr 21 16:18:57 2023 +0100 Fix RowFilter for ann during network messaging commit 45b2a20949fc5a61a8341e0ddacb0e27701fd142 Author: Jonathan Ellis Date: Fri Apr 21 08:03:24 2023 -0500 remove topK from Expression commit 6542e5d28b02f06f603429e522faeae9f7765755 Merge: bbe8275660 b4d8691541 Author: Jonathan Ellis Date: Fri Apr 21 05:57:55 2023 -0700 Merge pull request #4 from mike-tr-adamson/end-to-end Add AnnTest for end-to-end testing commit b4d8691541b30289030ea48a96f3baf0510afb1f Author: Mike Adamson Date: Fri Apr 21 13:09:01 2023 +0100 Add AnnTest for end-to-end testing commit bbe827566071c9a26da4be9974ef0825ba5e4d40 Merge: cf163f6900 d2fcfebac0 Author: Jonathan Ellis Date: Fri Apr 21 01:11:00 2023 -0500 Merge branch 'vsearch' of github.com:jbellis/cassandra into vsearch commit cf163f690067799bc44df8067cbdffe27bfe212c Author: Jonathan Ellis Date: Fri Apr 21 01:10:32 2023 -0500 r/m solved TODO commit d2fcfebac07d680b3afce72af16accd0b90989b9 Author: Zhao Yang Date: Fri Apr 21 10:03:35 2023 +0800 Optimize key filter with KeyRangeFilteringBits commit 0bf6d455d79bd88d1dc70ff0652b4384198d8f0d Merge: 9c79ef7356 cd2d96350c Author: Jonathan Ellis Date: Fri Apr 21 00:45:37 2023 -0500 Merge branch 'vsearch' of github.com:jbellis/cassandra into vsearch commit 9c79ef73565032dee0dc19752609aa0eff8bdfd5 Author: Jonathan Ellis Date: Thu Apr 20 22:42:01 2023 -0700 progress! index creation attempt is accepted (and results in bytes-read-exceeds-buffers-size error) commit 96ae50122bd101d2b39e3c9e96dcac356e334052 Author: Jonathan Ellis Date: Thu Apr 20 21:54:20 2023 -0700 rename to DenseFloat32Type for consistency w/ cql commit 9ad512998a736e8cc3c4232b4976cb5fce00dccf Author: Jonathan Ellis Date: Thu Apr 20 21:53:32 2023 -0700 leverage list literals for dense float32 literals commit 8139e7a2b275a09f375ad06a325cadb043e92f83 Author: Jonathan Ellis Date: Thu Apr 20 20:25:28 2023 -0700 add ANN to parser and lexer commit cd2d96350cb9ee6a8e9a85936763335d8a66600b Author: Zhao Yang Date: Fri Apr 21 10:03:35 2023 +0800 Filter keys inside HnswGraphSearcher if keyRange is not full ring. commit 29d01dfde5176b74cc643d8264263643b7b5133e Author: Jonathan Ellis Date: Thu Apr 20 13:54:53 2023 -0700 add VectorMemtableIndexTest commit 4f6b6236d2423b783a3699ed40585f906a1cefd9 Author: Jonathan Ellis Date: Thu Apr 20 11:48:24 2023 -0700 implement VectorMemtableIndex on Lucene's HnswGraphBuilder commit 47da08f7d9466e6a9a63011cf50e87e971efb2e2 Merge: 0987382acc 68bf99e146 Author: Jonathan Ellis Date: Thu Apr 20 11:07:23 2023 -0700 merging work from brief flirtation with hnswlib commit 68bf99e146ca05c529cc0f4d7c63dfe0f61775d8 Merge: 602a7d4d7c f6eb3a3ace Author: Jonathan Ellis Date: Thu Apr 20 08:25:55 2023 -0700 Merge pull request #3 from mike-tr-adamson/wire-index Wire VectorMemtableIndex into indexing chain commit f6eb3a3ace5fa1e6aabd90a1921028d9c77fc91a Author: Mike Adamson Date: Thu Apr 20 11:53:25 2023 +0100 Wire VectorMemtableIndex into indexing chain commit 602a7d4d7c14877db0c848e61e4a5e9e68397f8e Author: Jonathan Ellis Date: Wed Apr 19 17:40:35 2023 -0700 r/m broken hnsw indexwriter, for now commit a0a4f5c19a9a2c858a142e8fca61e801f14254c5 Author: Jonathan Ellis Date: Wed Apr 19 17:35:46 2023 -0700 leave Items as ByteBuffer to avoid 2x memory usage commit 3663f4b6e28cab4fe3943460a6668b46dabbf023 Author: Jonathan Ellis Date: Wed Apr 19 17:20:12 2023 -0700 adding hnswlib commit 8b2edcffe02346328757be1a62b141cdda73fac2 Author: Jonathan Ellis Date: Wed Apr 19 16:54:50 2023 -0700 MemtableIndex as interface with TrieMemtableIndex implementing it commit f663ce2be2ecd6ad55dbc9cff58e0a615c62b66a Author: Jonathan Ellis Date: Wed Apr 19 13:13:21 2023 -0700 stub commit a44baa6ad37423da9527e75b1ee54b6dc4b347b5 Author: Jonathan Ellis Date: Wed Apr 19 10:58:11 2023 -0700 rewrite Memtable vectorvalues to just convert into a list of float[] up front commit edea584b4d15f97895f1379d3756894b9441d240 Author: Jonathan Ellis Date: Tue Apr 18 20:26:50 2023 -0500 fumbling in the dark commit 1dac62f12f21363bdddb2ef6a9da917a12604e35 Author: Jonathan Ellis Date: Tue Apr 18 12:05:45 2023 -0500 add Float32DenseVectorType commit 0987382acc2f1efb41459bc6544b716aad47c864 Author: Jonathan Ellis Date: Wed Apr 19 13:13:21 2023 -0700 stub commit fe93d5ec351f68e2117897e98b3acec60c1114c1 Author: Jonathan Ellis Date: Wed Apr 19 10:58:11 2023 -0700 rewrite Memtable vectorvalues to just convert into a list of float[] up front commit 419264dd3bb17e967c981df15d6f481108ba6ace Author: Jonathan Ellis Date: Tue Apr 18 20:26:50 2023 -0500 fumbling in the dark commit 3b4e088126314a22bc315ac6437bbd4d759f0fed Author: Jonathan Ellis Date: Tue Apr 18 12:05:45 2023 -0500 add Float32DenseVectorType commit 41f4d6953058277e5c0ee40fcf6052099b0d58aa Author: Jonathan Ellis Date: Tue Apr 18 08:43:23 2023 -0500 mark Legacy tests (that use data files that were written in BE format") as ignored for now commit 61ff1f6119abdafb1b716ec84ef0ea5ffbde91b8 Merge: d41763b7f1 3523095f6b Author: Jonathan Ellis Date: Tue Apr 18 06:21:43 2023 -0500 Merge pull request #2 from mike-tr-adamson/lucene95 Change SAICodecUtils to use BE ints commit 3523095f6ba7baf63d92b26c7d6856b802a29a34 Author: Mike Adamson Date: Tue Apr 18 10:59:25 2023 +0100 Change SAICodecUtils to use BE ints commit d41763b7f120338266fcd5854680e05dd0bca5da Author: Jonathan Ellis Date: Mon Apr 17 23:15:29 2023 -0500 move oldlucene package to sai.disk commit 31cc2c89854dfef31663289a5acf898b23d4cefb Author: Jonathan Ellis Date: Sat Apr 15 13:40:40 2023 -0500 remove RAMIndexOutput and GrowableByteArrayDataOutput looked at removing ByteArrayIndexInput but it's a much bigger change commit 115191e6e49beaf19d6a808a2062d3b1fa30873e Author: Jonathan Ellis Date: Mon Apr 17 15:04:37 2023 -0500 replace RAMOutputStream with ResettableByteBuffersIndexOutput commit f2fce76e22df36d81c7c99f444ae911aed47371d Author: Jonathan Ellis Date: Mon Apr 17 08:48:44 2023 -0500 special case bitsPerValue == 0 commit 05ffa6b7b211825d2aa1bfc5e93d693a7db332ee Merge: 4ba646af3e 87c27130ea Author: Jonathan Ellis Date: Mon Apr 17 07:40:41 2023 -0500 Merge pull request #1 from mike-tr-adamson/lucene95 Remove use of DirectReaders in favour of DirectReader commit 87c27130eaa7941b0f741c85b55f10dbc0621508 Author: Mike Adamson Date: Mon Apr 17 12:17:38 2023 +0100 Remove use of DirectReaders in favour of DirectReader commit 4ba646af3e9c15e8c241a3cd2f6e7d351335d263 Author: Jonathan Ellis Date: Fri Apr 14 10:21:26 2023 -0500 debugging commit c88aa0ccaa0a2c5b79b8a2fd0772aa83a3fea689 Author: Jonathan Ellis Date: Fri Apr 14 10:21:12 2023 -0500 convert DirectReadersTest to SaiRandomizedTest commit 14aa3f0aacf7c94fade6fdea1788a16fbcdf005b Author: Jonathan Ellis Date: Fri Apr 14 09:04:23 2023 -0500 fix LE int read in ByteSliceReader commit 0dd02f06fa15edb918ce91a004def8695471e35a Author: Jonathan Ellis Date: Thu Apr 13 21:43:57 2023 -0500 convert DirectReaders to little-endian commit 7da55cdc55fd94df55f0e666337ecf50d3923304 Author: Jonathan Ellis Date: Thu Apr 13 21:43:46 2023 -0500 add debug logging commit 6d020edf05b61dfe1dce1b81aea13631e67e6768 Author: Jonathan Ellis Date: Thu Apr 13 16:23:26 2023 -0500 switch our Lucene DataInput subclasses to little-endian commit 287ad21c545bf77b212b65114fd9d3b01a84597c Author: Jonathan Ellis Date: Thu Apr 13 13:21:02 2023 -0500 add test-sai target commit 56e4dd29a07abe66f0ac66454fd357cc82804635 Author: Jonathan Ellis Date: Thu Apr 13 09:40:18 2023 -0500 upgrade to Lucene 9.5 (one FIXME in commented-out test code) commit b45f47d7097b2c9fbf6abe1c10ae60543b0068cb Author: Mike Adamson Date: Thu Apr 13 17:23:13 2023 +0100 Literal on-disk index and index write path (#9) This commit contains the following additions to SAI: - The index write path and index building based around StorageAttachedIndexBuilder and StorageAttachedIndexWriter - The on-disk index versioning using the SSTable Descriptor analog IndexDescriptor with Version and OnDiskFormat - The literal on-disk index using the LiteralIndexWriter patch by Mike Adamson; reviewed by Caleb Rackliffe and Andres de la Peña for CASSANDRA-18062 Co-authored-by: Mike Adamson Co-authored-by: Caleb Rackliffe Co-authored-by: Andres de la Peña Co-authored-by: Piotr Kołaczkowski Co-authored-by: Jason Rutherglen commit 7937878b36cb2d8292d5c48ef8cfc3f9ba2f2389 Author: Mike Adamson Date: Thu Jan 19 14:24:46 2023 +0000 In-memory index implementation with query path This includes the following elements of the Storage Attached Index: - Memtable-attached indexes backed by an in-memory trie structure for byte-comparable values - Query path for the in-memory index - Index status propagation - Randomized testing for Memtable-attached indexes patch my Mike Adamson; reviewed by Caleb Rackliffe and Andres de la Peña for CASSANDRA-18058 Co-authored-by: Mike Adamson Co-authored-by: Caleb Rackliffe commit 758b4dc9233fd415045a74d519c615625ba8ba52 Author: Zhao Yang Date: Tue May 19 17:08:36 2020 +0800 Add Index Group Interface for Storage Attached Index patch by Zhao Yang; reviewed by Caleb Rackliffe and Andres de la Peña for CASSANDRA-16092 Co-authored-by: Caleb Rackliffe Co-authored-by: Andres de la Peña commit 91f6a9aca8d3c22a03e68aa901a0b154d960ab07 Merge: 0ac60af949 bb36090fa0 Author: Ekaterina Dimitrova Date: Thu Apr 13 10:20:36 2023 -0400 Merge branch 'cassandra-4.1' into trunk commit bb36090fa0858456fb1ba80adf71fb3dc24a1aac Author: Ekaterina Dimitrova Date: Mon Apr 10 09:36:48 2023 -0400 Set correctly keepBriefBrief property patch by Ekaterina Dimitrova; reviewed by Andres de la Pena and Michael Semb Wever for CASSANDRA-18267 commit 0ac60af9495cbf413a1964db4cbf45c3c47bfb06 Author: Brandon Williams Date: Thu Apr 6 13:43:29 2023 -0500 Upgrade commons-io to 2.11.0 Patch by brandonwilliams; reviwed by adelapena for CASSANDRA-17364 commit 3155497486fa7217cd35428a469dd170f24f64fc Merge: ca6450bda4 5ec792a184 Author: Stefan Miklosovic Date: Tue Apr 11 09:45:13 2023 +0200 Merge branch 'cassandra-4.1' into trunk commit 5ec792a184ca386f3c2b95db22c87cc8fca0db63 Merge: edb34e2f62 e9f8f2efa2 Author: Stefan Miklosovic Date: Tue Apr 11 09:45:00 2023 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit e9f8f2efa2ba75f223f31ca6801aff3fe2964745 Author: Stefan Miklosovic Date: Tue Apr 11 09:31:56 2023 +0200 Prepare debian changelog for 4.0.9 commit ca6450bda4136f3c16992c807f510520580a60cb Author: Ekaterina Dimitrova Date: Mon Apr 10 12:02:49 2023 -0400 Rename internal state() method in AbstractFuture to not conflict with Java 19 changes patch by Sylwester Lachiewicz; reviewed by Ekaterina Dimitrova and Brandon Williams for CASSANDRA-18395 commit 99b618fec1015ccf4be6629ec9826c022b343263 Author: Ekaterina Dimitrova Date: Fri Apr 7 16:59:10 2023 -0400 Fix MessagePayloadTest running on JDK17 patch by Ekaterina Dimitrova; reviewed by Brandon Williams for CASSANDRA-18438 commit acd815898d1e5669e7d8ee2469808f6e8a14a7df Author: Yuriy Semchyshyn Date: Fri Apr 7 09:52:44 2023 -0700 Node Draining Should Abort All Current SSTables Imports patch by Yuriy Semchyshyn; reviewed by Dinesh Joshi, Yifan Cai for CASSANDRA-18373 commit e6e9a78d7eb21459264879e328d808aae377419f Merge: dc4e1fa942 edb34e2f62 Author: Brandon Williams Date: Fri Apr 7 13:44:27 2023 -0500 Merge branch 'cassandra-4.1' into trunk commit edb34e2f626be0c04d3055a66611cb0a2541bc91 Merge: 527f12e3ee 7d84ad9366 Author: Brandon Williams Date: Fri Apr 7 13:42:45 2023 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 7d84ad936622454afb01424d8a3c6db511654d60 Author: Brandon Williams Date: Fri Apr 7 13:42:18 2023 -0500 Revert "fix error in conflict edit" This reverts commit 21c3e9731ceb773ef6010dcfea9c8ede3204fda2. commit 5d315f869dcdda73f23f85e178a724391a51dccd Merge: 21c3e9731c 83dd275d5d Author: Brandon Williams Date: Fri Apr 7 13:41:56 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 21c3e9731ceb773ef6010dcfea9c8ede3204fda2 Author: Lorina Poland Date: Fri Apr 7 11:10:46 2023 -0700 fix error in conflict edit commit 9a3a37c5b3f4d92c10be3ce072c6b16251ad9eed Author: Lorina Poland Date: Tue Feb 14 10:53:29 2023 -0800 DOC - Added :stem: latexmath to Data Modeling page to fix display of equations patch by Lorina Poland; reviewed by Erick Ramirez for CASSANDRA-16906 commit 30639d8852825c14a98fb3a7ba46c89b559d56bf Author: Brandon Williams Date: Thu Mar 30 10:14:07 2023 -0500 Suppress CVE-2022-45688 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18389 commit 527f12e3eecb6a403eb0520324d606f2f4f3ee7e Author: Lorina Poland Date: Fri Apr 7 11:12:09 2023 -0700 fix error in conflict edit commit 68bab31b4d350b997fdbc181020a26730daabf67 Author: Lorina Poland Date: Tue Feb 14 10:53:29 2023 -0800 DOC - Added :stem: latexmath to Data Modeling page to fix display of equations patch by Lorina Poland; reviewed by Erick Ramirez for CASSANDRA-16906 commit 98dd5058d52165afb703b22cc2dc8a2b7262203b Author: Brandon Williams Date: Thu Mar 30 10:14:07 2023 -0500 Suppress CVE-2022-45688 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18389 commit dc4e1fa94222a271b51aaacf31000aa165a652bd Author: Lorina Poland Date: Fri Apr 7 11:13:22 2023 -0700 fix error in conflict edit commit 6b58ad0e5b72a60a430641301fb6a309d9c90fbc Author: Lorina Poland Date: Tue Feb 14 10:53:29 2023 -0800 DOC - Added :stem: latexmath to Data Modeling page to fix display of equations patch by Lorina Poland; reviewed by Erick Ramirez for CASSANDRA-16906 commit 99cd33e28faed1fb2cad0acf437e0b509c769e5e Author: Brandon Williams Date: Thu Mar 30 10:14:07 2023 -0500 Suppress CVE-2022-45688 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18389 commit 6a1a4d8459d2d6501db7c4c71e6f10530336836f Merge: c387462485 a8fa357bbc Author: Stefan Miklosovic Date: Thu Apr 6 23:39:41 2023 +0200 Merge branch 'cassandra-4.1' into trunk commit a8fa357bbc8331a6469c48cf465291d549c953d4 Merge: 3c2df9f1bd 03be6d7926 Author: Stefan Miklosovic Date: Thu Apr 6 23:29:22 2023 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 03be6d792679d8d4be560c9af4cba72e6b061f35 Author: Stefan Miklosovic Date: Thu Apr 6 10:53:29 2023 +0200 update zstd-jni library to version 1.5.5 patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18429 commit c3874624853f5a734a7751965baa9c28e0c83bfd Author: Ekaterina Dimitrova Date: Thu Apr 6 13:19:47 2023 -0400 ninja: CASSADRA-17199; to be reverted after CASSANDRA-18239 commit d1d733bee5ba85088a77d5c1430705dbb47ab68c Author: Ekaterina Dimitrova Date: Wed Apr 5 16:29:28 2023 -0400 Fix JDK17 startup issue after CASSANDRA-17199; temporary workaround, to be reverted after CASSANDRA-18239 Patch by Ekaterina Dimitrova; reviewed by Michael Semb Wever for CASSANDRA-18431 commit 83dd275d5d824abba4ced3cfbdada711f2e7bf63 Author: Lorina Poland Date: Tue Feb 14 10:53:29 2023 -0800 DOC - Added :stem: latexmath to Data Modeling page to fix display of equations patch by Lorina Poland; reviewed by Erick Ramirez for CASSANDRA-16906 commit 58f5301e78d7f7adbbe4b3c6d0528b1bcb4c27bc Author: Mick Semb Wever Date: Mon Mar 27 17:26:14 2023 +0200 ninja-fix stick affinity, and exclude it everywhere where it would pull in the imported boms patch by Mick Semb Wever; tested by David Capwell for CASSANDRA-18049 commit d3c55ed8f82c601d188fedbecb62a464171ea624 Author: Andrés de la Peña Date: Fri Mar 10 13:22:01 2023 +0000 Use snake case for the names of CQL native functions patch by Andrés de la Peña; reviewed by Berenguer Blasi for CASSANDRA-18037 commit 40f653396f8f6e027afaae7bd220cc5c6249498c Merge: e96c20e815 3c2df9f1bd Author: Stefan Miklosovic Date: Tue Apr 4 09:43:04 2023 +0200 Merge branch 'cassandra-4.1' into trunk commit 3c2df9f1bd50fe4244de0f1a442a24b31ef912f4 Merge: 085765a07c 99f62b7338 Author: Stefan Miklosovic Date: Tue Apr 4 09:42:33 2023 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 99f62b7338fc97a150e52e285f4eee3c636d6637 Author: Stefan Miklosovic Date: Tue Apr 4 09:29:07 2023 +0200 Prepare debian changelog for 4.0.9 commit e96c20e8150fd81716d203aa40301c09db0bc04a Merge: 6cf93f3192 085765a07c Author: Josh McKenzie Date: Mon Apr 3 16:18:33 2023 -0400 Merge branch 'cassandra-4.1' into trunk commit 085765a07c9eb87e05a700ba4395c9df672a60c1 Merge: f53b4149d0 f6fce7ab51 Author: Josh McKenzie Date: Mon Apr 3 16:11:01 2023 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit f6fce7ab51e41dfd8d9584ce4c8dd35e0ed61598 Author: Josh McKenzie Date: Wed Mar 15 11:53:07 2023 -0400 Improve debuggability and correctness of ref detection This patch is a backport of CASSANDRA-17205 and also raises logging level to 'error' from 'warn' in StrongLeakDetector. Patch by jmckenzie; reviewed by smiklosovic for CASSANDRA-18332 commit 6cf93f31920af1e17241e08a46acf467b83cfce2 Merge: a9911afe9a f53b4149d0 Author: Mick Semb Wever Date: Mon Apr 3 15:09:08 2023 +0200 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Fix the number of running trees per thread for LongBTreeTest hanging test commit f53b4149d04fceea8d9ae8c9437b761b020deaa5 Merge: fc36817033 b3256f5bf2 Author: Mick Semb Wever Date: Mon Apr 3 15:08:22 2023 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Fix the number of running trees per thread for LongBTreeTest hanging test commit b3256f5bf22c99a8f5410634dd41ca1643a34c1c Author: Maxim Muzafarov Date: Wed Mar 1 00:35:36 2023 +0100 Fix the number of running trees per thread for LongBTreeTest hanging test patch by Maxim Muzafarov; reviewed by Mick Semb Wever for CASSANDRA-18131 commit d41e69a6527fd42d0c17ac4f5a0ecb81d00837ca Author: Brandon Williams Date: Thu Mar 30 10:14:07 2023 -0500 Suppress CVE-2022-45688 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18389 commit a9911afe9a2787636cf9a39b564b1256e6285f86 Merge: 4effa4efbe fc36817033 Author: Brandon Williams Date: Mon Apr 3 06:08:52 2023 -0500 Merge branch 'cassandra-4.1' into trunk commit fc36817033d5e1f56d10fc478d1d97f750715766 Merge: 03806f95e4 ed904a72f4 Author: Brandon Williams Date: Mon Apr 3 06:07:57 2023 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ed904a72f41ed55acec9a26cff4f00a5fb1384c6 Merge: 853ae8c840 f9364e4f44 Author: Brandon Williams Date: Mon Apr 3 05:58:42 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit f9364e4f44b0d37905713a76d088218645ab72fb Author: Brandon Williams Date: Thu Mar 30 10:14:07 2023 -0500 Suppress CVE-2022-45688 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18389 commit 4effa4efbe456d2f5335ef4b6db39804eab21042 Author: Mick Semb Wever Date: Sun Apr 2 10:25:03 2023 +0200 ninja-fix CHANGES.txt remove merge sections In trunk there should be no "Merge from…" sections. Forward merged change should go under the previous major section's latest unreleased version (and nothing under the trunk version). When the first trunk release happens (typically at branch time) it is presumed (and understood) that all changes from previous releases as found in the same CHANGES.txt file are included. commit 9f078090e9e07d0917716e395e770b766a327fc7 Merge: e07b8dfe8d 03806f95e4 Author: Caleb Rackliffe Date: Fri Mar 31 11:59:19 2023 -0500 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Avoid loading the preferred IP for BulkLoader streaming commit 03806f95e427384107b3e943a73f1b2273b7f749 Merge: 14936d0bd3 853ae8c840 Author: Caleb Rackliffe Date: Fri Mar 31 11:58:59 2023 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Avoid loading the preferred IP for BulkLoader streaming commit 853ae8c84049be875921a40c9d5924724cc72792 Author: Caleb Rackliffe Date: Thu Mar 30 13:07:16 2023 -0500 Avoid loading the preferred IP for BulkLoader streaming patch by Caleb Rackliffe; reviewed by Jon Meredith for CASSANDRA-18370 Co-authored-by: Caleb Rackliffe Co-authored-by: Jon Meredith commit e07b8dfe8dc142356223e71e438c77ffe97140c1 Merge: e98258a132 f8dd1931ee Author: Jacek Lewandowski Date: Fri Mar 31 17:30:21 2023 +0200 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Save host id to system.local and flush immediately after startup commit 14936d0bd3716ed251e799a264f5ab16d51b893b Merge: 780f8b94ff 4b9c18235a Author: Jacek Lewandowski Date: Fri Mar 31 17:30:20 2023 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Save host id to system.local and flush immediately after startup commit f8dd1931eeacb290af9e246376468f3704891cc4 Merge: 780f8b94ff 4b9c18235a Author: Jacek Lewandowski Date: Fri Mar 31 17:30:20 2023 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Save host id to system.local and flush immediately after startup commit 4b9c18235ae10acc266d92ffe6c54936a789904a Merge: 5f61645778 ee586b53c2 Author: Jacek Lewandowski Date: Fri Mar 31 17:30:19 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Save host id to system.local and flush immediately after startup commit ee586b53c2cb7178c10841dc02f4023c64dda10e Merge: 955cf156c8 f96659c530 Author: Jacek Lewandowski Date: Fri Mar 31 17:30:18 2023 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Save host id to system.local and flush immediately after startup Remove asf.yaml from non-trunk branches Notify the corresponding JIRA issue as soon as the PR is raised commit f96659c5306e62666e21c371c2ded646dd51672b Author: Jacek Lewandowski Date: Tue Mar 14 14:29:39 2023 +0100 Save host id to system.local and flush immediately after startup patch by Adriano Bonacin and Jacek Lewandowski; reviewed by Stefan Miklosovic and Sam Tunnicliffe for CASSANDRA-18153 commit 20bf2273a0b5024a78b9564d2f23536cf18650a5 Author: Brandon Williams Date: Wed Mar 29 06:32:05 2023 -0500 Remove asf.yaml from non-trunk branches commit 24f5b64f2616daa3a3dabe34deb7fd6ebfc43429 Author: Maxim Muzafarov Date: Wed Mar 29 06:24:34 2023 -0500 Notify the corresponding JIRA issue as soon as the PR is raised Patch by Maxim Muzafarov; reviewed by brandonwilliams and smiklosovic for CASSANDRA-18372 commit e98258a132d6d27c8431e6971693a224953fdd89 Author: Maxim Muzafarov Date: Wed Mar 29 13:30:53 2023 +0200 Upgrade checkstyle to 10.8.1 for j11 and 8.45.1 for j8 Patch by Maxim Muzafarov; reviewed by brandonwilliams and smiklosovic for CASSANDRA-18262 commit 549bf38a073e9b467b1f95223ed0c05f2496c40a Merge: 297825773a 780f8b94ff Author: Brandon Williams Date: Wed Mar 29 06:32:26 2023 -0500 Merge branch 'cassandra-4.1' into trunk commit 780f8b94ffe32c385e7269b8e6ac13d47baa4a32 Merge: 5fa2f87741 5f61645778 Author: Brandon Williams Date: Wed Mar 29 06:32:20 2023 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 5f61645778ed8d0eb8c275ee6447f6169b2bfb31 Merge: 5b877b9d2b c4535a2e2a Author: Brandon Williams Date: Wed Mar 29 06:32:13 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit c4535a2e2ac80f36ce56af72f5e93c0dad792441 Merge: 11b84d2ef2 2f8d11335e Author: Brandon Williams Date: Wed Mar 29 06:32:09 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 2f8d11335ee2e997c803554e3cd58274ccd462c7 Author: Brandon Williams Date: Wed Mar 29 06:32:05 2023 -0500 Remove asf.yaml from non-trunk branches commit 297825773a3c72e7555c1dd7ac5f06487e77cfbc Merge: 6a5f84d0f4 5fa2f87741 Author: Brandon Williams Date: Wed Mar 29 06:26:54 2023 -0500 Merge branch 'cassandra-4.1' into trunk commit 5fa2f877417e138d1b62320f1a23c50be68dafa4 Merge: 6f401a8725 5b877b9d2b Author: Brandon Williams Date: Wed Mar 29 06:26:40 2023 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 5b877b9d2b973444b63553f924d3e864c07a97b9 Merge: 2974810607 11b84d2ef2 Author: Brandon Williams Date: Wed Mar 29 06:26:15 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 11b84d2ef23177b46c969b158056a184bcb6f44f Merge: 955cf156c8 f352f2b637 Author: Brandon Williams Date: Wed Mar 29 06:25:59 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit f352f2b6372cc879c63ff535c684cbaea9e0f152 Author: Maxim Muzafarov Date: Wed Mar 29 06:24:34 2023 -0500 Notify the corresponding JIRA issue as soon as the PR is raised Patch by Maxim Muzafarov; reviewed by brandonwilliams and smiklosovic for CASSANDRA-18372 commit 6a5f84d0f44f5562776a12a678c7d68168d2c7e7 Merge: eca75a42ef 6f401a8725 Author: Stefan Miklosovic Date: Tue Mar 28 19:02:50 2023 +0200 Merge branch 'cassandra-4.1' into trunk commit 6f401a872584f7bceb12e3fcb1927287da687e7d Author: Stefan Miklosovic Date: Mon Mar 27 09:30:00 2023 +0200 Return snapshots with dots in their name in nodetool listsnapshots patch by Stefan Miklosovic; reviewed by Paulo Motta for CASSANDRA-18371 commit eca75a42ef5a75744df68be8eae38513af1c66f6 Merge: 4444721b6d 4ac89e6451 Author: Stefan Miklosovic Date: Sun Mar 26 23:29:46 2023 +0200 Merge branch 'cassandra-4.1' into trunk commit 4ac89e6451b2ab0e62c04e61c60ba760f6c10223 Author: Stefan Miklosovic Date: Fri Mar 24 10:42:30 2023 +0100 Fix NPE when loading snapshots and data directory is one directory from root patch by Stefan Miklosovic; reviewed by Paulo Motta for CASSANDRA-18359 commit 4444721b6de555352bf0ac3ef7e36f94dc832f41 Author: Natnael Adere Date: Fri Mar 24 09:28:37 2023 -0700 Provide summary of failed SessionInfo's in StreamResultFuture patch by Natnael Adere; reviewed by David Capwell, Dinesh Joshi, Jon Meredith for CASSANDRA-17199 commit ffe669106b7092296a035c0525a57fcd72fa2a56 Merge: 2e2a49fcdc 7c2f97cd29 Author: Paulo Motta Date: Fri Mar 24 12:17:44 2023 -0400 Merge branch 'cassandra-4.1' into trunk commit 7c2f97cd29486196b50c65a093e92b0fcd9789d9 Author: Paulo Motta Date: Mon Mar 6 18:17:32 2023 -0500 Do not submit hints when hinted_handoff_enabled=false - Remove dead WriteCallbackInfo code Patch by Paulo Motta, Aleksey Yeschenko; Reviewed by Stefan Miklosovic, Claude Warren for CASSANDRA-18304 Co-authored-by: Aleksey Yeschenko commit 2e2a49fcdcf34ef2e8afcb3369f5a62cb168f18c Author: Andrés de la Peña Date: Wed Dec 7 17:28:24 2022 +0000 CEP-20: Dynamic Data Masking Allows to attach the native masking functions and UDFs to the definitions of table columns in the schema, as defined by CEP-20. The functions masking a column can be specified on CREATE TABLE queries, right at the end of the column definition. The mask of a column can also be changed or dropped with an ALTER TABLE query. Once a column is masked, SELECT queries will always return the masked value of the column. That masking is done on the coordinator, at the end of the query execution. Thus, masking won't affect any filters or ordering, which would be based on the clear values of the masked columns. Column masks are stored on the table system_schema.column_masks. A new UNMASK permission allows to see the clear data of columns with an attached mask. Also, a new SELECT_MASKED permission allows to run SELECT queries restricting the clear values of masked columns. Superusers have both permissions by default, whereas regular users don't have them. Dynamic data masking can be enabled/disabled with the config property dynamic_data_masking_enabled in cassandra.yaml. It is disabled by default. This is the combination of multiple tickets: * Add masking functions to column metadata (CASSANDRA-18068) * Add UNMASK permission (CASSANDRA-18069) * Add SELECT_MASKED permission (CASSANDRA-18070) * Add support for using UDFs as masking functions (CASSANDRA-18071) * Add feature flag for dynamic data masking (CASSANDRA-18316) patch by Andrés de la Peña; reviewed by Benjamin Lerer and Berenguer Blasi for CASSANDRA-17940 commit de1ff6d4d149f4983238ae1d38fd9b9c657bb282 Author: Ekaterina Dimitrova Date: Wed Mar 22 16:24:54 2023 -0400 Fix nodetool/sjk hh, Java17 patch by Ekaterina Dimitrova, reviewed by Brandon Williams for CASSANDRA-18242 commit b4d4cf716b9596120f565d0286cd7dfbbe0a84f1 Merge: 080ad4963c 70b95d8098 Author: Brandon Williams Date: Wed Mar 22 13:29:20 2023 -0500 Merge branch 'cassandra-4.1' into trunk commit 70b95d80985e9165528e0b22f32fec187098e0ff Author: Brandon Williams Date: Tue Mar 21 16:07:49 2023 -0500 Restore COPY ... TO STDOUT behavior in cqlsh Patch by brandonwilliams; reviewed by adelapena for CASSANDRA-18353 commit 080ad4963c62f402cf3e3342971719917d89ac0b Merge: 4138dd697a a6f8dd10cd Author: Brandon Williams Date: Wed Mar 22 10:20:08 2023 -0500 Merge branch 'cassandra-4.1' into trunk commit a6f8dd10cd8b283339c08b50ed6357b213b89dc4 Author: Brandon Williams Date: Wed Mar 22 06:24:05 2023 -0500 Remove six and Py2SaferScanner merge cruft Patch by brandonwilliams; reviewed by smiklosovic for CASSANDRA-18354 commit 4138dd697a1c6f3430b99f8b3feb4386d31f90ac Author: Aleksey Yeschenko Date: Wed Mar 22 14:49:55 2023 +0000 Fix trunk CHANGES.txt commit fc9efd6357fb4255174689b3c107b897254a7d2f Author: maxwellguo Date: Thu Mar 9 11:18:14 2023 +0800 Add virtual table system_views.snapshots patch by Maxwell Guo; reviewed by Stefan Miklosovic and Paulo Motta for CASSANDRA-18102 Co-authored-by: Stefan Miklosovic commit eee211fdbb517c1b0e0a8635e0813bda7cf655e5 Author: Mick Semb Wever Date: Fri Feb 10 09:16:14 2023 +0100 Update OpenHFT dependencies (chronicle-queue, chronicle-core, chronicle-bytes, chronicle-wire, chronicle-threads) Some transitive dependencies to these have also been added to raise their versions. asm-* transitive dependencies have been excluded. patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova for CASSANDRA-18049 commit 592cbeaab09e7e959dfaea591d4afe7c1613e3db Author: Stefan Miklosovic Date: Mon Mar 13 12:25:46 2023 +0100 Remove org.apache.cassandra.hadoop code patch by Stefan Miklosovic; reviewed by David Capwell and Brandon Williams for CASSANDRA-18323 commit 01363745456693d72bd20e2080000a36bf16e393 Merge: 11ea405952 3021c33cab Author: Brandon Williams Date: Tue Mar 21 07:00:40 2023 -0500 Merge branch 'cassandra-4.1' into trunk commit 3021c33cabc90f9f38fe8aba727a4d3ae4c451d8 Author: Brandon Williams Date: Tue Mar 21 07:00:29 2023 -0500 Increment version to 4.1.2 commit 11ea405952dea41bac3573eea5d54991dfd9a094 Merge: 0dea9c268a fb9699c231 Author: Jacek Lewandowski Date: Tue Mar 21 10:12:30 2023 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Fix RepairJob unnecessarily reporting cancellation error commit fb9699c231053ac6f4c3a1a4f4f7bbc0a949a407 Merge: d5b1483703 2974810607 Author: Jacek Lewandowski Date: Tue Mar 21 10:12:04 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Fix RepairJob unnecessarily reporting cancellation error commit 2974810607b3612fdd50a30d02221c927be9beec Merge: 5104008db9 955cf156c8 Author: Jacek Lewandowski Date: Tue Mar 21 10:09:31 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Fix RepairJob unnecessarily reporting cancellation error commit 955cf156c898cfadcc4225b34052f5f35ff8463a Merge: 58d06d34ab f6a04ca81f Author: Jacek Lewandowski Date: Tue Mar 21 10:06:31 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Fix RepairJob unnecessarily reporting cancellation error commit f6a04ca81f2d0d8973fb7a99569c118122c500a2 Author: Jacek Lewandowski Date: Sat Mar 18 08:28:40 2023 +0100 Fix RepairJob unnecessarily reporting cancellation error patch by Jacek Lewandowski; reviewed by Brandon Williams for CASSANDRA-17701 commit 0dea9c268a023b0bb78784c815dec23575f9c30f Author: Ekaterina Dimitrova Date: Wed Feb 15 18:21:32 2023 -0500 CircleCI config J17 + J11 patch by Ekaterina Dimitrova; reviewed by Michael Semb Wever and Brandon Williams for CASSANDRA-18247 commit 3eb605b4db0fa6b1ab67b85724a9cfbf00aae7de Author: Andrés de la Peña Date: Tue Mar 14 13:20:26 2023 +0000 Remove deprecated CQL functions dateOf and unixTimestampOf patch by Andrés de la Peña; reviewed by Berenguer Blasi and Stefan Miklosovic for CASSANDRA-18328 commit 58a3b12508f97e44d3812f6c97e5a969dc6b5a1b Author: Stefan Miklosovic Date: Mon Nov 14 15:07:03 2022 +0100 Remove DateTieredCompactionStrategy patch by Stefan Miklosovic; reviewed by Jacek Lewandowski for CASSANDRA-18043 commit 92c90cd4ef385cab0bd7acfbd9a10261110f85ee Merge: a76286795f d5b1483703 Author: David Capwell Date: Wed Mar 15 16:19:05 2023 -0700 Merge branch 'cassandra-4.1' into trunk commit d5b1483703b53c02fb0e616e58107afb814f9f81 Author: David Capwell Date: Tue Mar 14 10:36:38 2023 -0700 Incompatible file system thrown while running Simulator patch by David Capwell; reviewed by Brandon Williams, Caleb Rackliffe for CASSANDRA-18320 commit a76286795f4b79aaaada46d8d937e1d697c43144 Author: Stefan Miklosovic Date: Tue Mar 14 21:45:15 2023 +0100 Add system_views.max_sstable_size and system_views.max_sstable_duration tables patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18333 commit 4734dfc503e6b4307b63dd61f36da9f9c89d86f9 Author: Stefan Miklosovic Date: Mon Mar 13 12:51:34 2023 +0100 Extend implicit allow-filtering for virtual tables to clustering columns patch by Stefan Miklosovic; reviewed by Aleksey Yeschenko for CASSANDRA-18331 commit 034e009e93b2d2e63a469022f02f085f86bf9ba8 Merge: fb95112427 8d91b469af Author: Stefan Miklosovic Date: Wed Mar 15 09:34:00 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit 8d91b469afd3fcafef7ef85c10c8acc11703ba2d Author: Stefan Miklosovic Date: Wed Mar 15 09:09:07 2023 +0100 Prepare debian changelog for 4.1.1 commit fb95112427a8c64344563b6e499a6fb7bbf538d1 Author: Abe Ratnofsky Date: Tue Mar 14 15:57:14 2023 -0500 Upgrade maven-shade-plugin to 3.4.1 to fix shaded dtest JAR build patch by Abe Ratnofsky; reviewed by Caleb Rackliffe CASSANDRA-18136 commit 5c51f05dab0562c3b95b60224e4698e2a30d2ce7 Author: Brandon Williams Date: Fri Mar 10 09:47:56 2023 -0600 Remove JAVA8/JAVA11_HOME from circle configs Patch by brandonwilliams; reviewed by edimitrova and mck for CASSANDRA-18106 commit d1ba381d96cf1f864f1f9d0c59c5efa4cf660f49 Merge: 7ad746cc83 deb8abd2fd Author: Jacek Lewandowski Date: Tue Mar 14 15:19:22 2023 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Fix the ordering of sstables when running sstableupgrade tool commit deb8abd2fd086de5c4a36c6952ff819e19090e94 Merge: 8ba3888fcf 5104008db9 Author: Jacek Lewandowski Date: Tue Mar 14 15:17:50 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Fix the ordering of sstables when running sstableupgrade tool commit 5104008db947f7ad6e6b65b474281939dba92829 Merge: 3cfb222122 58d06d34ab Author: Jacek Lewandowski Date: Tue Mar 14 15:15:15 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Fix the ordering of sstables when running sstableupgrade tool commit 58d06d34ab949fadf1db2309b5be8e35352d9766 Merge: 4b3cb0a772 8df69c8016 Author: Jacek Lewandowski Date: Tue Mar 14 15:13:19 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Fix the ordering of sstables when running sstableupgrade tool commit 8df69c8016fb84e59e558e74508a0767ad3a7cc0 Author: Claude Warren Date: Tue Mar 14 09:24:25 2023 +0100 Fix the ordering of sstables when running sstableupgrade tool patch by Claude Warren; reviewed by Jacek Lewandowski and Stefan Miklosovic for CASSANDRA-18143 commit 7ad746cc839199ac9b8d7ba9a6f2b2f980ea3461 Author: Brandon Williams Date: Mon Mar 13 11:59:37 2023 -0500 Revert "Switch checkstyle to j11." This reverts commit 4e3447ae1727a0d01c9d81792acbdeeef94d86d2. commit 3eff9e755c46ca1e0011f2c91f8cb48e48cec025 Merge: 6eeb61d504 8ba3888fcf Author: Jacek Lewandowski Date: Mon Mar 13 09:57:48 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit 8ba3888fcf2f11bf426fe83768b42783567ec6d9 Merge: b7eaa2d209 3cfb222122 Author: Jacek Lewandowski Date: Mon Mar 13 09:54:01 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 3cfb2221223a7631f116108aec0a45646e775705 Author: Jaroslaw Grabowski Date: Thu Mar 9 12:35:48 2023 +0100 Fix BufferPool.memoryInUse counter The counter was incorrectly decremented by the size of the unused portion of the provided buffer. It is now decremented by the number of bytes actually returned to the pool (that may be different than "size"). The number is calculated as a difference between original and resulting buffer capacity. patch by Jaroslaw Grabowski , reviewed by Jacek Lewandowski for CASSANDRA-18311 commit 6eeb61d504510392a89879ad40d100505958645b Merge: 4e3447ae17 b7eaa2d209 Author: Stefan Miklosovic Date: Sun Mar 12 21:35:18 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit b7eaa2d209406f489776f47634257fbaa48c4dca Author: Stefan Miklosovic Date: Fri Mar 10 11:28:59 2023 +0100 Deprecate org.apache.cassandra.hadoop code patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-16984 commit 4e3447ae1727a0d01c9d81792acbdeeef94d86d2 Author: Brandon Williams Date: Fri Mar 10 10:41:10 2023 -0600 Switch checkstyle to j11. Patch by brandonwilliams; reviewed by edimitrova for CASSANDRA-18262 commit 6d62d2f4f2efb144824a433c0f8048c8a0eeed22 Merge: 91193e0fdf bfc7621c8c Author: Andrés de la Peña Date: Fri Mar 10 11:53:47 2023 +0000 Merge branch 'cassandra-4.1' into trunk commit bfc7621c8c4fa6fae87b535558c4a19cebe05a98 Merge: af82b941a8 2aedb58123 Author: Andrés de la Peña Date: Fri Mar 10 11:52:40 2023 +0000 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 2aedb58123e562ef6bdb7314033512d0128329ea Merge: a874e02eec 4b3cb0a772 Author: Andrés de la Peña Date: Fri Mar 10 11:51:25 2023 +0000 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 4b3cb0a772b7ab0de9dec19c148388d1aea729db Merge: 326b5aceb1 02048cffea Author: Andrés de la Peña Date: Fri Mar 10 11:50:04 2023 +0000 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 02048cffeae4267054541f96741b16b1d0d89a20 Author: Andrés de la Peña Date: Tue Feb 28 11:44:31 2023 +0000 Remove -l / -m / -h designation and have two options: free or paid tier circle config patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova and Josh McKenzie for CASSANDRA-18012 commit 91193e0fdf42ffdfcc351763855d356f3458c3f2 Merge: 414f4143e6 af82b941a8 Author: Mick Semb Wever Date: Thu Mar 9 20:55:49 2023 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Restore 4.1's doc update for virtual table system_views.clients commit af82b941a88f511605578d8eaa9b3d04634c34ac Author: Tibor Répási Date: Mon Mar 6 15:10:14 2023 +0100 Restore 4.1's doc update for virtual table system_views.clients patch by Tibor Répási; reviewed by Ekaterina Dimitrova, Michael Semb Wever for CASSANDRA-18303, CASSANDRA-17344 commit 414f4143e6128bcc129f731bbf98d84ac18da33d Author: Tibor Répási Date: Mon Mar 6 15:10:14 2023 +0100 Restore 4.1's doc update for virtual table system_views.clients patch by Tibor Répási; reviewed by Ekaterina Dimitrova, Michael Semb Wever for CASSANDRA-18303, CASSANDRA-17344 commit bace51ce8e24a8c6a97ac35309536e8b1c8132be Merge: f14fa978e4 0ea1d384d4 Author: Brandon Williams Date: Thu Mar 9 10:04:52 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit 0ea1d384d45765f8c8b1dbed8e7ce5595ca92412 Merge: 5f7175d59d a874e02eec Author: Brandon Williams Date: Thu Mar 9 10:03:42 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit a874e02eecb282287929db6dfe76f1694c87a217 Merge: 40f9ca60f1 326b5aceb1 Author: Brandon Williams Date: Thu Mar 9 10:02:33 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 326b5aceb188912f377aac1e36485e0054605851 Merge: 4b84a8d7ac be574fc34b Author: Brandon Williams Date: Thu Mar 9 10:01:06 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit be574fc34ba9834929f1618ab63dd74446cd2683 Author: Brandon Williams Date: Thu Mar 9 09:53:58 2023 -0600 Fix default file system error handler for disk_failure_policy die Patch by Runtian Liu; reviewed by brandonwilliams and smiklosovic for CASSANDRA-18294 commit f14fa978e42dd48fcf85f62a136ce15be9aa830a Author: Mick Semb Wever Date: Fri Feb 10 11:28:28 2023 +0100 Upgrade to Opcodes.ASM9 patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova for CASSANDRA-17971 commit 065c671e6500a06829712ab6e2504dc33ac76ca2 Merge: 4841794028 5f7175d59d Author: Jacek Lewandowski Date: Thu Mar 9 08:13:40 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit 5f7175d59da8b8d20e1081619eadad8b60867e86 Author: Jacek Lewandowski Date: Mon Mar 6 13:10:28 2023 +0100 Small fixes around Schema management - Denormalize distributedAndLocalKeyspaces to remove perf degradation - Set updated schema version once applied patch by Jacek Lewandowski, reviewed by Stefan Miklosovic for CASSANDRA-18291 commit 48417940280021d0012b7a7db6c9f823a98086e4 Author: Stefan Miklosovic Date: Fri Feb 24 14:56:50 2023 +0100 Add MaxSSTableSize and MaxSSTableDuration metrics and propagate them together with local read/write ratio to tablestats patch by Stefan Miklosovic; reviewed by Brandon Williams and Brad Schoening for CASSANDRA-18283 commit 180f0f0b5b47a48c1a7d88927223cc52871cc801 Author: Stefan Miklosovic Date: Tue Mar 7 21:31:18 2023 +0100 Add more logging around CompactionManager operations patch by Stefan Miklosovic; reviewed by Brandon Williams, Brad Schoening and Kan Maung for CASSANDRA-18268 Co-authored-by: Milan Krisko commit 0bed1967fdd1caeb2bbc38645c2a0cb7e728725c Merge: 7ca806c60a 6adcff8f30 Author: Jon Meredith Date: Tue Mar 7 10:41:05 2023 -0700 Merge branch 'cassandra-4.1' into trunk commit 6adcff8f30f445ddd7f0aeb60d72c5a15430aefe Merge: 426829cf57 40f9ca60f1 Author: Jon Meredith Date: Tue Mar 7 10:28:41 2023 -0700 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 40f9ca60f103783aa481bc9a91b92fd55b4ea625 Author: Benedict Elliott Smith Date: Wed Mar 1 19:08:20 2023 -0700 Improve memtable allocator accounting when updating AtomicBTreePartition patch by Benedict Elliott Smith; reviewed by Benjamin Lerer, Jon Meredith for CASSANDRA-18125 commit 7ca806c60a3e080d740fb163c639bb76a520f6ab Author: Mick Semb Wever Date: Sat Feb 25 11:25:35 2023 +0100 Change trunk from 4.2 to 5.0 – update MessagingService (remove the 4.1 version that was only a placeholder in test code bc serialisation test files were previously incorrectly serialising random UUIds for TimeUUIDs) – update upgrade jvm-dtests – remove older serialization binary files we are no longer testing against patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-17973 commit 35f8da66f9b05d18b4177f8d2e1b86c772ad2221 Author: Natnael Adere Date: Thu Mar 2 11:50:07 2023 -0800 Reduce memory allocations of calls to ByteBufer.duplicate() made in org.apache.cassandra.transport.CBUtil#writeValue patch by Natnael Adere; reviewed by Benedict Elliott Smith, David Capwell for CASSANDRA-18212 commit b7e1e44a909c3a1d11e9c387db680c74d31b879f Author: Jacek Lewandowski Date: Thu Mar 2 12:46:25 2023 +0100 SSTable format API Summary of the changes: Format, reader and writer --------------------------- There are a lot of refactorings around sstable related classes aiming to extract the most generic functionality to the top-level entities and push down implementation-specific stuff to the actual implementation. In Particular, the top-level, implementation agnostic classes/interfaces are SSTableFormat interface, SSTable, SSTableReader, SSTableWriter, IVerifier, and IScrubber. The rest of the codebase has been reviewed for explicit usages of big table format-specific usages of sstable classes and refactored. SSTable, SSTableReader, and SSTableWriter have their builders. Builders make a hierarchy that follows the same inheritance structure as readers and writers. There are also partial implementations that add support for some features and may or may not be used by the custom implementations. They include: - AbstractSSTableFormat - adds an implementation of some initialization methods - in practice, all of the format implementations should extend this class - SSTableReaderWithFilter - add support for Bloom filter to the reader - SortedTableWriter - generic implementation for a writer which writes partitions in the default order to the data file, supports Bloom filter and some index of partitions - IndexSummarySupport - interface implemented by the readers using index summaries - KeyCacheSupport - interface implemented by the readers using row key cache Descriptor --------------------------- Refactored the Descriptor class so that: - All paths are created from the base directory File rather than from a String - All the methods named *filename* producing full paths were made private; their current implementations are returning file names rather than paths (the naming was inconsistent) - The usages of the `filenameFor` method were refactored to use the `fileFor` method - The usages of the `fromFilename` method were refactored to use a `fromFileWithComponent(..., false).left` expression In essence, the Descriptor class is no longer working on String-based paths. Index summaries --------------------------- Removed the index summary from the generic SSTableReader class and created an interface IndexSummarySupport to be implemented by the readers that need it. Methods in related classes that refer back to the reader were refactored to support just readers of the SSTableReader & IndexSummarySupport type. Therefore, we will no longer need to assume that the generic SSTableReader has anything to do with an index summary. A new IndexSummaryComponent class encloses data fields from the index summary file (note that aside from the index summary itself, the file includes the first and last partition of the sstable). The class has been extracted to deal with those fields and have that logic in a single place. Filter --------------------------- Refactored IFilter and its serialization - in particular, added the `serialize` method to the IFilter interface and moved loading/saving logic to a separate utility class FilterComponent. Extracted the SSTableReaderWithFilter abstract reader extending the generic SSTableReader with filter support. Extracted bloom filter metrics into separate entities allowing to plug them in if the implementation uses a filter. Cache --------------------------- Refactored CacheService to support different key-cache values. CacheService now supports arbitrary IRowIndexEntry implementation as a key-cache value. A new version of the auto-saving cache was created ("g") because some information about the type of serialized row index entry needs to be known before it is deserialized (or skipped). Therefore, the SSTableFormat type ordinal number is stored, which is sufficient because the IRowIndexEntry serializer is specific to the sstable format type. Similarly to the IndexSummarySupport, a new KeyCacheSupport interface has to be implemented to mark the reader as supporting key-cache. It contains the default implementation of several methods the rest of the system relies on when the key-cache is supported. Other changes --------------------------- - Fixed disabling chunk cache - enable(boolean) method in ChunkCache does not make any sense - it makes a false impression it can disable chunk cache once enabled, while in fact, it only clears it. Added setFileCacheEnabled to DatabaseDescriptor - Made WrappingUnfilteredRowIterator an interface - DataInputStreamPlus extends InputStream - this makes it possible for input stream-based inheritors of DataInputPlus to extend DataInputStreamPlus. It simplifies coding because sometimes we want to get DataInputPlus implementation extending InputStream as an argument. - Table and keyspace metrics were made pluggable - in particular, added the ability for a certain format to register gauges that are specific only to that format and make no sense for others - Implemented mmapped region extension for compressed data - Refactored FileHandle so that it is no longer closable - Implemented WrappingRebufferer - Introduced the SSTable.Owner interface to make SSTable implementation not reference higher-level entities directly. SSTable accepts passing null as the owner when there is no owner (like sometimes in offline tools) or passing a mock when needed in tests. Individual commits --------------------------- [4a87cd36fe] Fix disabling chunk cache [c84c75ccf3] Made WrappingUnfilteredRowIterator an interface [253d2b828e] Add getType to SSTableFormat [3f169dcc20] Remove getIndexSerializer from SSTableFormat [05bae1833b] Pull down rowIndexEntrySerializer field [da675f2809] Moved RowIndexEntry [673f0c5c39] Reduce usages of RowIndexEntry [c72538be91] Refactor CacheService to support for different key cache values [54d33ee656] Minor refactoring of ColumnIndex [93862df967] Just moved AbstractSSTableIterator to o.a.c.io.sstable.format [9e4566a1de] Refactored AbstractSSTableIterator [a4e61e80bb] Extracted IScrubber and IVerifier interfaces [20f78c7419] Push down implementation of SSTableReader.firstKeyBeyond [f2c24e5774] Moved SSTableReader.getSampleIndexesForRanges to IndexSummary [b6c3a6c1ea] Moved SSTableReader.getKeySamples implementation to IndexSummary [c4b90ebb33] Refactor InstanceTidier so that it is more generic [918d5a9e74] Refactor dropping page cache [a52fb4d558] Refactor sstable metrics [f6d10f930f] NEW (fix up) - DataInputStreamPlus extends InputStream [8f6a56d972] Getting rid of index summary in SSTableReader [4a918bf725] Removed direct usages of primary index from SSTableReader [358fa32602] Refactor KeyIterator so that it is sstable format agnostic [14c09d89c2] Remove explicit usage of Components outside of format specific classes [feff14e137] Move clone methods implementation from SSTableReader to BigTableReader [64e9787b10] Move saveIndexSummary and saveBloomFilter to SSTableReaderBuilder [ae71fe6ed8] Moved indexSummary field to BigTableReader and made it private [df9fd8c4b9] Moved ifile field to BigTableReader and made it private [2be6ea9ecf] Moved static open methods for BigTableReader to the reader factory [bc0e55ac48] Minor refactoring around IFilter and its serialization [5b95704beb] Minor refactorings around IndexSummary [87812335e8] Extracted TOCComponent class to deal with TOC file [fdad092a6a] Extracted CompressionInfoComponent class [39b47e388d] Extracted StatsComponent as a helper for elements of SSTable metadata [cdb55bff47] Fix SSTable.getMinimalKey [b99c6d5805] Refactor FileHandle so that it is no longer closable [77b7f7ace5] Implement WrappingRebufferer [b6868914dd] Add progressPercentage to ProgressInfo [7fd4956e5b] Moved copy/rename/hardLink methods from SSTableWriter to SSTable [1ccc6bf148] Create generic SSTableBuilder and IOOptions [da58a81102] Refactor SSTableReaderBuilder [4501ddba1c] Refactor ColumnIndex [d4f9e1a64b] Extracted non-big-table-specific functionality from BigTableWriter to SortedTableWriter [379525d01e] Refactor BigTableZeroCopyWriter to SSTableZeroCopyWriter as it is not specific to big format [8ac37f83bc] Extract EmptySSTableScanner out from BigTableScanner [ee6673f1cf] Implement SSTableWriterBuilder [bb26629235] Refactor opening early / final [a327595015] Refactored SSTableWriter factory [16ffd7334b] Extract non-big-format-specific logic from scrubber and verifier [75e02db6af] Allow to specify the default SSTableFormat via system property [a7b9d0d628] Small fixes around streaming [407f977c36] Move guard collection size [0529e57d2f] Remove explicit references to big format [61509963ec] Unclassified minor changes [da28d1af3a] Replaced getCreationTimeFor(Component) with getDataCreationTime() [e99c834de6] !!! Reformatting [882b7baa5a] Rename SSTableReader.maybePresent and fix its redundant usages [b70c983bea] Implement mmapped region extension for compressed data [d7ff3970de] Introduce SSTable.Owner interface [e9feb9c462] Replaced getCreationTimeFor(Component) with getDataCreationTime() [ee8082fb07] Created SSTableFormat.deleteOrphanedComponents [e62950fd3d] Refactor metrics further [cefa5b3814] Extract key cache support into separate entity [dd55101ca1] Extracted SSTableReaderWithFilter [510b651824] Implement customizable component types [2be512d9fa] Pluggable SSTableFormat by making SSTableFormat.Type not an enum [670836b55d] Refactor CRC and digest validators [00c91103bc] Extract delete method to delete SSTables and purge row cache entries [0819dc9fc2] Extracted trySkipFileCacheBefore(key) to SSTableReader [732f841750] Added missing overrides in ForwardingSSTableReader [db623218fd] Update DatabaseDescriptorRefTest [c018c468e5] Cleanup [eafc836242] Add @SuppressWarnings("resource") where needed [3b7c911dd6] Documentation patch by Jacek Lewandowski, reviewed by Branimir Lambov for CASSANDRA-17056 Co-authored-by: @jacek-lewandowski Co-authored-by: @blambov commit 0a0e06847bf10aa88a3a30c239c507a64f949d74 Author: Kamalesh Palanisamy Date: Fri Mar 3 15:42:06 2023 -0700 org.apache.cassandra.db.compaction.CompactionStrategyManagerBoundaryReloadTest.testReload fails when running with TrieMemtables patch by Kamalesh Palanisamy; reviewed by David Capwell, Jon Meredith for CASSANDRA-18144 commit 4e17922b61d6e90151c5e165ddbf0731232e566a Author: David Capwell Date: Wed Mar 1 11:45:55 2023 -0800 Gossip stateMapOrdering does not have correct ordering when both EndpointState are in the bootstrapping set patch by David Capwell; reviewed by Marcus Eriksson for CASSANDRA-18292 commit c747f70c058aa94d6bcfe1f9132c410db6d2b65a Author: Marcus Eriksson Date: Tue Apr 19 12:15:58 2022 +0200 Snapshot only sstables containing mismatching ranges on preview repair mismatch patch by Marcus Eriksson, reviewed by Sam Tunnicliffe, Stefan Miklosovic for CASSANDRA-17561 Co-authored-by: Blake Eggleston commit d1e3c781b524308490bcb8af9f3c2f0d0bc2176f Author: Ekaterina Dimitrova Date: Mon Feb 13 21:17:57 2023 -0500 Add JDK17 startup scripts patch by Ekaterina Dimitrova; reviewed by Mick Semb Wever for CASSANDRA-18258 commit 98a055e51c8eb6b788d3d0fe94f3d2576ab14d39 Merge: 03996ecc6d 426829cf57 Author: Brandon Williams Date: Fri Feb 24 12:32:45 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit 426829cf57a6835d3b5192629bf4869fd9cec919 Merge: 474e778bcc ccede05bf5 Author: Brandon Williams Date: Fri Feb 24 12:32:31 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ccede05bf5fbf3d8e409960d6c93845f47ee6339 Merge: 1d532bce7b 4b84a8d7ac Author: Brandon Williams Date: Fri Feb 24 12:32:18 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 4b84a8d7ac5875bed35c81e1f0107c495452ba0b Merge: a5b05cea5e 5374f56a0a Author: Brandon Williams Date: Fri Feb 24 12:29:37 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 5374f56a0a8d95f30a1398368b86192bad935446 Author: Brandon Williams Date: Fri Feb 24 12:28:59 2023 -0600 Fix typo in suppressions commit 03996ecc6dcaa865f2a625f30912fcd61103a085 Merge: ece0c4a44c 474e778bcc Author: Brandon Williams Date: Wed Feb 22 10:25:39 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit 474e778bcc2ed11a6a8acbe5a790b26b3d7585b8 Merge: 9860c1e9d9 1d532bce7b Author: Brandon Williams Date: Wed Feb 22 10:25:24 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 1d532bce7bb4cee8073a15a5b7572fc4d9ad8bcd Merge: 3a4fff38b5 a5b05cea5e Author: Brandon Williams Date: Wed Feb 22 10:25:08 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit a5b05cea5e2843d7ada11c03de823b8c07fb5455 Merge: 788b4eca2d eb36a86a72 Author: Brandon Williams Date: Wed Feb 22 10:24:22 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit eb36a86a72c0692327545c19f8eed23a6d165246 Author: Brandon Williams Date: Fri Feb 17 07:28:01 2023 -0600 Removing trailing year from copyright. Patch by brandonwilliams; reviewed by mck for CASSANDRA-18261 commit ece0c4a44c9209dc64240f3f33ca7a57d7548daa Author: Jacek Lewandowski Date: Tue Feb 21 11:28:00 2023 +0100 ninjafix - add missing "resource" suppression (blame CASSANDRA-18134) commit 24ebd24c79175f7426f4c489dc5a006e75f09dfb Author: Jacek Lewandowski Date: Wed Jan 4 16:22:52 2023 +0100 More accurate skipping of sstables in read path This patch improves the following things: 1. SSTable metadata will store a covered slice instead of min/max clusterings. The difference is that for slices there is available the type of a bound rather than just a clustering. In particular it will provide the information whether the lower and upper bound of an sstable is opened or closed. 2. SSTable metadata will store a flag whether the SSTable contains any partition level deletions or not 3. The above two changes required to introduce a new major format for SSTables - oa 4. Single partition read command makes use of the above changes. In particular an sstable can be skipped when it does not intersect with the column filter, does not have partition level deletions and does not have statics; In case there are partition level deletions, but the other conditions are satisfied, only the partition header needs to be accessed (tests attached) 5. Skipping SSTables assuming those three conditions are satisfied has been implemented also for partition range queries (tests attached). Also added minor separate statistics to record the number of accessed sstables in partition reads because now not all of them need to be accessed. 6. Artificial lower bound marker is now an object on its own and is not implemented as a special case of range tombstone bound. 7. Extended the lower bound optimization usage due the 1 and 2 8. Do not initialize iterator just to get a cached partition and associated columns index. The purpose of using lower bound optimization was to avoid opening an iterator of an sstable if possible. 9. Add key range to stats metadata [f369595b1c] Add fields to sstable version and placeholders in stats serializer [f5c3f772e2] Add hasKeyRange and hasLegacyMinMax [3cde51f4e1] Add partition level deletion presence marker to sstable stats [67b2ee2152] Extract AbstractTypeSerializer [c77b475d6c] Refactor slices intersection checking [ceb5af3a38] Store min and max clustering as a slice in stats metadata as and improved min/max [d1f8973929] Implement MetadataCollectorBench [335369da84] Apply partition level deletion presence marker optimizations to single partition read command [2497a009b9] Lower bound optimization - add slices and isReverseOrder fields to UnfilteredRowIteratorWithLowerBound [e32ee31177] Lower bound optimization - Replace usage of RangeTombstoneMarker as a lower bound with ArtificialBoundMarker [e213e712c4] Lower bound optimization - improve usage of lower bound optimization [c4f93006b1] Apply read path improvements to partition range queries [5fa462266c] Add key range to StatsMetadata [79a7339ed4] Use key range from stats if possible [266ed2749b] Added new sstables for LegacySSTableTest patch by Jacek Lewandowski; reviewed by Branimir Lambov and C. Scott Andreas for CASSANDRA-18134 Co-authored-by: Branimir Lambov Co-authored-by: Sylvain Lebresne Co-authored-by: Jacek Lewandowski Co-authored-by: Jakub Zytka commit 4a555f47ee943ce9fd70862cc8127d707e3507a2 Author: Jyothsna Konisa Date: Fri Feb 17 13:50:06 2023 -0800 Fix flaky testOutboundConnectionsAreRejectedWhenAuthFails patch by Jyothsna Konisa; reviewed by Jon Meredith, Yifan Cai for CASSANDRA-17708 commit 9a08062f1bf12b745b755f72a290af359d8bfeee Merge: 8d54835d5b 9860c1e9d9 Author: Stefan Miklosovic Date: Fri Feb 17 16:38:44 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit 9860c1e9d9fb45342fa674782ecd135cf6875943 Author: Stefan Miklosovic Date: Wed Feb 15 22:33:21 2023 +0100 Fix copying of JAR of a trigger to temporary file While testing the fix for trigger, we fixed the corresponding trigger example. While on it, we detected that ssl-factory example was not working either so it is fixed in this commit as well. patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18264 commit 8d54835d5b84f8afe1e97be65f8cc1897b0a9c5f Merge: d7352209b2 cfe9641fbe Author: Stefan Miklosovic Date: Fri Feb 17 15:44:17 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit cfe9641fbec0dc62c9a0f4f156c702e2cfa6ad4e Author: Stefan Miklosovic Date: Fri Feb 17 15:42:03 2023 +0100 Fix possible NoSuchFileException when removing a snapshot patch by Stefan Miklosovic; reviewed by Jacek Lewandowski for CASSANDRA-18211 Co-authored-by: Jacek Lewandowski commit d7352209b214559cea944b5df564b653a3379e3a Author: Mick Semb Wever Date: Thu Feb 16 11:06:31 2023 +0100 ninja-fix: compile and use JdkProperties patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova for CASSANDRA-18179 commit 2ea47b1b740916a627b30d2e8e2f910026c9e3bd Merge: 6ed6b520fd b728a0011b Author: Jon Meredith Date: Thu Feb 16 15:09:42 2023 -0700 Merge branch 'cassandra-4.1' into trunk commit b728a0011bce9d1881128144d6027f7ca2caaa55 Merge: a49efd8b44 3a4fff38b5 Author: Jon Meredith Date: Thu Feb 16 15:07:55 2023 -0700 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 3a4fff38b5d89650d6ac96df22d850a0935d7730 Author: Jon Meredith Date: Thu Feb 16 15:05:37 2023 -0700 Ninja fix CASSANDRA-18242 - was not generating IDEA workspace for J11 correctly commit 6ed6b520fd5cd02e5ab7aa160c01ce12fc79b335 Merge: 63993e10df a49efd8b44 Author: Stefan Miklosovic Date: Wed Feb 15 13:39:31 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit a49efd8b443521b2540f705a6229a343f8ac57fb Merge: 62aab9c9df fcc16f5f2b Author: Stefan Miklosovic Date: Wed Feb 15 13:38:18 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit fcc16f5f2bf80cdbb299cd81e12ea0bbb35cc0af Author: Stefan Miklosovic Date: Wed Feb 15 00:01:10 2023 +0100 Update zstd-jni to version 1.5.4-1 patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18259 commit 63993e10dfcfb6770846c754aff4e01c489ed1bc Merge: 5e409f0bd1 62aab9c9df Author: Stefan Miklosovic Date: Wed Feb 15 10:41:26 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit 62aab9c9df4b73bd1d8ba6f342fea62211e476ab Merge: 5c4c7b69aa 7b079503eb Author: Stefan Miklosovic Date: Wed Feb 15 10:40:35 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 7b079503eba72aeb0025f0830c868efaa6c4f84c Author: Stefan Miklosovic Date: Wed Feb 15 10:12:15 2023 +0100 Increment version to 4.0.9 commit 5e409f0bd1ab108545e617a97d367bdea46b4c86 Author: Derek Chen-Becker Date: Fri Feb 3 17:46:32 2023 -0700 Remove invalid handshake test assertion Patch by Derek Chen-Becker; reviewed by brandonwilliams and smiklosovic for CASSANDRA-17979 commit e107b855718efc76f19dc9c438fb032c423f4ed0 Merge: 86604f5038 5c4c7b69aa Author: Jon Meredith Date: Mon Feb 13 17:49:39 2023 -0700 Merge branch 'cassandra-4.1' into trunk commit 5c4c7b69aa7578544d97d453ed3502dbdf1d67c0 Merge: f345ee8a4e 4e0aad3319 Author: Jon Meredith Date: Mon Feb 13 17:43:24 2023 -0700 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 4e0aad3319f90eaf7a5e5aed1fd3c5bb7a949fbd Author: Jon Meredith Date: Fri Feb 10 10:05:42 2023 -0700 Split and order IDEA workspace template VM_PARAMETERS patch by Jon Meredith; reviewed by David Capwell for CASSANDRA-18242 commit 86604f50388d7e2cec68196fcf5f82a334e68ae2 Author: Mick Semb Wever Date: Sat Jan 21 12:46:56 2023 +0100 Prepare for JDK17 experimental support Removes the use of CASSANDRA_USE_JDK11 and introduces the build.xml properties `java.default` and `java.supported`. patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova for CASSANDRA-18179 commit a0ec0ae303443b1d1cbfe845d94fccfb38123f8c Author: Ekaterina Dimitrova Date: Thu Feb 9 15:00:05 2023 -0500 Remove scripted UDFs, remove related tests or rewrite some tests to create java UDFs instead, where it makes sense patch by Ekaterina Dimitrova; reviewed by Michael Semb Wever for CASSANDRA-18252 commit ff6ad4e48a04e5f78215da287fcf037ecb556016 Merge: a8465376f7 f345ee8a4e Author: Ekaterina Dimitrova Date: Sat Feb 11 15:31:43 2023 -0500 Merge branch 'cassandra-4.1' into trunk commit a8465376f7f6414ff373ffacad7e3e7425032496 Author: Brandon Williams Date: Sat Feb 11 13:59:28 2023 -0600 ninja: Add jna cleaner thread to BulkLoaderTest for CASSANDRA-18050 commit f345ee8a4e7c8c2dc495b6282805d15acf35b737 Merge: 3e0d38dab2 cb4f90542d Author: Ekaterina Dimitrova Date: Sat Feb 11 14:37:00 2023 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit cb4f90542d005c0cacc7cbc464e5edb2fd748b60 Merge: 32c56df067 788b4eca2d Author: Ekaterina Dimitrova Date: Sat Feb 11 14:34:36 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 788b4eca2d0b5f3d508daf9d1b08426f075b8fca Merge: 74b0b2086f 476d31bce2 Author: Ekaterina Dimitrova Date: Sat Feb 11 14:32:37 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 476d31bce2597cf71cdb7dd446d5cac28ee69a82 Author: Ekaterina Dimitrova Date: Mon Oct 31 18:10:05 2022 -0400 Add Dtest-large and cqlshlib tests to CircleCI; Fix num tokens in CircleCI tests to match the default value; Fix CircleCI Python tests split bug; Fix CircleCI long unit tests patch by Ekaterina Dimitrova; reviewed by Andres de la Pena for CASSANDRA-18001 commit a8b1fb6ae6e934f4ba0a07deb03b6257c7dcb383 Author: Brandon Williams Date: Mon Jan 23 11:31:21 2023 -0600 Upgrade JNA to 5.13.0 Patch by brandonwilliams; reviewed by mck for CASSANDRA-18050 commit 452ad8ce709b69b3201819143f397b4f1857f201 Author: Stefan Miklosovic Date: Thu Feb 9 14:18:50 2023 +0100 Make virtual tables decide if they implicitly enable ALLOW FILTERING patch by Stefan Miklosovic; reviewed by Aleksey Yeschenko for CASSANDRA-18238 commit 5f8270eb8670b7d2f68c8b1a1d791a8b2cb849a0 Merge: a68c0983b4 3e0d38dab2 Author: Stefan Miklosovic Date: Tue Feb 7 23:16:32 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit 3e0d38dab200b0ceb1e6866bcc53b889796b213f Merge: bbe936a236 32c56df067 Author: Stefan Miklosovic Date: Tue Feb 7 23:11:47 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 32c56df067b72da8593c1ddaaf143fe8668459dd Author: Stefan Miklosovic Date: Tue Feb 7 22:35:13 2023 +0100 Prepare debian changelog for 4.0.8 commit a68c0983b400a305c9cf2d71de597f06106d566a Merge: 993095739d bbe936a236 Author: Stefan Miklosovic Date: Tue Feb 7 15:55:04 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit bbe936a2361bdbc29695c0c3c3669a1ff29f1879 Merge: 6a23fae4b8 b8494a8275 Author: Stefan Miklosovic Date: Tue Feb 7 15:48:23 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit b8494a82751ae2d31ab0f36412f7eacdcc1c3774 Author: Stefan Miklosovic Date: Tue Feb 7 10:21:16 2023 +0100 Log warning message on aggregation queries without key or on multiple keys patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18219 commit 993095739d57d5a1c9e6fefcc4fae80e03729755 Merge: 2d323cb565 6a23fae4b8 Author: Stefan Miklosovic Date: Tue Feb 7 14:39:32 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit 6a23fae4b82213f936457f2e42d13cc2eaf15a2d Merge: 209ba28b4d b74c86404a Author: Stefan Miklosovic Date: Tue Feb 7 14:37:16 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit b74c86404a0d131677286206562bb4bfa3e8e1a9 Author: nvharikrishna Date: Sat Feb 4 00:02:08 2023 +0530 Fix the output of FQL dump tool to properly separate entries patch by N V Harikrishna; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-18215 commit 2d323cb56572e867b13b6d102a61aaff8bd66c86 Author: Jordan West Date: Wed Dec 21 12:10:42 2022 -0800 Add row,tombstone,and sstable count to profileload Patch by Jordan West; Reviewed by David Capwell for CASSANDRA-18022 commit a07c15d72d575845624746b828067ed8d2dc3d9d Merge: 1e685219da 209ba28b4d Author: Brandon Williams Date: Thu Feb 2 11:46:40 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit 209ba28b4dc5abb32d053d2d9d2605076163d32c Merge: bede10f36c 0c58fbb8dd Author: Brandon Williams Date: Thu Feb 2 11:46:31 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 0c58fbb8dd25beab4b4a81650be1ed0ec888ff66 Author: yongj Date: Wed Feb 1 07:52:35 2023 +0000 Add cache type information for maximum memory usage warning message Patch by Yong Jiang; reviewed by brandonwilliams and smiklosovic for CASSANDRA-18184 commit 1e685219da4177fc5b5d6025618398532d2a0124 Author: Doug Rohrer Date: Mon Jan 23 13:53:09 2023 -0500 Add unit tests for per-row TTL and Timestamp usage in CQLSSTableWriter Patch by Doug Rohrer; reviewed by brandonwilliams and dcapwell for CASSANDRA-18187 CQLSSTableWriter supports per-row setting of both timestamp and TTL values, but it’s not tested or documented today. Add tests to cover setting both TTL and Timestamp values for rows using the CQLSSTableWriter. commit 67b483b5c884dfe0afa5ed221903387d212fe918 Merge: 763793e89d bede10f36c Author: Stefan Miklosovic Date: Wed Feb 1 15:29:48 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit bede10f36c68b6674fa7beb1b31ddc7cd847f131 Merge: dcc19cf865 ba434b8231 Author: Stefan Miklosovic Date: Wed Feb 1 14:47:57 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ba434b8231360347984c9a9906e5469767bfbcec Author: nvharikrishna Date: Wed Dec 14 00:25:54 2022 +0530 Fix NPE in fqltool dump on null value If fullquerylog is enabled and a prepared query executed with null value binded, then fqltool dump is running into NPE. Checking value for null before wrapping it. patch by N V Harikrishna; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-18113 commit 763793e89d0b29a8dc7fd33ef5e0e85b38d7ea9d Merge: bcaa1dca0b dcc19cf865 Author: Jacek Lewandowski Date: Wed Feb 1 09:48:05 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit dcc19cf865d4765160fabdc8852c32a7fe08f67a Merge: a479989020 9bf128aaa3 Author: Jacek Lewandowski Date: Wed Feb 1 09:44:58 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 9bf128aaa33c21427c826dab82414e3772d2ba24 Author: Jacek Lewandowski Date: Wed Jan 25 09:01:16 2023 +0100 Improve unit tests performance - Don't flush schema on every schema update in unit tests - Use unix command to delete test data - Shorten teardown - Stable processor count presented by JMX on Jenkins, CircleCI and local Patch by , reviewed by and for CASSANDRA-17427 commit bcaa1dca0b8bb47402b9e4a5d8857c1e4da4a2e0 Author: Brandon Williams Date: Tue Jan 31 06:32:08 2023 -0600 Fix typo in cassandra.yaml CASSANDRA-18209 commit 6c90950489a42a38a6ddfa45f79c84a110dffbe7 Author: Ekaterina Dimitrova Date: Thu Jan 19 20:30:43 2023 -0500 Fix tests post JDK-8210522 (rewrite reflection of "modifiers" field) patch by Ekaterina Dimitrova; reviewed by Michael Semb Wever for CASSANDRA-18181 commit 9ec90d0e69af661be4bf34dc83090d011d9329a0 Merge: 15d47f8e3f a479989020 Author: Brandon Williams Date: Mon Jan 30 15:49:40 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit a4799890208f442bc1ed134167e691cf1edfceab Merge: f18ffe1b97 ef2066fe60 Author: Brandon Williams Date: Mon Jan 30 15:49:30 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ef2066fe6084e424ae7dd57f669d06b86a3924a1 Merge: f8ad33ed4e 74b0b2086f Author: Brandon Williams Date: Mon Jan 30 15:49:20 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 74b0b2086f3a7e838acde21c8020f10f0a47955a Author: Brandon Williams Date: Mon Jan 30 07:46:54 2023 -0600 remove walrus operator in gen-nodetool-docs.py Patch by brandonwilliams; reviewed by smiklosovic for CASSANDRA-18207 commit 15d47f8e3f8b17e26ffe54cdb16b8a1d3435ebfa Merge: 1f58f48e6e f18ffe1b97 Author: Brandon Williams Date: Mon Jan 30 07:31:00 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit f18ffe1b97615b19b76a41a19d8a616015f5c0fe Merge: 561e3dc757 f8ad33ed4e Author: Brandon Williams Date: Mon Jan 30 07:30:54 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit f8ad33ed4e363418a768551fb34464b111820453 Merge: 0d0c5695df 48d4214f6c Author: Brandon Williams Date: Mon Jan 30 07:30:32 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 48d4214f6cac5eaf9f9d922c63ddcd0b57082edf Merge: 89266aadde fa18ea1852 Author: Brandon Williams Date: Mon Jan 30 07:29:48 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 89266aadde0eccacc5551ae2dbd32d5720235d4c Author: Brandon Williams Date: Mon Jan 23 07:07:14 2023 -0600 Suppress CVE-2022-41915 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18147 commit fa18ea185233d97b9f2771b1a3a88f46dc1610d4 Author: Brandon Williams Date: Mon Jan 23 07:07:14 2023 -0600 Suppress CVE-2022-41915 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18147 commit 1f58f48e6e87254a8fced6da6bd5f47ca1041484 Merge: 9b740da875 561e3dc757 Author: Brandon Williams Date: Mon Jan 30 07:26:44 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit 561e3dc757a3291f96169cddcaf3332e0ca783fc Merge: 5dbeee258c 0d0c5695df Author: Brandon Williams Date: Mon Jan 30 07:26:36 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 0d0c5695df574888a02bb457160aac9adc47c814 Merge: 0b7e3a8ee7 5ac75323cd Author: Brandon Williams Date: Mon Jan 30 07:26:16 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 5ac75323cd6cfef3e39a2fc476b174e9a70eff56 Merge: 8076f37c06 7ad693581b Author: Brandon Williams Date: Mon Jan 30 07:24:41 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 7ad693581bb87e75330c38daf266255dd78cbf6d Author: Brandon Williams Date: Mon Jan 23 07:07:14 2023 -0600 Suppress CVE-2022-41915 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18147 commit 9b740da8756bd1762c8cc54df83091cc52952ab5 Merge: 61a83b3cef 5dbeee258c Author: Brandon Williams Date: Mon Jan 30 07:05:49 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit 5dbeee258c761c3cdc16008cbfcfff5d3063a822 Merge: b81d39306c 0b7e3a8ee7 Author: Brandon Williams Date: Mon Jan 30 07:05:42 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 0b7e3a8ee7f9359eaf63208e12f32b19e6874e74 Author: Maciej Sokol Date: Fri Jan 27 10:04:43 2023 -0600 Connect to listen address when own broadcast address is requested Patch by Maciej Sokol; reviewed by brandonwilliams and smiklosovic for CASSANDRA-18200 commit 61a83b3cefb513db589972281920a17d1a71deec Merge: 4971483b7f b81d39306c Author: Stefan Miklosovic Date: Sat Jan 28 17:40:49 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit b81d39306c3c2002ea0f94a73e73199d94d572ae Merge: 1c71e57f46 2a8e1972fd Author: Stefan Miklosovic Date: Sat Jan 28 17:02:24 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 2a8e1972fd5444336ea4285fa38fbcabe873dc60 Merge: c2e9e0d841 8076f37c06 Author: Stefan Miklosovic Date: Sat Jan 28 16:40:41 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 8076f37c06c3f11dfb571b9c9169090c6fa5ebf5 Merge: 03c86cfcb0 4c8e9097bd Author: Stefan Miklosovic Date: Sat Jan 28 16:21:39 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 4c8e9097bd616a1ae4c611a542cbdea20ba871dc Author: Stefan Miklosovic Date: Sat Jan 28 16:18:51 2023 +0100 Introduce check for names of test classes patch by Stefan Miklosovic; reviewed by Berenguer Blasi and David Capwell for CASSANDRA-17943 commit 4971483b7f64369facf237f2d482250ba87d7587 Merge: 4892331eb4 1c71e57f46 Author: David Capwell Date: Fri Jan 27 12:35:18 2023 -0800 Merge branch 'cassandra-4.1' into trunk commit 1c71e57f46f704228d41fb367e31cd11f0602134 Author: David Capwell Date: Thu Jan 26 09:13:24 2023 -0800 PaxosPrepare may add instances to the Electorate that are not in gossip patch by David Capwell; reviewed by Blake Eggleston for CASSANDRA-18194 commit 4892331eb49f72d2e18432a2af56538d11c7c5d1 Author: Caleb Rackliffe Date: Wed Apr 27 15:08:11 2022 -0500 Coordinator level metrics for read response and mutation row and column counts patch by Caleb Rackliffe; reviewed by Berenguer Blasi for CASSANDRA-18155 commit 93e0ae9af62fb1bd0c4f120205d2e84809cf4e09 Author: Andrés de la Peña Date: Fri Aug 19 16:33:25 2022 +0100 Add CQL functions for dynamic data masking patch by Andrés de la Peña; reviewed by Benjamin Lerer for CASSANDRA-17941 commit f27790c96912ac9a83f052d8e6d0bfcdfe60ca0e Author: Nikita Eshkeev Date: Thu Jan 26 13:28:16 2023 +0100 Improve and clean up documentation and fix typos This patch includes all the changes from the PRs that introduce small changes related to typos and similar in the documentation. The changes are accumulated from the following PRs: - https://github.com/apache/cassandra/pull/206 - https://github.com/apache/cassandra/pull/359 - https://github.com/apache/cassandra/pull/366 - https://github.com/apache/cassandra/pull/390 - https://github.com/apache/cassandra/pull/450 - https://github.com/apache/cassandra/pull/567 - https://github.com/apache/cassandra/pull/615 - https://github.com/apache/cassandra/pull/618 - https://github.com/apache/cassandra/pull/746 - https://github.com/apache/cassandra/pull/984 - https://github.com/apache/cassandra/pull/1052 - https://github.com/apache/cassandra/pull/1088 - https://github.com/apache/cassandra/pull/1274 - https://github.com/apache/cassandra/pull/1378 - https://github.com/apache/cassandra/pull/1404 - https://github.com/apache/cassandra/pull/1504 - https://github.com/apache/cassandra/pull/1540 - https://github.com/apache/cassandra/pull/1544 - https://github.com/apache/cassandra/pull/1673 - https://github.com/apache/cassandra/pull/1697 - https://github.com/apache/cassandra/pull/1722 - https://github.com/apache/cassandra/pull/1815 - https://github.com/apache/cassandra/pull/1830 - https://github.com/apache/cassandra/pull/1863 - https://github.com/apache/cassandra/pull/1865 - https://github.com/apache/cassandra/pull/1879 - https://github.com/apache/cassandra/pull/2062 patch by Nikita Eshkeev, reviewed by Stefan Miklosovic, Lorina Poland, Michael Semb Wever for CASSANDRA-18185 Co-authored-by: kalmant Co-authored-by: Dmitry Co-authored-by: Tibor Répási Co-authored-by: Tzach Livyatan Co-authored-by: Jérôme BAROTIN Co-authored-by: Giorgio Giuffrè Co-authored-by: Siddhartha Tiwari <201851127@iiitvadodara.ac.in> Co-authored-by: Angelo Polo Co-authored-by: Tjeu Kayim <15987676+TjeuKayim@users.noreply.github.com> Co-authored-by: 陳傑夫 Co-authored-by: Bhouse99 Co-authored-by: Matthew Hardwick Co-authored-by: Paul Wouters Co-authored-by: Romain Hardouin Co-authored-by: Guilherme Poleto Co-authored-by: 陳傑夫 Co-authored-by: etc-crontab Co-authored-by: Prashant Bhuruk Co-authored-by: Jingchuan Zhu <56401528+codingswag998@users.noreply.github.com> Co-authored-by: Ryan Stewart Co-authored-by: utkarsh-agrawal-jm <107914361+utkarsh-agrawal-jm@users.noreply.github.com> Co-authored-by: Ben Dalling Co-authored-by: Terry L. Blessing Co-authored-by: gruzilkin Co-authored-by: Kevin Co-authored-by: yziadeh <121903189+yziadeh@users.noreply.github.com> Co-authored-by: Lorina Poland Co-authored-by: Stefan Miklosovic commit 93cc75ccdfd6747aa2e5b2344496a99c62e39d8a Merge: f4eef07ef3 60f5efe453 Author: Brandon Williams Date: Wed Jan 25 15:17:00 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit 60f5efe4536e43ea8dd00cee2ee332e17534a505 Merge: 057edbe92e c2e9e0d841 Author: Brandon Williams Date: Wed Jan 25 15:16:46 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit c2e9e0d8416866d6600444773f11f592711a5fe9 Merge: 299ead7534 03c86cfcb0 Author: Brandon Williams Date: Wed Jan 25 15:16:32 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 03c86cfcb0dd31a9c904e2561e3f8a1f26357a44 Author: Brandon Williams Date: Wed Jan 25 08:55:13 2023 -0600 Add concurrency to adoc generation in gen-nodetool-docs.py Patch by brandonwiliams; reviewed by mck for CASSANDRA-18197 commit f4eef07ef30f75b332fc043b97a7ccaad024fdde Merge: eee052dad3 057edbe92e Author: Brandon Williams Date: Tue Jan 24 12:27:02 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit 057edbe92e508006c84e610b2e7043d9488eebb6 Merge: 53156c24e1 299ead7534 Author: Brandon Williams Date: Tue Jan 24 12:26:51 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 299ead7534a97af44035ab90002565f88f439144 Merge: 5f54d64c78 e9aa5ec514 Author: Brandon Williams Date: Tue Jan 24 12:26:12 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit e9aa5ec514c436a8fb7dede10fa97b8d8cca200c Merge: 92245b0f07 9a64be3234 Author: Brandon Williams Date: Tue Jan 24 11:59:53 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 9a64be32340829038e4895a667eda025d5219f1b Author: Brandon Williams Date: Thu Jan 12 09:55:33 2023 -0600 Suppress CVE-2021-1471, CVE-2021-3064, CVE-2021-4235 commit eee052dad3ddce2b0ed951f7baa99f7efaf3684b Merge: 158875858c 53156c24e1 Author: Brandon Williams Date: Tue Jan 24 06:38:37 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit 53156c24e1b87b408e15b242a3ec73ad955095d7 Merge: da2dd2bfbc 5f54d64c78 Author: Brandon Williams Date: Tue Jan 24 06:34:17 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 5f54d64c78818125f47ecc509a757f4b55fcc893 Merge: 8bb9c72f58 92245b0f07 Author: Brandon Williams Date: Tue Jan 24 06:31:50 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 92245b0f07cda20d08a29bf5912018867ed16779 Merge: f52b316016 e7f55ab8c3 Author: Brandon Williams Date: Tue Jan 24 06:29:21 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit e7f55ab8c3bd6bac4c87354afec231d7237c35b8 Author: Brandon Williams Date: Thu Jan 12 06:50:40 2023 -0600 switch to SafeConstructor Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18150 commit 158875858c67a1518c70ce115530bc492df65ce2 Merge: 035705f494 da2dd2bfbc Author: Paulo Motta Date: Mon Jan 23 17:29:19 2023 -0500 Merge branch 'cassandra-4.1' into trunk commit da2dd2bfbc977811214c86a04ba9b3ae6dbf0ba3 Merge: 5c93731eee 8bb9c72f58 Author: Paulo Motta Date: Mon Jan 23 17:27:32 2023 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 8bb9c72f582de6bcc39522ba9ade91fd5bc22f67 Author: lzurovchak1 Date: Wed Dec 21 13:34:18 2022 -0500 Add safeguard so cleanup fails when node has pending ranges Patch by Lindsey Zurovchak; Reviewed by Paulo Motta, Stefan Miklosovic for CASSANDRA-16418 Closes #2061 commit 035705f49464a4854482e1f1280a7af45f7f0203 Author: williamvietnguyen Date: Wed Dec 21 11:10:35 2022 -0800 Print friendly error when nodetool attempts to connect to uninitialized server Patch by William Nguyen; Reviewed by Paulo Motta, Brandon Williams for CASSANDRA-11537 commit a77f941657d51f89148f8560f2644f98d21b75e0 Merge: 6226ec277b 5c93731eee Author: Stefan Miklosovic Date: Mon Jan 23 15:17:11 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit 5c93731eeef0570db6f817e555f8a78a4d444b30 Merge: b9b18aa5a2 669731b033 Author: Stefan Miklosovic Date: Mon Jan 23 15:14:58 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 669731b03379ab1d2c2b6b2acc7b705f6095c579 Merge: 7377e16121 f52b316016 Author: Stefan Miklosovic Date: Mon Jan 23 15:05:22 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit f52b3160168f1e53e2e99bd387be2c3fcd864c3f Merge: 6f719f811a 357c0d5a89 Author: Stefan Miklosovic Date: Mon Jan 23 14:58:54 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 357c0d5a896ca79065b0b3e18b4e6f33a87bd021 Author: Stefan Miklosovic Date: Fri Jan 20 13:25:26 2023 +0100 Expand build.dir property in rat targets patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18183 commit 6226ec277bb3140e241450b1893b04cff2f21a26 Merge: cab864a631 b9b18aa5a2 Author: Brandon Williams Date: Mon Jan 23 06:57:18 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit b9b18aa5a2a1f6ddafc45193ed1c36e9417e093e Merge: 6c96e2fd41 7377e16121 Author: Brandon Williams Date: Mon Jan 23 06:57:09 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 7377e16121057cc736dfee5449b5ad2fef9756f3 Merge: 9a0af4112e 6f719f811a Author: Brandon Williams Date: Mon Jan 23 06:56:00 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 6f719f811acec652178ec7c660640e4fc3bd798b Merge: fc3ad6525f fe71dfd7a1 Author: Brandon Williams Date: Mon Jan 23 06:53:53 2023 -0600 Merge branch 'cassandra-3.0' into 'cassandra-3.11' commit fe71dfd7a15ded62b70eee4a2981d54baf35be8d Author: Brandon Williams Date: Fri Jan 20 10:40:52 2023 -0600 Suppress CVE-2022-41881 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18148 commit cab864a6316d4091185559c7b21a8073bfcf42b2 Merge: 51e3149169 6c96e2fd41 Author: Andrés de la Peña Date: Mon Jan 23 11:43:39 2023 +0000 Merge branch 'cassandra-4.1' into trunk commit 6c96e2fd41368323844a080d806cbbb7b7fc9790 Merge: 5086b7d0a7 9a0af4112e Author: Andrés de la Peña Date: Mon Jan 23 11:42:24 2023 +0000 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 9a0af4112e87f5b97056aa39e63c5ab461b60237 Author: Andrés de la Peña Date: Mon Jan 9 13:50:47 2023 +0000 Fix legacy clustering serialization for paging with compact storage patch by Andrés de la Peña; reviewed by Berenguer Blasi and Piotr Kołaczkowski for CASSANDRA-17507 commit 51e31491695566cdf3e3114d05e592840d3621f1 Author: Brandon Williams Date: Tue Jan 17 14:18:56 2023 -0600 Upgrade cython Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18121 commit 4d192f728c9100c84a3215df31f29b5cda64baee Author: Brandon Williams Date: Fri Dec 16 10:09:39 2022 -0600 get newest pip in cassandra-cqlsh-tests.sh Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18121 commit 6f200472990b4693f9603d6e88a1460c44090bea Author: Brandon Williams Date: Thu Dec 8 12:05:16 2022 -0600 add py311 tests to circle Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18094 commit 25e4a89f76917076f0be56fb2b098a7d8f5ba06b Author: Brandon Williams Date: Sun Dec 4 13:05:19 2022 -0600 Accommodate python 3.11 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18088 commit 5086b7d0a7378fa6648d85900776e336a765361c Author: Brandon Williams Date: Tue Jan 17 14:18:56 2023 -0600 Upgrade cython Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18121 commit 4f32d43d88a36a5dbf0ae1d42bb0280805c99649 Author: Brandon Williams Date: Fri Dec 16 10:09:39 2022 -0600 get newest pip in cassandra-cqlsh-tests.sh Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18121 commit f66e7dcf33309b0b910f4d5199b00181c26436a4 Author: Brandon Williams Date: Thu Dec 8 11:47:03 2022 -0600 add py311 tests to circle Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18094 commit cd38edbb41bdf0babdb77c62f09650ae0a28f77c Author: Brandon Williams Date: Sun Dec 4 13:05:19 2022 -0600 Accommodate python 3.11 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18088 commit 9471ee762c59df1b5158286057df836c37075888 Merge: 7c86e18baf b43293e200 Author: Brandon Williams Date: Fri Jan 20 15:25:23 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit b43293e200dd95c98bfb3ee436fe05bb18da114f Author: Brandon Williams Date: Tue Jan 17 14:18:56 2023 -0600 Upgrade cython Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18121 commit f6933a041901861e526b44af6fc974b29b33982e Author: Brandon Williams Date: Fri Dec 16 12:29:57 2022 -0600 convert cqlshlib from nose to pytest Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18088 commit 563a26c6b1493f46ebb597085ae589f1835acd50 Author: Brandon Williams Date: Fri Dec 16 10:09:39 2022 -0600 get newest pip in cassandra-cqlsh-tests.sh Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18121 commit d38427a67b1b9684ca85d27a19f321325032dafe Author: Brandon Williams Date: Thu Dec 8 10:35:54 2022 -0600 add py311 tests to circle Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18094 commit 0b5248d8c3ef743a8dda42f7be5dadb70e373b51 Author: Brandon Williams Date: Sun Dec 4 13:05:19 2022 -0600 Accommodate python 3.11 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18088 commit 8413e9d6fdd114667deb81ea1d21d6eb486cb635 Merge: 45e00ea92f 7c86e18baf Author: Mick Semb Wever Date: Fri Jan 20 13:13:54 2023 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Update G1GC settings, and make it default in trunk commit 7c86e18baf10b3cf6c8f06ee9f1e27d2e21acf78 Author: Mick Semb Wever Date: Sat Jan 14 00:25:23 2023 +0100 Update G1GC settings, and make it default in trunk patch by Mick Semb Wever; patch by Anthony Grasso, Brandon Williams, Derek Chen-Becker, Jeremiah Jordan, Jon Haddad, Josh McKenzie for CASSANDRA-18027 commit 45e00ea92fd0f06830405b9839f801506632f934 Merge: 92fca2531f bb2c16c677 Author: Mick Semb Wever Date: Fri Jan 20 11:26:02 2023 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: EndpointState is mutable, so we must copy it before trying to serialize commit bb2c16c677b0f5ace6f6721069e8c21a584e33f9 Author: Benedict Elliott Smith Date: Sun Jan 15 12:20:12 2023 +0000 EndpointState is mutable, so we must copy it before trying to serialize patch by Benedict Elliott Smith; reviewed by Mick Semb Wever for CASSANDRA-18164 commit 92fca2531f4165c38821c01d41e32649d93965cb Merge: b07c3127cc 590d629302 Author: Stefan Miklosovic Date: Wed Jan 18 15:06:15 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit 590d629302a58f399272703b5d8e6bb91444334c Merge: be61ad9d03 815f7de346 Author: Stefan Miklosovic Date: Wed Jan 18 15:03:36 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 815f7de34657865228e35e5f593fbe9d82941d0a Merge: 175272fff4 fc3ad6525f Author: Stefan Miklosovic Date: Wed Jan 18 15:00:43 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit fc3ad6525f93234132e888d5d8d454e7eedcadb3 Merge: d32c52aba7 8ecd7616fe Author: Stefan Miklosovic Date: Wed Jan 18 14:57:15 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 8ecd7616fe5d3ce0cfe8f4621eda1905a9110db1 Author: German Eichberger Date: Wed Jan 18 14:55:19 2023 +0100 Default role is created with zero timestamp patch by German Eichberger; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-12525 Co-authored-by: Stefan Miklosovic commit b07c3127cce7e1a8b9d4c34e1359539e12f1b4a7 Author: ben Date: Mon Apr 2 10:49:35 2018 -0400 SimpleSeedProvider can resolve multiple IP addresses per DNS record patch by Ben Bromhead; reviewed by Stefan Miklosovic and Andrés de la Peña for CASSANDRA-14361 Co-authored-by: Stefan Miklosovic commit 1bba8ebee259c0e3cba16c44949cc8f2519794d0 Merge: 1528798a5b be61ad9d03 Author: Brandon Williams Date: Tue Jan 17 14:32:12 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit be61ad9d0371e4a2bbc8cfb664a334826a8bd088 Merge: 0ac3c55d65 175272fff4 Author: Brandon Williams Date: Tue Jan 17 14:31:11 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 175272fff46f8441890e481c031ad521df9c9983 Author: Maciej Sokol Date: Wed Jan 11 09:56:29 2023 +0100 Fix formatting of duration in cqlsh Patch by Maciej Sokol; reviewed by brandonwilliams and edimitrova for CASSANDRA-18141 commit 1528798a5bb36187cf2b952b5b5a8b0982d262fb Author: yziadeh Date: Tue Jan 3 16:11:06 2023 -0500 Add tests for CQL and cqlsh quote escaping Patch by Yaman Ziadeh; Reviewed by Paulo Motta and Brandom Williams for CASSANDRA-15458 commit 92e0d27a9d6ec7587ee5a5940c66812b3d225000 Merge: 36a99bcdcc 0ac3c55d65 Author: Brandon Williams Date: Tue Jan 17 09:59:55 2023 -0600 Merge branch 'cassandra-4.1' into trunk commit 0ac3c55d65f81dae917547c377f5505938d7e653 Merge: c9968a8e95 ac8ddfb321 Author: Brandon Williams Date: Tue Jan 17 09:58:59 2023 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ac8ddfb321551b8ffa75f75e5578713f9b083030 Merge: 6e6846892a d32c52aba7 Author: Brandon Williams Date: Tue Jan 17 09:57:51 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit d32c52aba72c182fab0c30cf3d32ce723dde582a Merge: de8daabab6 e0e63f4415 Author: Brandon Williams Date: Tue Jan 17 09:56:50 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit e0e63f44155a00a540407b402a6cae8480aa2c32 Author: Brandon Williams Date: Thu Jan 12 12:39:09 2023 -0600 Suppress CVE-2021-37533 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18146 commit 36a99bcdcc65eb2b3bb708f67e1bd75d35359f7f Author: Stefan Miklosovic Date: Mon Jan 16 13:37:21 2023 +0100 Remove mocking in InternalNodeProbe spying on StorageServiceMBean patch by Stefan Miklosovic; reviewed by Andrés de la Peña for CASSANDRA-18152 commit 49dfb805e9045c856181d6c2ac3b586b98d1a82a Author: maxwellguo Date: Mon Jan 16 19:49:38 2023 +0100 Add compaction_properties column to system.compaction_history table and nodetool compactionhistory command patch by Maxwell Guo; reviewed by Stefan Miklosovic and Jacek Lewandowski for CASSANDRA-18061 commit 0b47c57184cebeac54f5858571b880bcb90917c6 Author: Sharan Foga Date: Sun Nov 6 19:14:03 2022 +0100 Add documentation for partition denylisting patch by Sharan Foga; reviewed by Stefan Miklosovic and Josh McKenzie for CASSANDRA-17547 commit 6501f576e7c663d285a4db207e6b3dc7f887e8f4 Merge: eef76dbbed c9968a8e95 Author: Paulo Motta Date: Mon Jan 16 21:49:17 2023 -0500 Merge branch 'cassandra-4.1' into trunk commit c9968a8e95253a23846be1b5f502773a1eeea48d Merge: ff1ffd03da 6e6846892a Author: Paulo Motta Date: Mon Jan 16 21:46:49 2023 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 6e6846892a5d7ec2feaf2c35d380ed5975d09517 Author: Stefan Miklosovic Date: Thu Nov 5 11:17:02 2020 +0100 Fix sstable loading of keyspaces named snapshots or backups Patch by Stefan Miklosovic; Reviewed by Paulo Motta for CASSANDRA-14013 Co-authored-by: Paulo Motta commit eef76dbbed696074d67c7119fe464240a5f2d141 Merge: 530bc10bd0 ff1ffd03da Author: Bereng Date: Mon Jan 16 07:10:25 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit ff1ffd03dac2509233f8bb857b97f1db1f6dc653 Merge: 5be1038c5d 2c03055715 Author: Bereng Date: Mon Jan 16 07:08:21 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 2c03055715c874846a2970ec6ec13c2dcf8cdf8c Merge: a83de9bcd2 de8daabab6 Author: Bereng Date: Mon Jan 16 07:07:05 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit de8daabab662c23ee186461601226c236b8df5e0 Merge: b8a87abba4 f384e69482 Author: Bereng Date: Mon Jan 16 07:05:25 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit f384e694825a2cd0a95debf30accb153f00e80b0 Author: Bereng Date: Fri Jan 13 06:56:37 2023 +0100 When generate.sh fails its rc=0 - take 2 patch by Andres de la Peña and Berenguer Blasi; reviewed by Andres de la Peña for CASSANDRA-18032 Co-authored-by: Berenguer Blasi Co-authored-by: Andres de la Peña commit 530bc10bd0a053f5dcd8439fd3f5c72cd7952ea6 Author: Caleb Rackliffe Date: Wed Jan 11 13:04:06 2023 -0600 Remove ProtocolVersion entirely from the CollectionSerializer ecosystem patch by Caleb Rackliffe; reviewed by David Capwell for CASSANDRA-18114 commit 7df4530882c44f6de942b225af455f6399b66302 Merge: 995c3abc42 5be1038c5d Author: David Capwell Date: Thu Jan 12 09:46:57 2023 -0800 Merge branch 'cassandra-4.1' into trunk commit 5be1038c5d38af32d3cbb0545d867f21304f3a46 Author: David Capwell Date: Wed Jan 11 13:40:57 2023 -0800 Streaming progress virtual table lock contention can trigger TCP_USER_TIMEOUT and fail streaming patch by David Capwell; reviewed by Abe Ratnofsky, Jon Meredith for CASSANDRA-18110 commit 995c3abc420a6e44a6725616125445a0b024da12 Merge: 1b2e49e97c 438346aaa6 Author: Bereng Date: Thu Jan 12 08:21:36 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit 438346aaa6aa8b57ae6ee31fd6ab18e6369292f3 Merge: a0c6191238 a83de9bcd2 Author: Bereng Date: Thu Jan 12 08:19:22 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit a83de9bcd21a032c1dbc6dc323c7e53f828b2cb6 Merge: 6c65c9e49e b8a87abba4 Author: Bereng Date: Thu Jan 12 08:17:48 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit b8a87abba47441d97b3c85a0473d02919e1c071b Author: Bereng Date: Fri Dec 9 11:13:52 2022 +0100 Do not leak 2015 synthetic memtable Epoch patch by Berenguer Blasi; reviewed by Caleb Rackliffe for CASSANDRA-18118 commit 1b2e49e97c138124d2a74900dfefd5ea6fec1cef Merge: e936b2cc1b a0c6191238 Author: Bereng Date: Thu Jan 12 07:09:54 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit a0c6191238dfde1a21f078f6505b0a5ff42de1b2 Merge: da5ec9a8e3 6c65c9e49e Author: Bereng Date: Thu Jan 12 07:08:36 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 6c65c9e49e656fc1fc4ae23adda391d0e3d4f2b4 Merge: f17e431f24 7fe71274d0 Author: Bereng Date: Thu Jan 12 07:07:30 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 7fe71274d009b18c16f71c7e223eb8f225ed5a8e Merge: 156cc6fedc 69dfed3671 Author: Bereng Date: Thu Jan 12 07:06:04 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 69dfed3671144c019fb69793e3152b5a71d76421 Author: Bereng Date: Fri Nov 11 09:56:49 2022 +0100 When generate.sh fails its rc=0 patch by Berenguer Blasi; reviewed by Andres de la Peña for CASSANDRA-18032 commit e936b2cc1ba7f525c636de5f9fb1780ca70f1762 Author: Jordan West Date: Mon Jan 9 12:04:50 2023 -0800 fix serialization error in getsstables --show-levels Patch by Jordan West; Reviewed by Brandon Williams, Cheng Wang for CASSANDRA-18140; commit 386bf7815a0ac8df2185bf59d5bf1a9d0a50d069 Merge: baca1dd9be da5ec9a8e3 Author: Stefan Miklosovic Date: Tue Jan 10 09:07:40 2023 +0100 Merge branch 'cassandra-4.1' into trunk commit da5ec9a8e310cdf0abbcb102117f740b78e1c3b3 Merge: 073f7c36fa f17e431f24 Author: Stefan Miklosovic Date: Tue Jan 10 09:05:54 2023 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit f17e431f24f831321092b651a9d3adf7b30b8fca Merge: 7877d59d19 156cc6fedc Author: Stefan Miklosovic Date: Tue Jan 10 09:03:51 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 156cc6fedc2e1cbfd7ea6ca77224f7bc5c725cd2 Merge: fff313e447 877d6e6e95 Author: Stefan Miklosovic Date: Tue Jan 10 09:01:13 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 877d6e6e956ffa387a7d4ec1c7f0ca8ccb23f33e Author: Maxim Muzafarov Date: Tue Dec 20 00:28:25 2022 +0100 Add to the IntelliJ Git Window issue navigation links to Cassandra's Jira patch by Maxim Muzafarov; reviewed by David Capwell, Aleksey Yeschenko, Stefan Miklosovic for CASSANDRA-18126 commit baca1dd9be0321507a0ca0a69491507da1991614 Merge: d7c5c547f0 073f7c36fa Author: Jordan West Date: Sat Jan 7 13:45:28 2023 -0800 Merge branch 'cassandra-4.1' into trunk commit 073f7c36fa20e8d9410f306e57e7c7734ce74d1e Author: Jordan West Date: Fri Dec 30 09:20:35 2022 -0800 Return empty denylist entry instead of null on failure to load Patch by Jordan West; Reviewed by Josh McKenzie for CASSANDRA-18116 commit d7c5c547f025301780658b37e6e8a591bc4a5b36 Author: Ariel Weisberg Date: Tue Dec 6 15:20:14 2022 -0500 Use checked casts when reading vints as ints patch by Ariel Weisberg; reviewed by David Capwell and Caleb Rackliffe for CASSANDRA-18099 commit 5825a14f73bc3cfa8d32d7f74833a64d9fff4ad3 Merge: 7ecbad7ee8 1c2d299ad3 Author: Ekaterina Dimitrova Date: Fri Dec 23 15:12:19 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 1c2d299ad349c0d4a2022f668b04978f8b4f32c2 Merge: fabc6e7c7b 7877d59d19 Author: Ekaterina Dimitrova Date: Fri Dec 23 15:09:36 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 7877d59d19244b7aa701ebe7b2192d6e191ee3fb Merge: ee266d283e fff313e447 Author: Ekaterina Dimitrova Date: Fri Dec 23 15:05:40 2022 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit fff313e4470f1f64fd90da2a6aff84f3981a0189 Merge: 0767c83416 df0b139701 Author: Ekaterina Dimitrova Date: Fri Dec 23 15:01:11 2022 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit df0b13970198fcf7079a1044c5911d8ba0b6dac3 Author: Ekaterina Dimitrova Date: Thu Dec 22 17:21:48 2022 -0500 Use CircleCI Large executor instead of XLarge executor in MIDRES config for Python upgrade tests patch by Ekaterina Dimitrova; reviewed by Michael Semb Wever for CASSANDRA-18127 commit 7ecbad7ee846bf2648e2a91d8628f11ce95daece Merge: 69982767e5 fabc6e7c7b Author: Marcus Eriksson Date: Fri Dec 23 15:00:54 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit fabc6e7c7b1ceee21408fdb15822370042339e28 Merge: 203c61abaf ee266d283e Author: Marcus Eriksson Date: Fri Dec 23 14:59:59 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ee266d283e740785817d7cb6749e6c135f4aacbb Author: Marcus Eriksson Date: Thu Oct 20 08:18:06 2022 +0200 Avoid ConcurrentModificationException in STCS/DTCS/TWCS.getSSTables Patch by marcuse; reviewed by Aleksey Yeschenko for CASSANDRA-17977 commit 69982767e571abf8b414ac2f13a29d1cb0c7fef8 Merge: b171b4ba29 203c61abaf Author: Marcus Eriksson Date: Fri Dec 23 14:45:48 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 203c61abaf0a85ce6c9faa207987494ff5ce1338 Merge: 97f9ff7da3 95e26ad1ab Author: Marcus Eriksson Date: Fri Dec 23 14:44:19 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 95e26ad1ab29b52e720edf9792eba100817cce9d Merge: ace3920239 0767c83416 Author: Marcus Eriksson Date: Fri Dec 23 14:43:01 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 0767c834162d74bdc857f47615b9e8a5c7e76d5b Merge: baa9d0327f 730b898b74 Author: Marcus Eriksson Date: Fri Dec 23 14:41:50 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 730b898b7415e568138c6fc1e961f69b41ba1fa0 Author: Marcus Eriksson Date: Tue Oct 18 12:57:07 2022 +0200 Don't group TWCS sstables for anticompaction Patch by marcuse; reviewed by Aleksey Yeschenko for CASSANDRA-17970 commit b171b4ba294126e985d0ee629744516f19c8644e Merge: 227409d920 97f9ff7da3 Author: Stefan Miklosovic Date: Tue Dec 20 10:17:54 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 97f9ff7da3315426fb5df0ac33f741946993ce06 Merge: f869a2bb59 ace3920239 Author: Stefan Miklosovic Date: Tue Dec 20 10:14:36 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ace3920239f01ffbabbc424299900aa6d847946d Merge: f01d2b4a3c baa9d0327f Author: Stefan Miklosovic Date: Tue Dec 20 10:07:00 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit baa9d0327fcb656524df604ad242004ef2887a3c Merge: eb91e2c354 f55b2fb1b3 Author: Stefan Miklosovic Date: Tue Dec 20 10:02:32 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit f55b2fb1b3a8af03a3424abfe7afa2ac60ac9dff Author: Stefan Miklosovic Date: Tue Dec 6 15:49:09 2022 +0100 Do not spam the logs with MigrationCoordinator not being able to pull schemas patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18096 commit 227409d9201fa1aeb9f80b22f499577aedfe25bc Author: Jake Luciani Date: Mon Dec 19 16:05:12 2022 -0500 Add Mutation Serialization Caching Patch by T Jake Luciani; Reviewed by Josh McKenzie for CASSANDRA-17998 Co-authored-by: T Jake Luciani (jake@apache.org) Co-authored-by: Mike Adamson (mikeatdot@gmail.com) Co-authored-by: Robert Stupp (snazy@apache.org) commit b87299debe1bbbb5c48897d0199267d1a58067dc Merge: 070362c883 f869a2bb59 Author: Mick Semb Wever Date: Sat Dec 17 15:50:33 2022 -0800 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Restore custom param types over messaging system commit f869a2bb59ee8267ac7470f1df6d6351e3f21a69 Merge: af5029d643 f01d2b4a3c Author: Mick Semb Wever Date: Sat Dec 17 15:49:18 2022 -0800 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Restore custom param types over messaging system commit f01d2b4a3ca114ac58d95befcab3dbbad9d960aa Author: Mick Semb Wever Date: Sat Nov 12 12:02:35 2022 +0100 Restore custom param types over messaging system patch by Mick Semb Wever; reviewed by Aleksey Yeschenko for CASSANDRA-17981 commit 070362c883d6b2609966837645d320d2d442510f Merge: 94bcb4e5ec af5029d643 Author: Mick Semb Wever Date: Fri Dec 16 11:34:53 2022 -0800 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Increment versions to 4.1.1 commit af5029d643f459049b5d19f316471b3c4685f44a Author: Mick Semb Wever Date: Fri Dec 16 11:34:23 2022 -0800 Increment versions to 4.1.1 commit 94bcb4e5ec4fb99b73276d90b9d08def6f3b4d30 Author: Marcus Eriksson Date: Thu Sep 1 09:43:47 2022 +0200 Only reload compaction strategies if disk boundaries change Patch by Aleksey Yeschenko and marcuse; reviewed by Aleksey Yeschenko for CASSANDRA-17874 Co-authored-by: Aleksey Yeschenko commit 4941b279bceb3a0fbe388be4305d25ec771f7b78 Author: David Capwell Date: Tue Dec 13 13:37:40 2022 -0800 CEP-10: Simulator Java11 Support patch by David Capwell; reviewed by Benedict Elliott Smith, Ekaterina Dimitrova for CASSANDRA-17178 commit 2e1695426bf363113f5ace90b08ae5d1fade40b1 Merge: f0ad7eadbe f9e033f519 Author: Mick Semb Wever Date: Tue Dec 13 08:22:47 2022 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Prepare debian changelog for 4.1.0 commit f9e033f519c14596da4dc954875756a69aea4e78 Author: Mick Semb Wever Date: Wed Dec 7 21:53:57 2022 +0100 Prepare debian changelog for 4.1.0 commit f0ad7eadbeb3208e08a9339881931222fdab253b Merge: 3f83396b76 81c616826a Author: Ariel Weisberg Date: Wed Dec 7 15:23:00 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 81c616826ab3c8aa96467771d0e074a874efdd77 Author: Marianne Lyne Manaog Date: Thu Nov 24 15:02:16 2022 +0000 Fix ContentionStrategy backoff and Clock.waitUntil Also for CASSANDRA-18087 patch by Matt Fleming, Marianne Lyne Manaog, Benedict Elliott Smith; reviewed by Joshua Mckenzie, Ariel Weisberg for CASSANDRA-18086 Co-authored-by: Marianne Lyne Manaog Co-authored-by: Matt Fleming Co-authored-by: Benedict Elliott Smith commit 3f83396b76115a25b8189410c68775f87f31f8ac Author: maxwellguo Date: Thu Nov 17 21:04:23 2022 +0800 Set the major compaction type correctly for compactionstats Patch by Maxwell Guo; reviewed by brandonwilliams and blambov for CASSANDRA-18055 commit ccada788c47882bfb10d7cb86f7f39c9865428f2 Author: Stefan Miklosovic Date: Wed Nov 30 21:31:43 2022 +0100 Print exception message without stacktrace when nodetool commands fail on probe.getOwnershipWithPort() Consequently, there is also alignement of nodetool ring command returning exit code 1 in case there is unrecoverable exception thrown, same as was already done for status and describecluster commands. patch by Stefan Miklosovic; reviewed by Brandon Williams and Yifan Cai for CASSANDRA-18079 commit 235d2df0eea4a2c70bbdcb1b9f4d2e121080f4c3 Merge: 33c60d8daf 27fff06bb7 Author: Yifan Cai Date: Mon Dec 5 18:57:19 2022 -0800 Merge branch 'cassandra-4.1' into trunk commit 27fff06bb761381c95dfad2100fb79586005bf32 Merge: 8889b27c9c f22263cd8a Author: Yifan Cai Date: Mon Dec 5 18:52:56 2022 -0800 Merge branch 'cassandra-4.0' into cassandra-4.1 commit f22263cd8afe353e51ff6135e9bc6f2eeaafa2b2 Merge: c2bbee2020 eb91e2c354 Author: Yifan Cai Date: Mon Dec 5 18:50:39 2022 -0800 Merge branch 'cassandra-3.11' into cassandra-4.0 commit eb91e2c354d4436913066bf8c15594a64c564424 Merge: b7762e2aa2 473656c1d5 Author: Yifan Cai Date: Mon Dec 5 18:48:39 2022 -0800 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 473656c1d53edb998aa60d414221e397797de52b Author: Yifan Cai Date: Tue Nov 29 15:19:06 2022 -0800 Fix incorrect resource name in LIST PERMISSION output patch by Yifan Cai; reviewed by Berenguer Blasi, Sam Tunnicliffe for CASSANDRA-17848 commit 33c60d8daf7b25ffc80136289aaa4e6c9589b2c0 Merge: 279f284da5 8889b27c9c Author: Brandon Williams Date: Mon Dec 5 10:10:37 2022 -0600 Merge branch 'cassandra-4.1' into trunk commit 8889b27c9cfd7820a4a7d31b4fb63d2059b35ff9 Merge: cc4c8a3637 c2bbee2020 Author: Brandon Williams Date: Mon Dec 5 10:08:09 2022 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit c2bbee2020af7b07eb478c10df21a8d081ec6a7e Merge: bba7ab3eca b7762e2aa2 Author: Brandon Williams Date: Mon Dec 5 10:06:17 2022 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit b7762e2aa276ecf9cad2dd26ee52fe2463ae52db Merge: 5a53c36515 92019df4d8 Author: Brandon Williams Date: Mon Dec 5 10:02:35 2022 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 92019df4d8540b384d7fb8655f7c02293f7f7ec1 Author: Brandon Williams Date: Wed Nov 30 09:44:25 2022 -0600 Suppress CVE-2022-41854 and similar Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18083 commit 279f284da5cfe8b4766921d3b1b4d6e299dbe66d Author: Jordan West Date: Wed Nov 9 12:59:34 2022 -0800 Add option to print level with getsstables output Patch by Jordan West; reviewed by Brandon Williams and Cheng Wang for CASSANDRA-18023 commit 797b96958af07945b0a1799024c1ef664dfee2e2 Author: Stefan Miklosovic Date: Fri Nov 18 20:45:37 2022 +0100 Implement a guardrail for not having zero default_time_to_live on tables with TWCS patch by Stefan Miklosovic; reviewed by Andrés de la Peña, Josh McKenzie, Brandon Williams and Brad Schoening for CASSANDRA-18042 Co-authored-by: Andrés de la Peña commit 13654332fe4f8da4124d29454afbe5d3cab59fee Merge: b7c7972a51 cc4c8a3637 Author: Jordan West Date: Tue Nov 29 18:32:25 2022 -0800 Merge branch 'cassandra-4.1' into trunk commit cc4c8a36374d507215ea0e44cf6f73f308a9764f Merge: b807f97b37 bba7ab3eca Author: Jordan West Date: Tue Nov 29 18:29:23 2022 -0800 Merge branch 'cassandra-4.0' into cassandra-4.1 commit bba7ab3eca536744cb04cf92d19a9291eb1012f2 Merge: 1df8174d67 5a53c36515 Author: Jordan West Date: Tue Nov 29 18:27:45 2022 -0800 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 5a53c36515e5f29982458c9a09f3f900d4ac20a4 Merge: 7572725def 13d495aa7d Author: Jordan West Date: Tue Nov 29 18:26:40 2022 -0800 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 13d495aa7d5b7a7c121fcc9e105f79107c5c2a1c Author: Jordan West Date: Mon Jan 10 16:42:47 2022 -0800 fix intermittent failures in nodetool toppartitions caused by failure to validate invalid bytes Patch by Jordan West; Reviewed by Brandon Williams and Cheng Wang for CASSANDRA-17254 commit b7c7972a51ab6be6e5f410d2b12c770f5b7ebc98 Author: Andrés de la Peña Date: Wed Nov 16 12:23:43 2022 +0000 Add CQL scalar functions for collection aggregation patch by Andrés de la Peña; reviewed by Benjamin Lerer and Berenguer Blasi for CASSANDRA-18060 commit 3775c8f11dc998ba9c74769a14b9443272c750c6 Author: Stefan Miklosovic Date: Wed Nov 16 13:27:09 2022 +0100 Make cassandra.replayList property for CommitLogReplayer possible to react on keyspaces only Property is meant to contain comma-separated entities which are either names of keyspaces or keyspaces and tables or their mix. Examples: - just keyspaces: -Dcassandra.replayList=ks1,ks2,ks3 - specific tables: -Dcassandra.replayList=ks1.tb1,ks2.tb2 - mix of tables and keyspaces: -Dcassandra.replayList=ks1.tb1,ks2 If only keyspaces are specified, mutations for all tables in such keyspace will be replayed. patch by Stefan Miklosovic; reviewed by Branimir Lambov for CASSANDRA-18044 commit 88dc64d2086c9a91f00ee024b8ef13cb2c193ee6 Author: XV4DE Date: Sat May 14 12:28:40 2022 -0700 Add Mathematical Functions Patch by Simon Chess; review by Benjamin Lerer Ekaterina Dimitrova for CASSANDRA-17221 This patch add the abs, exp, log, log10, and round functions for the numeric types. commit 48d4897828e9f3fbbd033feb383ea6fbe6d0d96c Merge: 707d6bb527 b807f97b37 Author: Mick Semb Wever Date: Tue Nov 22 16:48:11 2022 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Prepare debian changelog for 4.1.0 commit b807f97b37933fac251020dbd949ee8ef245b158 Author: Mick Semb Wever Date: Tue Nov 22 15:58:24 2022 +0100 Prepare debian changelog for 4.1.0 commit 707d6bb527c378fda9cc2a7b3c446593bdffdd19 Merge: ee8b66da8c 676b3e4483 Author: Mick Semb Wever Date: Tue Nov 22 15:43:03 2022 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Increment version to 4.1.0 commit 676b3e448375b409a2721ff74f830c8db70410be Author: Mick Semb Wever Date: Tue Nov 22 15:41:18 2022 +0100 Increment version to 4.1.0 commit ee8b66da8ce3bdac0378f89159d8bd7e45a91363 Author: maxwellguo Date: Thu Oct 27 22:05:49 2022 +0800 Make incremental backup configurable per table patch by Maxwell Guo; reviewed by Stefan Miklosovic and Benjamin Lerer for CASSANDRA-15402 commit fb09a5c1bbd73ddf7f5891fbbbc0bffe53cbee56 Merge: 3860a90c5f bc16752bd7 Author: Ekaterina Dimitrova Date: Fri Nov 18 15:44:30 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit bc16752bd73356364cdf990a093521771b5e488a Author: Ekaterina Dimitrova Date: Sat Nov 5 15:29:21 2022 -0400 Restore Virtual Tables page, and update nav for new pages patch by Ekaterina Dimitrova; reviewed by Mick Semb Wever for CASSANDRA-17976 Co-authored-by: Mick Semb Wever commit 3860a90c5fa9497c1e7858750244e04b25a7d351 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Sat Nov 12 18:31:10 2022 -0500 Change shebangs of Python scripts to resolve Python 3 from env command patch by Brad Schoening; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17832 Co-authored-by: Stefan Miklosovic commit 9f99e72aae812b86d277883601450bc0e7bb9463 Author: Andrés de la Peña Date: Fri Nov 4 12:39:21 2022 +0000 Add reasons to guardrail messages and consider guardrails in the error message for needed ALLOW FILTERING * Add optional reason for guarding an operation to every guardrail * Add reason for disabling ALLOW FILTERING into cassandra.yaml * Consider allow_filtering_enabled guardrail in the error message for needed ALLOW FILTERING patch by Andrés de la Peña; reviewed by Berenguer Blasi and Josh McKenzie for CASSANDRA-17967 commit 6da9e33602fad4b8bf9466dc0e9a73665469a195 Author: Andrés de la Peña Date: Fri Sep 2 13:39:10 2022 +0100 Add support for CQL functions on collections, tuples and UDTs patch by Andrés de la Peña; reviewed by Benjamin Lerer for CASSANDRA-17811 commit 145dbd1f0875ae4c54392125e09ed05153c0dd8f Author: Saranya Krishnakumar Date: Wed Oct 19 16:58:41 2022 -0700 Add flag to exclude nodes from local DC when running nodetool rebuild Patch by Saranya Krishnakumar; reviewed by Dinesh Joshi, Marcus Eriksson, Yifan Cai for CASSANDRA-17870 commit f4748e977f3a372d0858fd8917185e13fd4e2499 Merge: 4a65120f70 d6822c45ae Author: Mick Semb Wever Date: Thu Nov 17 12:28:36 2022 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Prepare debian changelog for 4.1-rc1 commit d6822c45ae3d476bc2ff674cedf7d4107b8ca2d0 Author: Mick Semb Wever Date: Thu Nov 17 11:33:50 2022 +0100 Prepare debian changelog for 4.1-rc1 commit 4a65120f709d106b14f9870096b981375c39db1b Merge: 930f141fa0 6be7fed242 Author: Mick Semb Wever Date: Thu Nov 17 10:56:39 2022 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Prepare for 4.1-rc1 commit 6be7fed24276713f4f330b27e37afc126dcc264e Author: Mick Semb Wever Date: Thu Nov 17 10:55:14 2022 +0100 Prepare for 4.1-rc1 commit 930f141fa0f203df020d14903548e72aae2743f2 Merge: e08c7a6295 9f58d76f38 Author: Branimir Lambov Date: Wed Nov 16 10:30:39 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 9f58d76f3841864be11f5b9c4534027451328569 Author: Branimir Lambov Date: Mon Nov 14 13:59:05 2022 +0200 Avoid schema mismatch problems on memtable API misconfiguration patch by Branimir Lambov; reviewed by Caleb Rackliffe for CASSANDRA-18040 commit e08c7a6295eed716e9973fbd809dfca32d68a3e8 Author: Branimir Lambov Date: Mon Nov 14 13:59:05 2022 +0200 Avoid schema mismatch problems on memtable API misconfiguration patch by Branimir Lambov; reviewed by Caleb Rackliffe for CASSANDRA-18040 commit 36e16ee3c911c710129fcf3a69595038c3dbd385 Author: Jyothsna Konisa Date: Mon Nov 14 14:16:07 2022 -0800 Adding endpoint verification option to client_encryption_options patch by Jyothsna Konisa; reviewed by Jon Meredith, Yifan Cai for CASSANDRA-18034 commit d5fd0dceee8f9ac5ed454e781a8c319a7b1b6677 Merge: f317eb4f2f 444da423d5 Author: Ekaterina Dimitrova Date: Tue Nov 15 16:41:42 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 444da423d5a0e846852a51c80bb7a128b7a50d24 Author: Ekaterina Dimitrova Date: Fri Nov 11 22:07:12 2022 -0500 CirleCI: Add large Dtests and J8/J11 Java distributed tests; Fix CQLSHLIB tests config and Python DTests split; CQLSHLIB tests should run with NUM_TOKENS=16 patch by Ekaterina Dimitrova; reviewed by Andres de la Pena for CASSANDRA-18001 commit f317eb4f2f22d5babbc83f8fe5b0160def613523 Merge: 15be17ecef ca00caffab Author: Bereng Date: Tue Nov 15 09:35:06 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit ca00caffab452199bfae3f950dc0ea873cc7c917 Merge: 19f55cf0b8 1df8174d67 Author: Bereng Date: Tue Nov 15 09:32:51 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 1df8174d67f67a75b5b17e2dff7f9838dc3e3ff6 Author: Bereng Date: Mon Nov 14 13:41:26 2022 +0100 Cqlshlib tests are j11 greedy Patch by Berenguer Blasi; reviewed by Ekaterina Dimitrova and Michael Semb Wever for CASSANDRA-18039 commit 15be17ecef53adf575732fc8aa0f86eb1a774092 Merge: 8a5bdf61db 19f55cf0b8 Author: Stefan Miklosovic Date: Mon Nov 14 13:52:35 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 19f55cf0b82f8c861be7e9616913961c80dfcd80 Author: Stefan Miklosovic Date: Thu Nov 10 12:45:37 2022 +0100 Start Paxos auto repair in CassandraDaemon This patch also renames PaxosRepairTest2 to PaxosRepair2Test to be picked up by CI. patch by Stefan Miklosovic; reviewed by Benedict Elliott Smith for CASSANDRA-18029 commit 8a5bdf61db8dfe1d83d03f41753a46087082b2ee Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Thu Nov 10 22:24:28 2022 -0500 replace local wcwidth.py with pypi module 'wcwidth' Patch by Brad Schoening; reviewed by brandonwilliams and smiklosovic for CASSANDRA-17287 commit fdc88a96eb7e279e27af6018643bb493662783e6 Merge: cbafc28c5a 5cd012736e Author: Alex Sorokoumov Date: Fri Nov 11 19:53:25 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 5cd012736e4680bbb25928ee7fbcea4859878fff Author: Alex Sorokoumov Date: Tue Nov 1 11:52:31 2022 +0100 VIntCoding handles BB with less than 8 bytes Without this fix VIntCoding#writeUnsignedVInt(long, ByteBuffer) throws a runtime error if the BB size is less than 8 bytes. This method also silently does not write to BB if it has less than 8 bytes left. patch by Alex Sorokoumov; reviewed by Benjamin Lerer for CASSANDRA-15215 commit cbafc28c5aec422e3b953eaa2f03a31d554312da Merge: 39ded1844f 207045030e Author: Stefan Miklosovic Date: Fri Nov 11 13:57:49 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 207045030ef4ff9ae370d9d30b342f395ffbc8dc Merge: 3e86412eb4 2638051af1 Author: Stefan Miklosovic Date: Fri Nov 11 13:47:44 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 2638051af186eb7cee110c103ae011f853ae8bb3 Author: Stefan Miklosovic Date: Thu Nov 10 18:53:40 2022 +0100 Harden parsing of boolean values in CQL in PropertyDefinitions patch by Stefan Miklosovic; reviewed by Aleksey Yeschenko for CASSANDRA-17878 commit 39ded1844f64ce04ec089625948e8073a09ddb99 Merge: fca550bf0a 3e86412eb4 Author: Bereng Date: Fri Nov 11 11:00:37 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 3e86412eb4d20ffffd19c13214aee89a5d776ac5 Merge: 106bc89d74 0f2fb1eeb3 Author: Bereng Date: Fri Nov 11 10:59:25 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 0f2fb1eeb3e1616b80bae9bae32b824c17de6ae5 Author: Bereng Date: Fri Nov 11 07:13:58 2022 +0100 Cqlsh Cython failure on 4.0 Patch by Berenguer Blasi; reviewed by Michael Semb Wever for CASSANDRA-18036 commit fca550bf0a70c9d79a377f45888b660ce89d7a5f Merge: f3865771cb 106bc89d74 Author: Brandon Williams Date: Thu Nov 10 13:40:54 2022 -0600 Merge branch 'cassandra-4.1' into trunk commit 106bc89d741925e19a0c80c1d84e2b8794d1ece1 Merge: 98f11efe44 35826b7447 Author: Brandon Williams Date: Thu Nov 10 13:37:59 2022 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 35826b7447b0c3233ca3dacad2e44de082e76896 Author: Natnael Adere Date: Wed Oct 19 14:34:05 2022 -0400 Fix error message about type hints (CASSANDRA-17915) Patch by Natnael Adere; reviewed by blerer and brandonwilliams for CASSANDRA-17915 commit f3865771cb3c09304db3129b52d6fe47b89ab4e5 Merge: bdfa27f246 98f11efe44 Author: Ekaterina Dimitrova Date: Thu Nov 10 12:37:53 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 98f11efe447271db2b8e68810b309cc22d554e80 Merge: 9fe9b04d85 2b3d0d32a7 Author: Ekaterina Dimitrova Date: Thu Nov 10 12:34:07 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 2b3d0d32a7da99949c0fa7f91af89c12deb20e3f Merge: 38806e795b 7572725def Author: Ekaterina Dimitrova Date: Thu Nov 10 12:31:07 2022 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 7572725defd4d5bba12097046b8338a955d8d032 Author: Derek Chen-Becker Date: Wed Oct 26 14:54:05 2022 -0600 [CASSANDRA-17950] Enable dtest-offheap in CircleCI The dtest-offheap test was only added to the Jenkins configuration, so this commit adds it to the CircleCI build for parity. Patch by Derek Chen-Becker; reviewed by Ekaterina Dimitrova, Derek Chen-Becker, Michael Semb Wever, and Andres de la Pena for CASSANDRA-17950 Co-authored-by: Ekaterina Dimitrova Co-authored-by: Derek Chen-Becker commit bdfa27f2464ae99db20b2cef01ce13148caf62a8 Merge: 6e3770bc15 9fe9b04d85 Author: Mick Semb Wever Date: Thu Nov 10 14:42:23 2022 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Upgrade Cython, minimum required now is 0.27.2 commit 9fe9b04d8589b2fe8ff592ca94d04af483438c42 Merge: 8f47d232fa 38806e795b Author: Mick Semb Wever Date: Thu Nov 10 14:41:54 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Upgrade Cython, minimum required now is 0.27.2 commit 38806e795b4f8745616f7c48c9bdae0760dcf7b5 Merge: 70478a1578 a44209cf43 Author: Mick Semb Wever Date: Thu Nov 10 14:41:04 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Upgrade Cython, minimum required now is 0.27.2 commit a44209cf4314a59f49884f5e35b201af5a200d37 Merge: 9213335f59 d9ef794ef6 Author: Mick Semb Wever Date: Thu Nov 10 14:40:10 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Upgrade Cython, minimum required now is 0.27.2 commit d9ef794ef6a03c2d7e0b1476cc0d29a008040095 Author: Mick Semb Wever Date: Thu Nov 10 12:08:04 2022 +0100 Upgrade Cython, minimum required now is 0.27.2 patch by Mick Semb Wever; reviewed by Berenguer Blasi for CASSANDRA-18028 commit 6e3770bc154ffd201b306febd92cfc14101efbbf Author: Stefan Miklosovic Date: Thu Nov 10 11:08:11 2022 +0100 Harden parsing of boolean values in CQL in PropertyDefinitions This patch also cleans up related classes and fixes methods to return primitive values instead of objects. patch by Stefan Miklosovic; reviewed by Aleksei Yeschenko for CASSANDRA-17878 commit d0e0e20f80f7d5a4b37d542b65f2a781ce1d3015 Merge: d8e0991775 8f47d232fa Author: Brandon Williams Date: Wed Nov 9 12:00:31 2022 -0600 Merge branch 'cassandra-4.1' into trunk commit 8f47d232faeacb44087712dd5524db96d92aae46 Merge: e12dc48c87 70478a1578 Author: Brandon Williams Date: Wed Nov 9 11:59:49 2022 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 70478a1578fe4facb0f59fcb40529e0a4040ac81 Author: Brandon Williams Date: Mon Oct 31 11:51:41 2022 -0500 Fix testCommitLogInitWithException: signal condition instead of killing thread Patch by brandonwilliams; reviewed by adelapena and bereng for CASSANDRa-17928 commit d8e0991775f8063c72c8ddc69282bf22a4789bcd Merge: 0778ae362c e12dc48c87 Author: Jacek Lewandowski Date: Wed Nov 9 17:22:35 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit e12dc48c879d5c8ddc5b270f4c7c8d9f098ec375 Merge: c100d34a1a 67c530171c Author: Jacek Lewandowski Date: Wed Nov 9 17:20:35 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 67c530171c7db1196c83783b346cc368494c6aa3 Merge: cea850d67d 9213335f59 Author: Jacek Lewandowski Date: Wed Nov 9 17:17:54 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 9213335f59293926b2d643fa8a156a882495dd42 Author: Jacek Lewandowski Date: Thu Nov 3 17:29:18 2022 +0100 Fix Splitter sometimes creating more splits than requested Spliter.splitOwnedRanges for some inputs creates an extra split. For example, when we request 7 ranges from 0..31 range, it will return 8 ranges. There is an assertion in that method which verifies whether it returns the requested number of splits. Since those numbers differs, when Cassandra is be started with assertions enabled, it would fail. patch by Jacek Lewandowski; reviewed by Marcus Eriksson for CASSANDRA-18013 commit 0778ae362c56354959b5313be20017b9ea5d2763 Merge: af3eea7558 c100d34a1a Author: Mick Semb Wever Date: Wed Nov 9 11:41:37 2022 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Restore streaming_keep_alive_period functionality on the netty control streaming channel commit c100d34a1a1d497eee85ca10c5cc1e43eb4871c4 Author: Aleksey Yeschenko Date: Wed Nov 2 17:30:57 2022 +0000 Restore streaming_keep_alive_period functionality on the netty control streaming channel patch by Aleksey Yeschenko, Mick Semb Wever; reviewed by Mick Semb Wever, Berenguer Blasi for CASSANDRA-17768 Co-authored-by: Aleksey Yeschenko commit af3eea7558e4da3976def20ac78070e79f8165f3 Merge: 9490f9667d 6f431c13a6 Author: Andrés de la Peña Date: Tue Nov 8 17:50:32 2022 +0000 Merge branch 'cassandra-4.1' into trunk commit 6f431c13a6694173b12c62770b889ad09b9598b4 Merge: a805e32675 cea850d67d Author: Andrés de la Peña Date: Tue Nov 8 17:49:50 2022 +0000 Merge branch 'cassandra-4.0' into cassandra-4.1 commit cea850d67d5b726fe7ca9fef39c8578b46210155 Merge: 7dbbe6907b 7b7762826e Author: Andrés de la Peña Date: Tue Nov 8 17:49:06 2022 +0000 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 7b7762826e367b2a82ba1e99b12f8f19c583b920 Merge: f9b2cd6f3f 955231cacf Author: Andrés de la Peña Date: Tue Nov 8 17:48:29 2022 +0000 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 955231cacfc2732dd1fd4275049e224ab220d107 Author: Andrés de la Peña Date: Tue Nov 8 13:27:49 2022 +0000 CircleCI: Remove repeated jobs from default LOWRES, MIDRES and HIGHRES files patch by Andrés de la Peña; reviewed by Berenguer Blasi and Ekaterina Dimitrova for CASSANDRA-18024 commit 9490f9667dd6c551d1d962c890f5af43ac9d7f76 Merge: 873e024a32 a805e32675 Author: Jacek Lewandowski Date: Tue Nov 8 08:23:48 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit a805e3267572a86dc731df509f0540fc85e08b78 Author: Doug Rohrer Date: Wed Sep 21 15:12:12 2022 -0400 Move Schema.FORCE_LOAD_KEYSPACES and Schema.FORCE_LOAD_KEYSPACES_PROP to CassandraRelevantProps patch by Doug Rohrer; reviewed by Brandon Williams and Jacek Lewandowski for CASSANDRA-17783 commit 873e024a32d37de08550c8106a8d7fd52bda588b Author: Cheng Wang Date: Fri Jun 17 15:53:34 2022 -0700 CASSANDRA-17711: Add nodetool forcecompact commit 037e709ee6f59fdf5adef828783ccc11e5de8941 Merge: addc5748a7 9dcba20866 Author: Andrés de la Peña Date: Mon Nov 7 17:31:41 2022 +0000 Merge branch 'cassandra-4.1' into trunk commit 9dcba208667589aa2309e86cfdd90c7d09682ff1 Merge: acda6fad68 7dbbe6907b Author: Andrés de la Peña Date: Mon Nov 7 17:31:02 2022 +0000 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 7dbbe6907be58138f2c88d0adaab5c99a038e2b1 Author: Andrés de la Peña Date: Mon Nov 7 15:41:47 2022 +0000 CircleCI: Fix j11_utests_fqltool executor patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova for CASSANDRA-18020 commit addc5748a74680b432ffabfe248245020902112c Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Tue Oct 4 16:21:10 2022 -0400 Add missing cqlsh completion and round out cqlsh completion test coverage IF (NOT) EXISTS is now also offered in cqlsh completion for CREATE TYPE, DROP TYPE, CREATE ROLE and DROP ROLE patch by Brad Schoening; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-16640 commit f236f40fc526dd75db648f4c6cc34d7f33fbd8f3 Merge: 4469f017c0 acda6fad68 Author: Branimir Lambov Date: Mon Nov 7 15:05:16 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit acda6fad688e95e0a9a34de28e3b61a775e59736 Author: Marianne Lyne Manaog Date: Mon Nov 7 10:33:02 2022 +0000 Bug fix for WriteTimeoutException when using Paxos v2 with a single datacenter patch by Marianne Lyne Manaog; reviewed by Branimir Lambov and Benedict Elliott Smith for CASSANDRA-17999 commit 4469f017c0253cbbdea9f1fa656417f704f421b1 Merge: f67f5613f4 2695bf7d38 Author: Mick Semb Wever Date: Sat Nov 5 16:14:19 2022 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: ninja-fix antcall maybe-build-test needs inheritRefs (CASSANDRA-18000) commit 2695bf7d381deffc1e056970553a22e65531820f Merge: 1cdb747b4c 17e6942775 Author: Mick Semb Wever Date: Sat Nov 5 16:00:35 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: ninja-fix antcall maybe-build-test needs inheritRefs (CASSANDRA-18000) commit 17e694277597404aadfec664dfe4c11da7ed24a4 Merge: 5e705a8492 f9b2cd6f3f Author: Mick Semb Wever Date: Sat Nov 5 15:51:47 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: ninja-fix antcall maybe-build-test needs inheritRefs (CASSANDRA-18000) commit f9b2cd6f3f1bdfa775b6e8bde7e5b22dc6e60d0b Merge: bdd6606486 30a3266780 Author: Mick Semb Wever Date: Sat Nov 5 15:50:56 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: ninja-fix antcall maybe-build-test needs inheritRefs (CASSANDRA-18000) commit 30a3266780c841f52a6accfece48d138d839e32d Author: Mick Semb Wever Date: Sat Nov 5 15:50:28 2022 +0100 ninja-fix antcall maybe-build-test needs inheritRefs (CASSANDRA-18000) commit f67f5613f4a925baae243d3950ed35784a3bf87a Merge: 31341ee4bd 1cdb747b4c Author: Andrés de la Peña Date: Fri Nov 4 16:27:38 2022 +0000 Merge branch 'cassandra-4.1' into trunk commit 1cdb747b4ca11b4d3d8ea76633bdf5de64175fc3 Merge: 1e67ef02f5 5e705a8492 Author: Andrés de la Peña Date: Fri Nov 4 16:26:51 2022 +0000 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 5e705a84927bd4de1020adec94b4dcc86bd23354 Merge: 6885bf9724 bdd6606486 Author: Andrés de la Peña Date: Fri Nov 4 16:26:08 2022 +0000 Merge branch 'cassandra-3.11' into cassandra-4.0 commit bdd6606486e91024e123afc4e56465a95b102d1a Merge: 43dcb5cd14 95400649d8 Author: Andrés de la Peña Date: Fri Nov 4 16:25:25 2022 +0000 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 95400649d83f5759a452a381eb6034f4c62bda94 Author: Andrés de la Peña Date: Thu Nov 3 18:57:34 2022 +0000 Improve behaviour of the Ant flag to skip test building, no-build-test patch by Andrés de la Peña; reviewed by Berenguer Blasi for CASSANDRA-18000 commit 31341ee4bd610848609abf6ecd5af805785404de Merge: 51cc20a9f8 1e67ef02f5 Author: Ekaterina Dimitrova Date: Fri Nov 4 10:54:13 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 1e67ef02f5e36a22e2f1987d38996349398b1d36 Merge: 162d504be0 6885bf9724 Author: Ekaterina Dimitrova Date: Fri Nov 4 10:50:05 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 6885bf9724167416a89ea5ba4e5cb4a495ebecd2 Merge: 8f06ec372e 43dcb5cd14 Author: Ekaterina Dimitrova Date: Fri Nov 4 10:41:47 2022 -0400 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 43dcb5cd14d06ad1708d884280b80f00345ec599 Merge: b82dca6f81 90fe8608a4 Author: Ekaterina Dimitrova Date: Fri Nov 4 10:38:12 2022 -0400 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 90fe8608a401dd086c8918a045d01ff288849f49 Author: Ekaterina Dimitrova Date: Wed Sep 21 12:17:33 2022 -0400 CirleCI: fix Python upgrade DTests config patch by Ekaterina Dimitrova; reviewed by Michael Semb Wever for CASSANDRA-17912 commit 51cc20a9f8eca301c6ba117d055623c610c55279 Merge: e027229238 162d504be0 Author: Mick Semb Wever Date: Fri Nov 4 13:40:58 2022 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Update What's New in 4.1 docs page WEBSITE - Updated What's New page for 4.1 commit 162d504be0a17e86ed44d58ff17136ef53e794d4 Author: Mick Semb Wever Date: Wed Nov 2 19:08:51 2022 +0100 Update What's New in 4.1 docs page patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova, Berenguer Blasi, Erick Ramirez for CASSANDRA-17976 commit e027229238cafc105fd7913d0aab7e95b6930a34 Author: Erick Ramirez Date: Fri Nov 4 11:58:12 2022 +1100 WEBSITE - Updated What's New page for 5.0 in trunk patch by Michael Semb Wever, Erick Ramirez; reviewed by Ekaterina Dimitrova, Berenguer Blasi, Erick Ramirez for CASSANDRA-17976 Co-authored by: Mick Semb Wever Co-authored by: Erick Ramirez commit a4b90fbc011f2cfab464f375c564af67720a0a1d Author: Erick Ramirez Date: Fri Nov 4 11:33:04 2022 +1100 WEBSITE - Updated What's New page for 4.1 patch by Michael Semb Wever, Erick Ramirez; reviewed by Ekaterina Dimitrova, Berenguer Blasi, Erick Ramirez for CASSANDRA-17976 Co-authored by: Mick Semb Wever Co-authored by: Erick Ramirez commit 7278db376e8b933e5ae67ecb7c8b8c7ce5a263f0 Merge: 6f801d4232 756aeed1f4 Author: Brandon Williams Date: Thu Nov 3 08:08:49 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 756aeed1f4cf70ae2515da1914c9892b6ac08cd9 Merge: 1d1866b86d 8f06ec372e Author: Brandon Williams Date: Thu Nov 3 08:08:39 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 8f06ec372e20c55b85a083e43aa3b8e0993b648d Merge: 6152cb08db b82dca6f81 Author: Brandon Williams Date: Thu Nov 3 08:08:27 2022 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit b82dca6f81f67994af2e62f4cfd74a74ddbb45ca Merge: 7df905a429 e0331f631e Author: Brandon Williams Date: Thu Nov 3 08:07:57 2022 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit e0331f631e419d82977d94b561cac2411860f279 Merge: 116ce3bc05 bba519501c Author: Brandon Williams Date: Thu Nov 3 08:07:29 2022 -0500 Merge branch 'cassandra-2.2' into cassandra-3.0 commit bba519501cbc7f47577fd7ba4d08969ff8e1d90c Author: Brandon Williams Date: Wed Aug 31 13:30:31 2022 -0500 add python2-dev to build-deps Patch by brandonwilliams; reviewed by edimitrova for CASSANDRA-17871 commit 6f801d42324e24cc6a57274b57705e724501eb98 Merge: 0bde8a2e20 1d1866b86d Author: Andrés de la Peña Date: Thu Nov 3 11:46:20 2022 +0000 Merge branch 'cassandra-4.1' into trunk commit 1d1866b86d524bcaf10cd9d5f547c9ca57a184c6 Merge: b7ef579703 6152cb08db Author: Andrés de la Peña Date: Thu Nov 3 11:43:48 2022 +0000 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 6152cb08dba770f46496d84cde067c6776c66fa0 Merge: 0db0b8f781 7df905a429 Author: Andrés de la Peña Date: Thu Nov 3 11:41:39 2022 +0000 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 7df905a4293af692b793b9364f2011f1c0a3a0d9 Merge: deede44af5 116ce3bc05 Author: Andrés de la Peña Date: Thu Nov 3 11:36:22 2022 +0000 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 116ce3bc0562f69e698f569db0904debd9f87cd0 Author: Andrés de la Peña Date: Thu Nov 3 11:30:38 2022 +0000 CircleCI: Add jobs for missing specialized unit tests Fix run_repeated_utests for non-testsome targets patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova and Berenguer Blasi for CASSANDRA-17987 commit 0bde8a2e20ebf66d11dc91c7de09cd66a3e603f7 Author: Josh McKenzie Date: Wed Nov 2 13:19:14 2022 -0400 Fix BootstrapTest failure on vnodes Patch by Josh McKenzie; reviewed by David Capwell for CASSANDRA-17679 commit 9603d03938b72b66dcb4f3447751b01432f43e11 Merge: 9a0dd603b1 b7ef579703 Author: Mick Semb Wever Date: Wed Nov 2 18:33:07 2022 +0100 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Update NetBeans project file for dependency changes since 7th July 2021 commit b7ef5797036d2e854d29b2e54c9723b5c0b1cbf8 Merge: 3c60323ca9 0db0b8f781 Author: Mick Semb Wever Date: Wed Nov 2 18:32:18 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Update NetBeans project file for dependency changes since 7th July 2021 commit 0db0b8f781ee90921b0d60d2fc41a289e6d4d389 Author: Mick Semb Wever Date: Tue Nov 1 20:25:01 2022 +0100 Update NetBeans project file for dependency changes since 7th July 2021 patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova for CASSANDRA-18002 commit 9a0dd603b18ab5034df27443a4bbc10111edcd8b Author: Natnael Adere Date: Tue Nov 1 10:45:02 2022 -0700 Log JVM Arguments at in-JVM Test Class Initialization patch by Natnael Adere; reviewed by Caleb Rackliffe, David Capwell for CASSANDRA-16664 commit 8ec04361b9e098430023e4776baf1941be958475 Author: Leonard Ma Date: Fri Oct 28 13:33:20 2022 -0700 track and handle errors during nodetool bootstrap resume properly patch by Leonard Ma; reviewed by Caleb Rackliffe and Aleksey Yeschenko for CASSANDRA-16491 commit 39a470235af13837a1a022ab0a1b6f8f062bcf6a Author: Josh McKenzie Date: Tue Sep 20 15:22:51 2022 -0400 Disable resumable bootstrap by default Patch by Marcus Eriksson; reviewed by Jordan West, Blake Eggleston, and Josh McKenzie for CASSANDRA-17679 Co-authored-by: Marcus Eriksson Co-authored-by: Josh McKenzie commit ac8d9c1c2ad20badf8eea10121f8d29f7b78b95a Merge: d45c4cbcbf 3c60323ca9 Author: Andrés de la Peña Date: Fri Oct 28 13:55:08 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 3c60323ca9228955113ef40431f4e796e928a64d Merge: 0e21007b4f 842cbe28a6 Author: Andrés de la Peña Date: Fri Oct 28 13:54:53 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 842cbe28a6d1cee00fefcf73fcdd7330f07e78ea Merge: 35ef5b9957 deede44af5 Author: Andrés de la Peña Date: Fri Oct 28 13:54:40 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit deede44af5b0d8d6ec19ecf1e613544262d85144 Merge: ed07f7d6bc ccc0b14d56 Author: Andrés de la Peña Date: Fri Oct 28 13:54:23 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit ccc0b14d56d52221c64de8d029950fa265791eb3 Author: Andrés de la Peña Date: Fri Oct 28 13:54:01 2022 +0100 CircleCI: Fix generate.sh without options modifying config patch by Andrés de la Peña; reviewed by Brandon Williams and Berenguer Blasi for CASSANDRA-17995 commit d45c4cbcbf96afba8d9a2ad4550b8d47db86f648 Merge: 507db24aad 0e21007b4f Author: Stefan Miklosovic Date: Thu Oct 27 16:07:59 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 0e21007b4f3f90c5502e5adf7bcce01c1d01c62c Merge: dd53a609c6 35ef5b9957 Author: Stefan Miklosovic Date: Thu Oct 27 15:59:31 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 35ef5b99577ef8b04b8d4b326154775f510ade42 Author: Stefan Miklosovic Date: Mon Oct 24 22:11:37 2022 +0200 Fix possible race condition on repair snapshots patch by Stefan Miklosovic; reviewed by David Capwell for CASSANDRA-17955 commit 507db24aadd614c105d2ff916ffb3878ebb4f890 Merge: 230fe8e647 dd53a609c6 Author: Stefan Miklosovic Date: Thu Oct 27 11:19:47 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit dd53a609c692c12ffb94acf9fcf81343e35723e9 Author: Maxim Chanturiay Date: Thu Oct 13 16:37:24 2022 +0300 Add --resolve-ip option to nodetool gossipinfo patch by Maxim Chanturiay; reviewed by Stefan Miklosovic and Caleb Rackliffe for CASSANDRA-17934 commit 230fe8e64722ac02dbf8cdafb7d4fef120726dd7 Author: Abe Ratnofsky Date: Fri Feb 4 11:15:42 2022 -0800 Include Git SHA in --verbose flag for nodetool version Patch by Abe Ratnofsky; review by Brandon Williams, Caleb Rackliffe, Michael Semb Wever and Stefan Miklosovic for CASSANDRA-17753 commit afdf567d53bdd123ec10e4674372539404e55e71 Merge: 2f9ab68453 50dc696198 Author: Mick Semb Wever Date: Sun Oct 23 23:27:34 2022 +0200 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Increment versions to 3.0.29, 3.11.15, 4.0.8 commit 50dc696198f9d372f815239a997ac66b33be48fb Merge: 9c7d9a36ba ac165325bc Author: Mick Semb Wever Date: Sun Oct 23 23:27:13 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Increment versions to 3.0.29, 3.11.15, 4.0.8 commit ac165325bc281364833013dd3bf5e1916fe2ac62 Merge: 8429229b7d ed07f7d6bc Author: Mick Semb Wever Date: Sun Oct 23 23:25:42 2022 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Increment versions to 3.0.29, 3.11.15, 4.0.8 commit ed07f7d6bc3e2185997f8e0180b9643bcbf57543 Merge: 761d9f9516 a3b4119b5a Author: Mick Semb Wever Date: Sun Oct 23 23:19:59 2022 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Increment versions to 3.0.29, 3.11.15, 4.0.8 commit a3b4119b5a8a90e53d4075371e98671e69783495 Author: Mick Semb Wever Date: Sun Oct 23 23:18:45 2022 +0200 Increment versions to 3.0.29, 3.11.15, 4.0.8 commit 2f9ab6845311bf02344c9bac132e0df816357e19 Author: Ekaterina Dimitrova Date: Fri Sep 23 15:06:13 2022 -0400 Update Byteman to 4.0.20 and Jacoco to 0.8.8 patch by Ekaterina Dimitrova; reviewed by Jacek Lewandowski and Michael Semb Wever for CASSANDRA-16413 commit c71f04bd5b19994e2f2c52a67f2ea1d139fcbfd3 Author: Stefan Miklosovic Date: Fri Oct 21 16:01:36 2022 +0200 Add memtable option among possible tab completions for a table patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-17982 commit 9074ee7ef8e041e1b15116373be0df80b985e3d9 Author: Branimir Lambov Date: Thu Oct 20 12:32:07 2022 +0300 Renames MemtableTrie to InMemoryTrie patch by Branimir Lambov; reviewed by Andres de la Peña and Caleb Rackliffe for CASSANDRA-17240 commit 7c55c73825e341315e520381968338d57afbb67a Author: Branimir Lambov Date: Wed Jan 20 15:42:36 2021 +0200 Adds a trie-based memtable implementation patch by Branimir Lambov; reviewed by Jason Rutherglen, Jacek Lewandowski, Andres de la Peña and Caleb Rackliffe for CASSANDRA-17240 commit 562cb26010659830dd1192939ac815a0f6cb3502 Author: Branimir Lambov Date: Thu Nov 11 15:39:21 2021 +0200 MemtableTrie using multiple buffers The replaces the size doubling and copying required to grow the trie with an allocation of a new buffer. This improves the cost of expansion at the expense of increasing individual read and write costs. patch by Branimir Lambov; reviewed by Jason Rutherglen, Jacek Lewandowski, Andres de la Peña and Caleb Rackliffe for CASSANDRA-17240 commit 30641ea7b6b8253651562aeb0102778a0f9a405b Author: Branimir Lambov Date: Mon Jan 11 16:02:12 2021 +0200 Provides the Trie interface with MemtableTrie implementation also includes functionality to merge, intersect and iterate on tries. patch by Branimir Lambov; reviewed by Jason Rutherglen, Jacek Lewandowski, Andres de la Peña and Calib Rackliffe for CASSANDRA-17240 commit 49e0c61107005b1a83799f7f1e6c0a855d159c29 Author: Branimir Lambov Date: Tue Nov 9 13:59:48 2021 +0200 Further improves precision of memtable heap tracking Partition key ByteBuffer and columns btree were not taken into account and some ByteBuffers were not measured correctly. Also fixes flakes in MemtableSizeTest caused by including allocator pool in measurements and updates it to test all memtable allocation types. patch by Branimir Lambov; reviewed by Ekaterina Dimitrova, Andres de la Peña and Caleb Rackliffe for CASSANDRA-17240 commit 5f3e60cfd44700aab4643366f99a251b4502e1b0 Merge: 6f709ca0ec 9c7d9a36ba Author: Stefan Miklosovic Date: Fri Oct 21 10:09:02 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 9c7d9a36bae7a7eaf7ceff25091a95a2deb139ac Merge: 1df5dd671e 8429229b7d Author: Stefan Miklosovic Date: Fri Oct 21 10:07:33 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 8429229b7d55182f2a6450dad2a48b3bfadd74f8 Merge: ce1584ccc8 761d9f9516 Author: Stefan Miklosovic Date: Fri Oct 21 10:05:47 2022 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 761d9f9516799a3aba1a42da7f14ab98dc7db66f Merge: c7d665dcc7 2b31675fbd Author: Stefan Miklosovic Date: Fri Oct 21 10:04:28 2022 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 2b31675fbd2fa217d864f314536dc15eea082638 Author: Stefan Miklosovic Date: Wed Oct 19 13:53:36 2022 +0200 Fix running Ant rat targets without git patch by Stefan Miklosovic; reviewed by Michael Semb Wever for CASSANDRA-17974 commit 6f709ca0ece43209f04c56c13c138ef76e14ba90 Merge: ac68f24512 1df5dd671e Author: Ekaterina Dimitrova Date: Thu Oct 20 14:00:06 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 1df5dd671e0b66c29c35d2af14596b2415d0c691 Merge: f90148a344 ce1584ccc8 Author: Ekaterina Dimitrova Date: Thu Oct 20 13:53:47 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ce1584ccc8ff65dc7e600f13f31b553732a1f12e Author: Ekaterina Dimitrova Date: Thu Sep 1 17:11:40 2022 -0400 Fix ASM bytecode version inconsistency patch by Ekaterina Dimitrova; reviewed by Berenguer Blasi and Benedict Elliot Smith for CASSANDRA-17873 commit ac68f24512dc123a197cf49e9116afcfa3f786f9 Author: Stefan Miklosovic Date: Wed Oct 19 21:54:37 2022 +0200 Call only LoggingSupportFactory in Instance instead ThreadAwareSecurityManager to fix SSTableIdGenerationTest This relates especially to CASSANDRA-17948. In CassandraDaemon, this is called via ThreadAwareSecurityManager.install() but we do not need that whole body of the method to be executed because it also installs SecurityManager which would clash with SecurityManager SSTableIdGenerationTest installs in beforeClass method so it would fail testRestartWithUUIDDisabled. patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-17975 commit 83b6f04496afb70f033685239be0dd4838f8aa77 Author: Jérôme Barotin Date: Fri Sep 23 14:26:04 2022 +0200 Fix formatting of metrics documentation patch by Jérôme Barotin; reviewed by Stefan Miklosovic for CASSANDRA-17961 commit 7e6799a4b903b4cdc1a467ada69f2f12a6099fcb Author: Marcus Eriksson Date: Tue Oct 18 08:26:21 2022 +0200 Keep sstable level when streaming for decommission and move Patch by marcuse; reviewed by David Capwell for CASSANDRA-17969 commit 89e01ea2eba6f8729a9eed2825a37e3ce691e9b8 Author: Natnael Adere Date: Wed Oct 19 11:01:07 2022 -0700 Add Unavailables metric for CASWrite in the docs patch by Natnael Adere; reviewed by David Capwell for CASSANDRA-16357 commit b69ac3ffcc2a4736d6eeb28535f14f657222624e Merge: cf15ac80a2 f90148a344 Author: Andrés de la Peña Date: Wed Oct 19 12:51:24 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit f90148a344737102994289fa2828939c11ad829b Merge: 046bedaf55 a9df34c62a Author: Andrés de la Peña Date: Wed Oct 19 12:50:21 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit a9df34c62a3596a9fc40de200adffe10b960146a Merge: 3251b85def c7d665dcc7 Author: Andrés de la Peña Date: Wed Oct 19 12:49:45 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit c7d665dcc7c06120c6c149a7b9fda27b8ea13b67 Merge: d465ac2dc9 2b511c5714 Author: Andrés de la Peña Date: Wed Oct 19 12:48:49 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 2b511c57143063553fe1256679d5ce8aefc50129 Author: Andrés de la Peña Date: Wed Oct 19 12:47:59 2022 +0100 ninja-fix: Fix bashisms on .circleci/generate.sh patch by Andrés de la Peña; reviewed by Brandon Williams for CASSANDRA-17939 commit cf15ac80a2c2ee9874b70659556b7dbaf91eea9c Merge: b0ae3de1c9 046bedaf55 Author: Mick Semb Wever Date: Wed Oct 19 11:33:24 2022 +0200 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Prepare debian changelog for 4.0.7 Prepare debian changelog for 3.11.14 Prepare debian changelog for 3.0.28 commit 046bedaf55b0375d3ecf5248bd2b7f696b4e598c Merge: fbb3079144 3251b85def Author: Mick Semb Wever Date: Wed Oct 19 11:33:13 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Prepare debian changelog for 4.0.7 Prepare debian changelog for 3.11.14 Prepare debian changelog for 3.0.28 commit 3251b85def05d963342e9799e046dd4a833ba37c Merge: 277fa4fca4 d465ac2dc9 Author: Mick Semb Wever Date: Wed Oct 19 11:32:42 2022 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Prepare debian changelog for 3.11.14 Prepare debian changelog for 3.0.28 commit d465ac2dc93f494b9585d7ef6789278c6c987243 Merge: 9d3327ef13 96c5332ee1 Author: Mick Semb Wever Date: Wed Oct 19 11:32:23 2022 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Prepare debian changelog for 3.0.28 commit 277fa4fca4a80eb327be6559f993c91e42dd4009 Author: Mick Semb Wever Date: Wed Oct 19 10:57:20 2022 +0200 Prepare debian changelog for 4.0.7 commit 9d3327ef1321fe1bf4e7fc73ed6111da7c994553 Author: Mick Semb Wever Date: Wed Oct 19 10:27:41 2022 +0200 Prepare debian changelog for 3.11.14 commit 96c5332ee15f45ca5410caaa787cc88d6947b3c9 Author: Mick Semb Wever Date: Wed Oct 19 09:59:09 2022 +0200 Prepare debian changelog for 3.0.28 commit b0ae3de1c91fbdb8bb50a129d0114ea3b9fea2df Merge: a62f618b08 fbb3079144 Author: Mick Semb Wever Date: Tue Oct 18 22:56:53 2022 +0200 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Fix JMX security vulnerabilities commit fbb3079144851e704a7912b8630f72c9345e0bb5 Merge: 0aa4ef1a8e 9a24fa81e5 Author: Mick Semb Wever Date: Tue Oct 18 22:54:11 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Fix JMX security vulnerabilities commit 9a24fa81e5f44476c9994b9d9a7aabf9b89f3f8b Merge: a124f099e0 95d0a936f9 Author: Mick Semb Wever Date: Tue Oct 18 22:53:05 2022 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Fix JMX security vulnerabilities commit 95d0a936f9611e6c43b79f6e4ba470c27d384567 Merge: 6b0cd7db04 b2660bcf78 Author: Mick Semb Wever Date: Tue Oct 18 22:49:07 2022 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Fix JMX security vulnerabilities commit b2660bcf78ccf08a3a0ae39a8c9ffb397efef9ff Author: Abe Ratnofsky Date: Tue Aug 30 15:23:26 2022 -0700 Fix JMX security vulnerabilities patch by Abe Ratnofsky; reviewed by Jon Meredith, Mick Semb Wever, Sam Tunnicliffe for CASSANDRA-17921 commit a62f618b08fd2aa2f07b23a64e89bbe7165ea243 Merge: 0ab2aa25d4 0aa4ef1a8e Author: Brandon Williams Date: Tue Oct 18 11:19:38 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 0aa4ef1a8e57336fbe5b4ac8584ab0804b0215c6 Merge: 36e73bdf55 a124f099e0 Author: Brandon Williams Date: Tue Oct 18 11:18:32 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit a124f099e036a83b7ef2b69975b122dc83ba3e5f Merge: cc622a20e1 6b0cd7db04 Author: Brandon Williams Date: Tue Oct 18 11:16:20 2022 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 6b0cd7db043095904c500660a36ae27826f693a7 Merge: f13cdddcad 3d4e6944ef Author: Brandon Williams Date: Tue Oct 18 11:08:02 2022 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 3d4e6944ef11d96858507779b9280f5b9c198ffa Author: Brandon Williams Date: Mon Oct 17 10:49:22 2022 -0500 Suppress CVE-2019-2684 Patch by brandonwilliams; reviewed by smiklosovic for CASSANDRA-17965 commit 0ab2aa25d46e743c9a1f5a325bcf96541a3f0e5b Merge: 81f9068b11 36e73bdf55 Author: Andrés de la Peña Date: Tue Oct 18 16:55:07 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 36e73bdf55e4a5e088e91eecea8896c1b47915cf Merge: 1e18eda3d9 cc622a20e1 Author: Andrés de la Peña Date: Tue Oct 18 16:53:48 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit cc622a20e1908d94626217b85737948b1df9b96a Merge: 4157e7a8e0 f13cdddcad Author: Andrés de la Peña Date: Tue Oct 18 16:51:29 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit f13cdddcad17a4c674c3011e7fff714d4f89f683 Merge: 2e6528542b 9aa28a81ec Author: Andrés de la Peña Date: Tue Oct 18 16:47:08 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 9aa28a81ecb65d8e586039f419ca3272c1de64c2 Author: Andrés de la Peña Date: Sat Oct 15 15:38:37 2022 +0100 CircleCI: Automatically detect and repeat new or modified JUnit tests patch by Andrés de la Peña; reviewed by Josh McKenzie for CASSANDRA-17939 commit 81f9068b11a3a51235d6dc14616eb805385df388 Merge: c089818881 1e18eda3d9 Author: Brandon Williams Date: Tue Oct 18 10:38:54 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 1e18eda3d9b3f6d390a19f5b0d57185018427af1 Merge: 15b54687a5 4157e7a8e0 Author: Brandon Williams Date: Tue Oct 18 10:37:05 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 4157e7a8e04655af4553d9003b9cb46897dddc2c Merge: 488c0c75a8 2e6528542b Author: Brandon Williams Date: Tue Oct 18 10:31:37 2022 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 2e6528542b21a5d79eeba6d22ddc2a289805f98c Author: Brandon Williams Date: Mon Oct 17 10:12:08 2022 -0500 Suppress CVE-2022-42003 and CVE-2022-42004 Patch by brandonwilliams; reviewed by smiklosovic for CASSANDRA-17966 commit c08981888181392017761606f4490cca7f977db9 Author: Stefan Miklosovic Date: Wed Oct 5 17:02:34 2022 +0200 Make Cassandra logs able to be viewed in the virtual table system_views.system_logs patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-17948 commit c5f8e63f67d105f483c56dd4f09b5ee6e46a4241 Merge: 04cfb57dde 15b54687a5 Author: Stefan Miklosovic Date: Tue Oct 11 23:21:25 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 15b54687a53c930e1526ebd5b6d362f1b9197baf Merge: 3fc9bc033a 488c0c75a8 Author: Stefan Miklosovic Date: Tue Oct 11 23:14:29 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 488c0c75a8f632f2db4e3db39f2ebcf8a489971e Author: Stefan Miklosovic Date: Tue Oct 11 23:13:13 2022 +0200 Remove empty cq4 log files to prevent BinLog from failing to start This patch also backports CASSANDRA-17595. patch by Stefan Miklosovic; reviewed by Caleb Rackliffe for CASSANDRA-17933 commit 04cfb57ddee573eaea00ce8c7e3d99a2f84ca78f Merge: af9439bcd7 3fc9bc033a Author: Aleksey Yeschenko Date: Tue Oct 11 15:47:43 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 3fc9bc033a523e27b3e8208c82677d6527a13cb6 Merge: 9524c22990 e13356d75d Author: Aleksey Yeschenko Date: Tue Oct 11 15:43:49 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit e13356d75d2d3c200f1636337cf15329bd1b829b Author: Piotr Kołaczkowski Date: Fri May 6 09:39:21 2022 +0200 Fix multiple BufferPool bugs patch by Piotr Kołaczkowski; reviewed by Aleksey Yeschenko for CASSANDRA-16681 commit af9439bcd781756a0d845215c4c03c5fd8e74b86 Author: David Capwell Date: Mon Oct 10 15:39:52 2022 -0700 IllegalArgumentException in Gossiper#order due to concurrent mutations to elements being applied patch by David Capwell; reviewed by Blake Eggleston for CASSANDRA-17908 commit 4fc2d9e53985dc89b93bbac80bea9faa4a3d708b Author: Josh McKenzie Date: Fri Sep 23 14:04:42 2022 -0400 Include estimated active compaction remaining write size when starting a new compaction Patch by Marcus Eriksson; reviewed by Chris Lohfink, Stefan Podkowinski, Caleb Rackliffe,and Josh McKenzie for CASSANDRA-17931 Co-authored-by: Marcus Eriksson Co-authored-by: Josh McKenzie commit ca75ffe4d09a3e7b26a56345c0bdacaa284eaab7 Author: Jyothsna Konisa Date: Fri Oct 7 10:03:16 2022 -0700 Mixed mode support for internode authentication during TLS upgrades patch by Jyothsna Konisa; reviewed by Jon Meredith, Yifan Cai for CASSANDRA-17923 commit d62d845c7d86b6ca5ea4d63042123d1b3802ae5e Merge: 472dc30faa 9524c22990 Author: Ariel Weisberg Date: Thu Oct 6 14:11:50 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 9524c22990de62d42c7909ff4e2635e238e7ee48 Merge: fb4974d455 3bdd2caa22 Author: Ariel Weisberg Date: Thu Oct 6 14:11:00 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 472dc30faa7e23990d7fbd585cf8409659926004 Merge: ba1a3fb8ae fb4974d455 Author: Aleksey Yeschenko Date: Thu Oct 6 17:04:20 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit fb4974d455e2843a75ce00c8f639c70acecb489c Merge: 0b083d3e73 0c4daa1ddc Author: Aleksey Yeschenko Date: Thu Oct 6 16:10:24 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ba1a3fb8ae92f51965d4771738d55a2163490c70 Merge: ace5662143 0b083d3e73 Author: Aleksey Yeschenko Date: Thu Oct 6 15:36:57 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 0b083d3e737de166e02371cc90df18c94ba820a8 Merge: c4bccb000a 4e1d31e729 Author: Aleksey Yeschenko Date: Thu Oct 6 15:29:01 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ace5662143fbe0dabd7f65711d9057d5a0469000 Merge: 3d0bf61d43 c4bccb000a Author: Mick Semb Wever Date: Wed Oct 5 17:17:56 2022 -0500 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Increment version to 4.1-beta2 commit c4bccb000adec62dd8e42d0b18e4a7adef00762f Author: Mick Semb Wever Date: Wed Oct 5 17:14:37 2022 -0500 Increment version to 4.1-beta2 commit 3d0bf61d435b6bdcbf4728f289162afbcaf6c300 Merge: c4c4ad5cbc 8419fb21e0 Author: Caleb Rackliffe Date: Wed Oct 5 14:13:11 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 8419fb21e016389f15be0954e8721cf4f6918249 Author: Caleb Rackliffe Date: Tue Oct 4 17:44:34 2022 -0500 Allow pre-V5 global limit on bytes in flight to revert to zero asynchronously in RateLimitingTest patch by Caleb Rackliffe; reviewed by Brandon Williams for CASSANDRA-17927 commit c4c4ad5cbc0fc263c783ab0c937214844c77f348 Merge: 719d1948df f0fd551c7c Author: Stefan Miklosovic Date: Tue Oct 4 20:21:30 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 719d1948df827e864ff66e44e22c7fad334c3100 Author: Ekaterina Dimitrova Date: Mon Oct 3 16:37:00 2022 -0400 Revert Mockito downgrade from CASSANDRA-17750 patch by Ekaterina Dimitrova; reviewed by Michael Semb Wever and Abe Ratnofsky for CASSANDRA-17946 commit f0fd551c7ce32a52e8bfa9a1f40b3ae3d56468a3 Merge: 00ad7f2ed9 e9b411e3e0 Author: Stefan Miklosovic Date: Mon Oct 3 16:26:14 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 35578a4a9f6a5614a388939354fe68d03dc20459 Merge: 872e34c2d6 00ad7f2ed9 Author: Benjamin Lerer Date: Wed Sep 28 15:57:25 2022 +0200 Merge branch cassandra-4.1 into trunk commit 00ad7f2ed9dfbff07b51703f9e2e6ee0278839bd Merge: 5d9d93ea08 46d5ef4148 Author: Benjamin Lerer Date: Wed Sep 28 15:02:27 2022 +0200 Merge branch cassandra-4.0 into cassandra-4.1 commit 872e34c2d613a43b2c2be75efa016b6943bf87ad Author: Stefan Miklosovic Date: Mon Aug 15 13:56:58 2022 +0200 add --older-than and --older-than-timestamp options for nodetool clearsnapshot patch by Stefan Miklosovic; reviewed by Paulo Motta for CASSANDRA-16860 commit 526f41899a7273d262ff3bd86047e6d0d6f85d91 Merge: 26ba82f6a2 5d9d93ea08 Author: Mick Semb Wever Date: Tue Sep 27 00:52:57 2022 +0200 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: Prepare debian changelog for 4.1-beta1 commit 5d9d93ea08d9c76402aa1d14bad54bf9ec875686 Author: Mick Semb Wever Date: Tue Sep 27 00:08:43 2022 +0200 Prepare debian changelog for 4.1-beta1 commit 26ba82f6a2820ddeaff9eb1aae4c95d5aa521197 Merge: ff47de11d6 65c9875b39 Author: Mick Semb Wever Date: Mon Sep 26 23:45:18 2022 +0200 Merge branch 'cassandra-4.1' into trunk * cassandra-4.1: increment version to 4.1-beta1 commit 65c9875b3964d6f7bd533c6a3a55ce371a1d69bf Author: Mick Semb Wever Date: Mon Sep 26 20:55:03 2022 +0200 increment version to 4.1-beta1 ref: https://lists.apache.org/thread/ho4r1lk2gsgs1z61ldzg64h84x3rbg4o commit ff47de11d6238ac78b796f565904c600b7a20e92 Merge: f4b69ba0e8 5d3c747719 Author: Stefan Miklosovic Date: Mon Sep 26 22:25:18 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 5d3c747719f01d87c9086c405c806317405a8e43 Author: Stefan Miklosovic Date: Thu Sep 22 10:20:31 2022 +0200 add checkstyle modules for checking redundant and unused imports in Java code patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-17876 Co-authored-by: Ling Mao commit f4b69ba0e82bb051e56a92d792142034d9f617f0 Author: Josh McKenzie Date: Mon Sep 19 14:49:10 2022 -0400 Fix "open RT bound as its last item" exception Patch by Marcus Eriksson; reviewed by Aleksey Yeschenko and Josh McKenzie for CASSANDRA-17810 Co-authored-by: Marcus Eriksson Co-authored-by: Josh McKenzie commit e5c9cf4cbe6634ba9e148b6e27b6bad98648e597 Author: Leonard Ma Date: Fri Sep 2 17:54:04 2022 -0700 Handle leak of non-standard Java types as clients using JMX cannot handle them -deprecate and replace JMX setters that throw non-standard exceptions -deprecate and replace respective JMX getters as well to make JMX usage consistent patch by Leonard Ma; reviewed by Ekaterina Dimitrova and David Capwell for CASSANDRA-17668 commit 7adfdc85cf5c1cb505e6507e5c08de84834c2224 Author: Claude Warren, Jr Date: Fri Jul 22 08:56:13 2022 +0100 Deprecate Throwables.propagate usage patch by Claude Warren; reviewed by Caleb Rackliffe and David Capwell for CASSANDRA-14218 commit ecceb446e00ea9e567ba45f1b422cb04862ef044 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Wed Sep 14 14:28:10 2022 +0300 Refactor internals of cqlsh.py to cqlshlib patch by Brad Schoening; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17531 commit 0db158d3bda5d26998cf7e5d45cb11ed898f3f2a Merge: 85f113bce3 0f9ddb8b6c Author: Brandon Williams Date: Wed Sep 21 05:53:19 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 0f9ddb8b6ce40056508be9fdc226cf7a7cda9091 Merge: 286d2ee053 756fb41512 Author: Brandon Williams Date: Wed Sep 21 05:49:54 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 85f113bce38278653ffca5139cd7871aee7fe786 Author: Claude Warren, Jr Date: Fri Aug 19 07:42:25 2022 +0100 Add pull request template and modify README to include Jira and mailing list link patch by claudenw; reviewed by dritfx, dcapwell, josh-mckenzie, michaelsembwever and smiklosovic for CASSANDRA-17837 commit 73c0f7f2af061e8b7e79566e986769a06b06412a Merge: 38a356b10c 286d2ee053 Author: Blake Eggleston Date: Tue Sep 20 12:23:48 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit 286d2ee053ac2e6fe50749fa2833533254697662 Author: Blake Eggleston Date: Fri Sep 16 14:26:41 2022 -0700 Rework flaky CASTests Patch by Blake Eggleston; Reviewed by Benedict Elliott Smith for CASSANDRA-17461 commit 38a356b10cb242d219894f9954684b8bf62fb2ac Merge: 0aaad92f48 3608f67076 Author: Josh McKenzie Date: Tue Sep 20 14:19:49 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 3608f6707606bb940a3545568d5df634241c12e1 Merge: 1f90c9eb3d 22ec7eee7f Author: Josh McKenzie Date: Tue Sep 20 14:18:09 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 0aaad92f48e8f5d0b5d1ed85c20edfac7de3f4d4 Merge: d80d934ed2 1f90c9eb3d Author: Josh McKenzie Date: Tue Sep 20 12:59:59 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 1f90c9eb3d2063771f402450b6886f26afd81222 Merge: 4c85c6a403 29c0ad80c9 Author: Josh McKenzie Date: Tue Sep 20 12:57:51 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit d80d934ed233341d7fec12df8d98c37a6465ce62 Merge: e89b214d06 4c85c6a403 Author: Ekaterina Dimitrova Date: Mon Sep 19 18:29:43 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 4c85c6a4030d7d613230cd4b2204c42f941fd552 Author: Ekaterina Dimitrova Date: Mon Sep 19 10:28:54 2022 -0400 key_cache_save_period, row_cache_save_period, counter_cache_save_period are not deprecated and we should not emit deprecation warning on startup for them Patch by Ekaterina Dimitrova; reviewed by Stefan Miklosovic for CASSANDRA-17904 commit e89b214d069321c8968871b8eb7d51d4dfba7c33 Author: Josh McKenzie Date: Tue Sep 13 12:48:00 2022 -0400 Allow disabling hotness persistence for high sstable counts Patch by Caleb Rackliffe; reviewed by Chris Lohfink and Josh McKenzie for CASSANDRA-17868 Co-authored-by: Caleb Rackliffe Co-authored-by: Josh McKenzie commit ff5f4833aa3e11fcffd6bff1b15597fd5a38b864 Author: Josh McKenzie Date: Thu Sep 15 13:56:00 2022 -0400 Prevent NullPointerException when changing neverPurgeTombstones from true to false Patch by Marcus Eriksson; reviewed by Caleb Rackliffe and Josh McKenzie for CASSANDRA-17897 Co-authored-by: Marcus Eriksson Co-authored-by: Josh McKenzie commit bd2ecb3454649d8c84cf0b1ce8c1e94ed1e06e74 Author: Josh McKenzie Date: Thu Sep 15 14:22:34 2022 -0400 Add metrics around storage usage and compression Patch by Caleb Rackliffe; reviewed by Abe Ratnofsky and Josh McKenzie for CASSANDRA-17898 Co-authored-by: Caleb Rackliffe Co-authored-by: Josh McKenzie commit 56ea39ec704a94b5d23cbe530548745ab2420cee Merge: f77e6cd3a0 d8bbeb9e39 Author: Jacek Lewandowski Date: Fri Sep 16 09:56:02 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit d8bbeb9e398723e5efe13166e16372d104fb93ce Merge: 67e9f7bdc4 76be530a36 Author: Jacek Lewandowski Date: Fri Sep 16 09:51:04 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit f77e6cd3a020f3ad777c6bd36973ca084a978f1f Author: Josh McKenzie Date: Thu Sep 8 13:09:57 2022 -0400 Remove usage of deprecated javax certificate class Patch by Doug Rohrer; reviewed by Jyothsna Konisa, Francisco Guerrero Hernandez, Josh McKenzie, and Caleb Rackliffe for CASSANDRA-17867 Co-authored-by: Doug Rohrer Co-authored-by: Josh McKenzie commit e73b7ba2c03b0fca0c4f4312af2a0a93c88fa988 Merge: 9355e70eaf 67e9f7bdc4 Author: Josh McKenzie Date: Mon Sep 12 13:07:10 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 67e9f7bdc4859654f2946d5dfe1772ef1cbd7ec5 Merge: 3f46c7d88b f65f123780 Author: Josh McKenzie Date: Mon Sep 12 13:04:40 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 # Conflicts: # test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeMessageForwardTest.java commit 9355e70eaf623614a7a99cc1a81ed9a10b2d4f40 Merge: 51944c5e68 3f46c7d88b Author: Josh McKenzie Date: Mon Sep 12 12:47:29 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 3f46c7d88ba3bb53d35a778ee41a3d1354d437f9 Merge: 49cd4b5f79 ac25d6929e Author: Josh McKenzie Date: Mon Sep 12 12:45:42 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 # Conflicts: # CHANGES.txt commit 51944c5e68bfcee0c2c8e2aeb6b572eae0167965 Author: Josh McKenzie Date: Wed Sep 7 13:12:22 2022 -0400 Make sure preview repairs don't optimise streams unless configured to Patch by Chris Lohfink; reviewed by Josh McKenzie and Marcus Eriksson for CASSANDRA-17865 Co-authored-by: Chris Lohfink Co-authored-by: Josh McKenzie commit 703ccdee29f7e8c39aeb976e72e516415d609cf4 Merge: d6aee7e08c 49cd4b5f79 Author: Benjamin Lerer Date: Mon Sep 12 14:58:17 2022 +0200 Merge branch cassandra-4.1 into trunk commit 49cd4b5f79e4cda16668cce8aea7d93d480f48fc Author: Benedict Elliott Smith Date: Tue May 31 13:54:56 2022 +0100 Fix PaxosRepairTest#paxosRepairVersionGate Patch by Benedict Elliott Smith; review by Blake Eggleston for CASSANDRA-17573 commit d6aee7e08c658db9d394a6b7e3e27791b4d6854f Author: Caleb Rackliffe Date: Thu Sep 1 11:47:22 2022 -0500 Optionally avoid hint transfer during decommission patch by Caleb Rackliffe; reviewed by Jon Meredith and Stefan Miklosovic for CASSANDRA-17808 commit b6d8e2ce6bac816279ce0490fd6292aa7e4124b6 Author: Stefan Miklosovic Date: Wed Sep 7 16:23:01 2022 +0200 Make disabling auto snapshot on selected tables possible patch by Tommy Stendahl; reviewed by Stefan Miklosovic and Aleksey Yeschenko for CASSANDRA-10383 Co-authored-by: Tommy Stendahl Co-authored-by: Stefan Miklosovic commit 97a5ff9925802c81e22dc38a6bf58e8835b2c20a Merge: 377e114cb1 adb0128485 Author: Caleb Rackliffe Date: Wed Sep 7 15:52:23 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit adb01284852a6c7943f7038dcee3cae101f47d23 Merge: d25adb498a c56952efb5 Author: Caleb Rackliffe Date: Wed Sep 7 15:40:29 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 377e114cb1459895423c292cb0bf7f921fd30e43 Author: Josh McKenzie Date: Thu Aug 25 15:27:24 2022 -0400 Introduce compaction priorities to prevent upgrade compaction inability to finish Patch by Alex Petrov; reviewed by Josh McKenzie and Marcus Eriksson for CASSANDRA-17851 Co-authored-by: Alex Petrov Co-authored-by: Josh McKenzie commit 24b92f4959a2013d45920ca30cc208ba5f1c5bc6 Merge: 1b2c561e9e d25adb498a Author: David Capwell Date: Tue Sep 6 18:07:23 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit d25adb498abde240cc162cfe4a9630c01381c7f9 Author: David Capwell Date: Tue Sep 6 16:47:04 2022 -0700 upsert with adder support is not consistent with numbers and strings in LWT patch by David Capwell; reviewed by Benedict Elliott Smith, Caleb Rackliffe for CASSANDRA-17857 commit 1b2c561e9ed391c2b7d8862b724752680c32b2a6 Merge: 5fce990be4 46700918f8 Author: Brandon Williams Date: Tue Sep 6 11:55:09 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 46700918f8faec82f231aa15ff719c58e5f1ee95 Author: Brandon Williams Date: Thu Aug 25 18:18:32 2022 -0500 Catch NSE race condition and return after failing connections Patch by brandonwilliams; reviewed by adelapena and aleksey for CASSANDRA-17618 commit 5fce990be45ca30bf6c2b50571159a981216e15d Merge: 387c274a62 6748b8b7ea Author: Jon Meredith Date: Tue Sep 6 10:01:43 2022 -0600 Merge branch 'cassandra-4.1' into trunk commit 6748b8b7ead6fff7820045ebbfe23f6c050f2efe Author: Jon Meredith Date: Fri Sep 2 10:28:21 2022 -0600 Speculative execution threshold unit mismatch patch by Jon Meredith; reviewed by Caleb Rackliffe, Yifan Cai for CASSANDRA-17877 commit 387c274a624c2a85036e909af1d734d87ee00541 Merge: b39596b65c a61edad15f Author: Stefan Miklosovic Date: Fri Sep 2 15:22:45 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit a61edad15fb1e80cdad4d613b9e3d7d5a27ff747 Author: Stefan Miklosovic Date: Fri Sep 2 00:21:54 2022 +0200 remove pure_sasl from the repository This is follow-up for CASSANDRA-16456. The dependency should be downloaded and put into the lib/ folder. patch by Stefan Miklosovic; reviewed by Mick Semb Wever for CASSANDRA-16456 commit b39596b65c6bdbed5f2be35008291c69b41f1675 Merge: 1e2b608213 1ee5df02b1 Author: Andrés de la Peña Date: Wed Aug 31 12:23:33 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 1ee5df02b1f98cf38f126d47a7f3fb153f790d52 Author: Paulo Motta Date: Sun Aug 14 21:00:04 2022 -0300 Fix flakiness of testAutoSnapshotTTlOnDropAfterRestart Patch by Paulo Motta; Reviewed by Caleb Rackliffe and Andrés de la Peña for CASSANDRA-17804 commit 1e2b60821327c158cba1c11d98eea68531178893 Author: Stefan Miklosovic Date: Thu Aug 11 16:39:04 2022 +0200 Prevent a user from manually removing ephemeral snapshots patch by Stefan Miklosovic; reviewed by Paulo Motta for CASSANDRA-17757 commit 1b9c6d83ec30d701c46addf5ffbada843139b718 Merge: 756d506bd9 cd576a0d14 Author: Marcus Eriksson Date: Mon Aug 29 13:29:14 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit cd576a0d140c94007fd0181044df453c0681252f Merge: 726b67b1e4 5beab63b55 Author: Marcus Eriksson Date: Mon Aug 29 13:27:46 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 756d506bd999c4d0b9e0fce3663c34baf91140f1 Merge: ea44835bf1 726b67b1e4 Author: Jacek Lewandowski Date: Fri Aug 26 17:40:22 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 726b67b1e430c378c5506d94494c8d2cde8ea7b4 Merge: df32dd3559 2b492f17b9 Author: Jacek Lewandowski Date: Fri Aug 26 17:38:48 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ea44835bf19c97d2fc8da4e2672d8604d741893d Author: Milan Krisko Date: Tue Aug 23 22:05:46 2022 -0400 Mark antora.yml in trunk as `prelease:true` patch by Milan Krisko; reviewed by Mick Semb Wever for CASSANDRA-17823 commit d4acee0e5d4e2ba33d02c9cf347c03a001568947 Merge: 1e27ffc6ad df32dd3559 Author: Mick Semb Wever Date: Thu Aug 25 17:54:04 2022 +0200 Merge branch 'cassandra-4.1' into trunk * cassandra-4.0: Increment version to 4.0.7 commit df32dd3559bfa62c528e5187320369da7d5d6a46 Merge: 704141541a d82cb5703d Author: Mick Semb Wever Date: Thu Aug 25 17:53:40 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 * cassandra-4.0: Increment version to 4.0.7 commit 1e27ffc6aded5b54b0207b114b154cffbc64dda5 Author: Abe Ratnofsky Date: Wed Aug 24 13:23:09 2022 -0700 Remove dependency on Maven Ant Tasks patch by Abe Ratnofsky; reviewed by David Capwell, Michael Semb Wever for CASSANDRA-17750 commit 616ddf812a05c6bddf7ca176c26b33a8ba11864a Merge: dbadd08585 704141541a Author: Josh McKenzie Date: Wed Aug 24 15:24:40 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 704141541a0ff4f2a00e98b3f8c4b3d5c809c562 Merge: f4deef9d58 73bea0ac1e Author: Josh McKenzie Date: Wed Aug 24 15:24:11 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit dbadd08585e78fbfc0c2f26575fd6ce68e8d7ba2 Author: Ekaterina Dimitrova Date: Thu Aug 18 15:14:56 2022 -0400 Update ASM(9.1 to 9.3), Mockito(1.10.10 to 1.12.13) and ByteBuddy(3.2.4 to 4.7.0) patch by Ekaterina Dimitrova; reviewed by Brandon Williams for CASSANDRA-17835 commit 17810295ca3b05b011a0ff7061d27435b531ea32 Author: Stefan Miklosovic Date: Tue Aug 23 19:53:16 2022 +0200 fix StandaloneUpgraderOnSStablesTest This is follow-up for CASSANDRA-13010 where a bug was introduced in Upgrader which made StandaloneUpgraderOnSStablesTest flaky. patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-17849 commit d0b9532f2b87a17a0508d0637556f2f3e8d0fd94 Author: Josh McKenzie Date: Mon Aug 22 15:04:19 2022 -0400 Add the ability for operators to loosen the definition of "empty" for edge cases Patch by David Capwell; reviewed by Josh McKenzie, Yifan Cai, and Sam Tunnicliffe for CASSANDRA-17842 Co-authored-by: David Capwell Co-authored-by: Josh McKenzie commit 0e855c4b7c157b7ba63bb7377bc441260d76556f Author: Josh McKenzie Date: Mon Aug 22 14:28:50 2022 -0400 Fix potential out of range exception on column index downsampling Patch by Marcus Eriksson; reviewed by Josh McKenzie, Jon Meredith, and Caleb Rackliffe for CASSANDRA-17839 Co-authored-by: Marcus Eriksson Co-authored-by: Josh McKenzie commit 1c714e43e6bad82ca24e095385a24fe9b33dd4f4 Author: Josh McKenzie Date: Mon Aug 22 14:16:30 2022 -0400 Log anticompaction cancellation at INFO level Patch by Marcus Eriksson; reviewed by Caleb Rackliffe, David Capwell, and Josh McKenzie for CASSANDRA-17841 Co-authored-by: Marcus Eriksson Co-authored-by: Josh McKenzie commit 4b20ad797ca42643be77aa7d7cf7e35cf2480813 Merge: 6a6d2214e9 f4deef9d58 Author: Stefan Miklosovic Date: Mon Aug 22 19:16:13 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit f4deef9d58f739370d07cec6cc898087df560eca Author: Stefan Miklosovic Date: Sat Aug 20 15:01:01 2022 +0200 Fix test/distributed/org/apache/cassandra/distributed/test/IncRepairCoordinatorErrorTest patch by Stefan Miklosovic; reviewed by Josh McKenzie for CASSANDRA-17843 commit 6a6d2214e95d1b79d4d0068cd4b21a1a9eb8129a Merge: e7c9ac05f9 31b607c92a Author: Yifan Cai Date: Mon Aug 22 09:31:15 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit 31b607c92a98de5eeb33cadfd973c5fc76656b07 Author: Yifan Cai Date: Mon Aug 22 09:31:05 2022 -0700 ninja fix: NEWS.txt commit e7c9ac05f99cc8a5ee958169c49326e85ab4b25b Author: Stefan Miklosovic Date: Fri Aug 19 16:50:49 2022 +0200 Introduce target directory to vtable output for sstable_tasks and for compactionstats patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-13010 Co-authored-by: Alex Lourie commit c4b1c0614e42b4ea2064822d31c28aa5d4f1450a Author: David Capwell Date: Fri Aug 19 16:42:56 2022 -0700 Read/Write/Truncate throw RequestFailure in a race condition with callback timeouts, should return Timeout instead patch by David Capwell; reviewed by Caleb Rackliffe for CASSANDRA-17828 commit 4aa3bbda799f0909f37b4b1963f176d379395a5e Merge: 4526b3fcbd 83c169ec9e Author: Yifan Cai Date: Fri Aug 19 14:19:58 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit 83c169ec9e36324f27bf562951362f4a03c3c688 Author: Francisco Guerrero Date: Fri Aug 19 10:20:57 2022 -0700 Fix BulkLoader to load entireSSTableThrottle and entireSSTableInterDcThrottle patch by Francisco Guerrero; reviewed by Ekaterina Dimitrova, Yifan Cai for CASSANDRA-17677 commit 4526b3fcbde22d09065820286dd434d93ecc89ba Author: Josh McKenzie Date: Tue Aug 16 14:19:46 2022 -0400 Add ability to log load profiles at fixed intervals Patch by Yifan Cai; reviewed by Josh McKenzie, Dinesh Joshi, and Chris Lohfink for CASSANDRA-17821 Co-authored-by: Yifan Cai Co-authored-by: Josh McKenzie commit fd7b1bf81e17681567ba54b0577c788842c05bec Merge: 9f8646ed49 b95a6931f5 Author: Josh McKenzie Date: Fri Aug 19 12:36:08 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit b95a6931f5b1a4e25cce81f87dd97063c0e9dd87 Merge: 958889cfeb 0353df7542 Author: Josh McKenzie Date: Fri Aug 19 12:34:33 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 # Conflicts: # CHANGES.txt # src/java/org/apache/cassandra/repair/consistent/CoordinatorSession.java commit 9f8646ed497a0dd0f8ba3a2d49a4d87116c2ee60 Merge: 902dcba740 958889cfeb Author: Mick Semb Wever Date: Thu Aug 18 22:12:54 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 958889cfebe506246bb20d1a0a2943923c5089ca Merge: f579bddb75 eb23757184 Author: Mick Semb Wever Date: Thu Aug 18 22:11:52 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 902dcba740300797684ed3c8c355ecc84373210d Author: Josh McKenzie Date: Wed Aug 17 16:05:49 2022 -0400 Protect against Gossip backing up due to a quarantined endpoint without version information Patch by Marcus Eriksson; reviewed by Josh McKenzie and Sam Tunnicliffe for CASSANDRA-17830 Co-authored-by: Marcus Eriksson Co-authored-by: Josh McKenzie commit dd74bd4a03db02b97c0710e4b50b85dc03c041eb Merge: 7fe5f9a876 f579bddb75 Author: Brandon Williams Date: Thu Aug 18 05:57:59 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit f579bddb75eb43d7edc43e8823595494cf655b10 Merge: d432fa1e94 9ec945acf9 Author: Brandon Williams Date: Thu Aug 18 05:57:21 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 7fe5f9a876bfa9e0b58b061823cbbfb84f001c91 Merge: b70e14e2ac d432fa1e94 Author: Bereng Date: Thu Aug 18 09:36:41 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit d432fa1e94188bd878cc9ecd92b5ea52886ae027 Merge: 43036ecadd 9f97adf63c Author: Bereng Date: Thu Aug 18 09:35:08 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit b70e14e2ac89f13570413cc214bcc6f25b902013 Merge: 3f2e8d1883 43036ecadd Author: Ekaterina Dimitrova Date: Wed Aug 17 18:30:59 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 43036ecadda82dca35cc79eed0db52eec8d495ff Author: Ekaterina Dimitrova Date: Thu Aug 11 15:31:47 2022 -0400 Update docs and NEWS.txt; config docs are reachable now on the website patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe for CASSANDRA-17778 commit 3f2e8d1883c586bdb9cd7a23076ceaaeefa4bd8c Author: David Capwell Date: Wed Aug 17 09:47:17 2022 -0700 NPE in org.apache.cassandra.cql3.Attributes.getTimeToLive patch by David Capwell; reviewed by Caleb Rackliffe for CASSANDRA-17822 commit 0bf9955abf12b65325af01d2618ba0d3f4eb701c Merge: cbc127ab79 590de42a06 Author: Yifan Cai Date: Tue Aug 16 15:25:20 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit 590de42a0637e7de177d6d9a69c769085b814bfa Merge: e02923305c 84e10d0cad Author: Yifan Cai Date: Tue Aug 16 15:21:43 2022 -0700 Merge branch 'cassandra-4.0' into cassandra-4.1 commit cbc127ab79d75a87a0ded714f41218c4075d712e Merge: 4b7069c06c e02923305c Author: Mick Semb Wever Date: Tue Aug 16 21:49:06 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit e02923305cfd204e81d5290877c7174e5d6ce539 Merge: 6405a75dd8 5fafa709fb Author: Mick Semb Wever Date: Tue Aug 16 21:47:05 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 4b7069c06c7913e1f3f2d5622d8cbc04746b0bac Author: Andrés de la Peña Date: Mon Jun 13 20:08:55 2022 +0100 Add guardrail for column size patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova and David Capwell for CASSANDRA-17151 commit 7bde2c23abacf60d156de7c444f136dcfe9fe276 Author: Josh McKenzie Date: Fri Aug 12 12:43:26 2022 -0400 Replace getStderr calls with getCleanedStderr calls in tests checking for emptiness Patch by Jon Meredith; reviewed by Josh McKenzie and Berenguer Blasi for CASSANDRA-17815 Co-authored-by: Jon Meredith Co-authored-by: Josh McKenzie commit 2c4eff0006cb583dde163b321680ffbc09f5ed52 Merge: 9184dd5a99 6405a75dd8 Author: Bereng Date: Tue Aug 16 09:41:48 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 6405a75dd82768578697b91479e73894135c1c5f Merge: 71cef76faa 6591048051 Author: Bereng Date: Tue Aug 16 09:39:00 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 9184dd5a998366dc2b5c18d4954b13b033efcf80 Author: Francisco Guerrero Date: Mon Aug 15 09:23:56 2022 -0700 When doing a host replacement, we need to check that the node is a live node before failing with "Cannot replace a live node..." patch by Francisco Guerrero; reviewed by Brandon Williams, David Capwell for CASSANDRA-17805 commit 3e0b94565acc64e903d73af3a14b23c875abc5b3 Author: Josh McKenzie Date: Tue Aug 2 16:02:03 2022 -0400 Add support to generate a One-Shot heap dump on unhandled exceptions Patch by Caleb Rackliffe; reviewed by Josh McKenzie, David Capwell, and Jon Meredith for CASSANDRA-17795 Co-authored-by: Caleb Rackliffe Co-authored-by: Josh McKenzie commit 09b282d1fdd7d6d62542137003011d144c0227be Author: Josh McKenzie Date: Thu Aug 11 14:02:27 2022 -0400 Rate-limit new client connection auth setup to avoid overwhelming bcrypt Patch by Chris Lohfink; reviewed by Caleb Rackliffe, Yifan Cai, and Josh McKenzie for CASSANDRA-17812 Co-authored-by: Chris Lohfink Co-authored-by: Josh McKenzie commit 77e4eb4770ee1013e711e7f76f0c510b025b4629 Merge: 60db95cba1 71cef76faa Author: Stefan Miklosovic Date: Mon Aug 15 10:13:12 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 71cef76faa3a9ddae12342f953fd12a1076a65e7 Merge: e7bdee507f 5023a816e3 Author: Stefan Miklosovic Date: Mon Aug 15 10:11:15 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 60db95cba10d3ba0f8b1dcf377b42571cd2101ea Author: David Capwell Date: Fri Aug 12 15:20:37 2022 -0700 DataOutputBuffer#scratchBuffer can use off-heap or on-heap memory as a means to control memory allocations patch by David Capwell; reviewed by Caleb Rackliffe for CASSANDRA-16471 commit 1fa8d71da41f2d38670e9c6ee377b84cddff6f64 Merge: 2ed8c5c732 e7bdee507f Author: Brandon Williams Date: Fri Aug 12 08:49:35 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit e7bdee507f022b03813219e3f7c4b1d8aa447bd3 Merge: 78f3fd8075 6483386caa Author: Brandon Williams Date: Fri Aug 12 08:49:28 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 2ed8c5c732518bd0b24541dba879a67e359f60c7 Merge: 777c5b4f52 78f3fd8075 Author: Brandon Williams Date: Fri Aug 12 08:34:17 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 78f3fd807533d6fb79849dd42820dec52ba11b60 Author: Brandon Williams Date: Fri Aug 12 08:34:11 2022 -0500 Add changes for 17765 commit 777c5b4f527879d88dab10b22bc169c18af34f3f Merge: c8c8635a4c d9a52d4aee Author: Brandon Williams Date: Fri Aug 12 08:31:36 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit d9a52d4aee781f861a7f9364863c0b831da43de0 Merge: c1d89c32d2 3b49283ef9 Author: Brandon Williams Date: Fri Aug 12 08:28:48 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit c8c8635a4c902ef051a46845919a5430f8d71e3f Author: Andrés de la Peña Date: Fri Jun 17 11:45:43 2022 +0100 Add ability to read the TTLs and write times of the elements of a collection and/or UDT patch by Andrés de la Peña; reviewed by Yifan Cai and Benjamin Lerer for CASSANDRA-8877 commit 570732375e4186741388adb81afeab6f155f57b9 Author: Jacek Lewandowski Date: Fri Jun 10 11:43:53 2022 +0200 Fix a race condition where a keyspace can be opened while it is being removed patch by Jacek Lewandowski; reviewed by Andrés de la Peña and Ekaterina Dimitrova for CASSANDRA 17658 commit ab3b59ec42666d6fd3f92a26283e65b3d7f3aa4f Merge: 5279d7f57e c1d89c32d2 Author: Jacek Lewandowski Date: Fri Aug 12 13:33:45 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit c1d89c32d27921d1f77f05d29ee248b8922a4c76 Author: Jacek Lewandowski Date: Fri Jun 10 11:43:53 2022 +0200 Fix a race condition where a keyspace can be opened while it is being removed patch by Jacek Lewandowski; reviewed by Andrés de la Peña and Ekaterina Dimitrova for CASSANDRA 17658 commit 5279d7f57edf8760ccfb0b388b750b020dd97ac2 Merge: a45e70a860 16be1ab073 Author: Andrés de la Peña Date: Fri Aug 12 10:33:06 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 16be1ab073f6fac7df1ac838c7a89e862cf9f2e1 Merge: f5dc9a1c1e 174e0dc399 Author: Andrés de la Peña Date: Fri Aug 12 10:31:30 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit a45e70a860e9d68745f463f5a06aa7b870ff69df Merge: 9a4a677823 f5dc9a1c1e Author: Brandon Williams Date: Wed Aug 10 14:13:56 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit f5dc9a1c1e27ac5ea0cb972ceaf68d5993804d6f Author: Brandon Williams Date: Wed Aug 10 08:19:50 2022 -0500 Set default FD in DD client initialization Patch by Doug Rohrer; reviewed by brandonwilliams and maedhroz for CASSANDRA-17782 commit 9a4a67782311e4a6ffe3aad0516c33016f187c54 Author: Bernardo Botella Corbi Date: Mon Aug 1 10:57:04 2022 -0700 Removed Python < 2.7 support from formatting.py Patch by Bernardo Botella Corbi, reviewed by Brad Schoening, ycai and brandonwilliams for CASSANDRA-17694 commit afa7dfb5a48ecb56abc2d8bbb1de0fc8f1ca77b9 Merge: 72c2270a80 2bc4a36179 Author: Blake Eggleston Date: Tue Aug 9 08:59:38 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit 2bc4a3617931a1def709300cf6e220c978ad9e61 Author: Blake Eggleston Date: Tue Aug 2 10:51:15 2022 -0700 Use seeded crc for PaxosBallotTracker checksum Patch by Blake Eggleston; Reviewed by Benedict Elliot Smith for CASSANDRA-17793 commit 72c2270a80f2acc8ece3eade4d6a2f8e8cb12356 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Wed Jul 27 22:57:52 2022 -0400 Resolve pylint issues in pylexotron.py and improve readability Patch by Brad Schoening; reviewed by brandonwilliams and smiklosovic for CASSANDRA-17779 commit c7d2e97da04e09dd908b1ea238a8498cfe5a3edb Author: David Capwell Date: Mon Aug 8 10:37:30 2022 -0700 NPE bug in streaming checking if SSTable is being repaired patch by David Capwell; reviewed by Marcus Eriksson for CASSANDRA-17801 commit d0e566a9537148428def91092f6ed7e634d080d3 Merge: b9c40e2573 bd08798c13 Author: Ekaterina Dimitrova Date: Mon Aug 8 15:25:56 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit bd08798c1385ddb132701d0f5c200066e3329d77 Author: Ekaterina Dimitrova Date: Thu Aug 4 11:11:00 2022 -0400 Add tests for MILLIS_CUSTOM_DURATION and MILLIS_DURATION_DOUBLE patch by Ekaterina Dimitrova; reviewed by Andres de la Pena for CASSANDRA-17737 commit b9c40e25738f0ca7fc38e51a026743ead3420ab7 Author: Blake Eggleston Date: Tue Aug 2 10:51:15 2022 -0700 Use seeded crc for PaxosBallotTracker checksum Patch by Blake Eggleston; Reviewed by Benedict Elliot Smith for CASSANDRA-17793 commit 61d9ba6dff07feabf20c262a3cc3d75483b23a94 Merge: 8691d9b6d7 a7d7708976 Author: Ekaterina Dimitrova Date: Mon Aug 8 13:11:08 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit a7d7708976347b0cbf9d1ad44d0a4fe8a1cdbaf8 Merge: 63231aa826 e5e13c02cc Author: Ekaterina Dimitrova Date: Mon Aug 8 12:47:01 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 8691d9b6d7a9500585e1582458fe38e71e4a5bd7 Author: Josh McKenzie Date: Thu Aug 4 14:14:55 2022 -0400 Add UUID to nodetool import logging Patch by Marcus Eriksson; reviewed by Josh McKenzie, Jordan West, and Sam Tunnicliffe for CASSANDRA-17800 Co-authored-by: Marcus Eriksson Co-authored-by: Josh McKenzie commit db7f2a97c6d2cff8de28a0ef4e91fb76e03d80f8 Merge: 881b08f701 63231aa826 Author: Andrés de la Peña Date: Mon Aug 8 13:57:20 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 63231aa826d126499ea2a2fcbd55a118e39136af Merge: 5e39c54678 821e7ab7bd Author: Andrés de la Peña Date: Mon Aug 8 13:56:52 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 881b08f7015a4342833079e648e478526cc3b31a Merge: 45f4f8c1e8 5e39c54678 Author: Caleb Rackliffe Date: Fri Aug 5 15:38:07 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 5e39c54678b0d70145b5c699593eb9ce7fa7706a Author: Caleb Rackliffe Date: Wed Jun 22 17:12:41 2022 -0600 Avoid initializing schema via SystemKeyspace.getPreferredIP() with the BulkLoader tool patch by Caleb Rackliffe; reviewed by Jon Meredith for CASSANDRA-17740 Co-authored-by: Caleb Rackliffe Co-authored-by: Jon Meredith commit 45f4f8c1e89e4b221b569ff3bd3e78675eff7747 Author: Josh McKenzie Date: Tue Aug 2 14:30:06 2022 -0400 Users of NativeLibrary should handle lack of JNA appropriately when running in client mode Patch by Doug Rohrer; reviewd by Josh McKenzie and Caleb Rackliffe for CASSANDRA-17794 Co-authored-by: Doug Rohrer Co-authored-by: Josh McKenzie commit f4f04fbac0eaa247dd028ddcb5a4e36dd8cd11ba Author: Josh McKenzie Date: Fri Aug 5 07:29:02 2022 -0400 Fix checkstyle failures on JDK8 introduced by CASSANDRA-17777 Patch by Josh McKenzie; reviewed by Marcus Eriksson for CASSANDRA-17777 commit 037149377224c5d6854fa4a0cacf44139273bce3 Author: Josh McKenzie Date: Tue Jul 26 15:15:48 2022 -0400 Warn on unknown directories found in system keyspace directory rather than kill node during startup checks Patch by Jeff Jirsa; reviewed by Josh McKenzie, Sam Tunnicliffe, and Marcus Eriksson for CASSANDRA-17777 Co-authored-by: Jeff Jirsa Co-authored-by: Josh McKenzie commit e36aeb49e008568a2f551bb749bbb55aeaa80a72 Author: Josh McKenzie Date: Mon Aug 1 14:37:26 2022 -0400 Log duplicate rows found during nodetool verify and scrub Patch by Marcus Eriksson; reviewed by Josh McKenzie for CASSANDRA-17789 Co-authored-by: Marcus Eriksson Co-authored-by: Josh McKenzie commit 652e49a8bb2f8849ce0ebb5161c30e3889c14608 Author: Josh McKenzie Date: Wed Jul 27 12:57:28 2022 -0400 Add separate thread pool for Secondary Index building so it doesn't block compactions Patch by Chris Lohfink; reviewed by Caleb Rackliffe, Josh McKenzie, Sam Tunnicliffe, and Marcus Eriksson for CASSANDRA-17781 Co-authored-by: Chris Lohfink Co-authored-by: Josh McKenzie commit 62ac5da78a1b99c7d28d9558954c6dba040d52e1 Merge: 0daf21244f dd08314ed6 Author: Ekaterina Dimitrova Date: Mon Aug 1 20:39:44 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit dd08314ed654aafa60b2a82fc4953aac171ba3ef Author: Ekaterina Dimitrova Date: Thu Jun 30 17:06:58 2022 -0400 Uncomment prepared_statements_cache_size, key_cache_size, counter_cache_size, index_summary_capacity which were commented out by mistake in a previous patch; Fix breaking change with cache_load_timeout; cache_load_timeout_seconds <=0 and cache_load_timeout=0 are equivalent and they both mean disabled; Deprecate public method setRate(final double throughputMbPerSec) in Compaction Manager in favor of setRateInBytes(final double throughputBytesPerSec); Revert breaking change removal of StressCQLSSTableWriter.Builder.withBufferSizeInMB(int size). Deprecate it in favor of StressCQLSSTableWriter.Builder.withBufferSizeInMiB(int size); Fix precision issues, add new -m flag (for nodetool/setstreamthroughput, nodetool/setinterdcstreamthroughput, nodetool/getstreamthroughput and nodetoo/getinterdcstreamthroughput), add new -d flags (nodetool/getstreamthroughput, nodetool/getinterdcstreamthroughput, nodetool/getcompactionthroughput); Fix a bug with precision in nodetool/compactionstats; Deprecate StorageService methods and add new ones for stream_throughput_outbound, inter_dc_stream_throughput_outbound, compaction_throughput_outbound in the JMX MBean `org.apache.cassandra.db:type=StorageService`; Removed getEntireSSTableStreamThroughputMebibytesPerSec in favor of new getEntireSSTableStreamThroughputMebibytesPerSecAsDouble in the JMX MBean `org.apache.cassandra.db:type=StorageService`; Removed getEntireSSTableInterDCStreamThroughputMebibytesPerSec in favor of getEntireSSTableInterDCStreamThroughputMebibytesPerSecAsDouble in the JMX MBean `org.apache.cassandra.db:type=StorageService` Patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, Francisco Guerrero for CASSANDRA-17225 commit 0daf21244fc0187d092616834d38df1a77dcabf0 Author: Josh McKenzie Date: Mon Jul 25 11:58:42 2022 -0400 Add JMX call to getSSTableCountPerTWCSBucket for TWCS Patch by Stefan Podkowinski; reviewed by Caleb Rackliffe and Marcus Eriksson for CASSANDRA-17774 Co-authored-by: Stefan Podkowinski Co-authored-by: Josh McKenzie commit b94bf3f2186aa083a25ddb6cf7f6b2f45fc8b566 Merge: 0e3bdeab73 fe28f482db Author: Ekaterina Dimitrova Date: Fri Jul 29 13:27:30 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit fe28f482db0e1bc81936a62d0564e9657f428d2a Author: Ekaterina Dimitrova Date: Wed Jul 27 09:41:44 2022 -0400 Fix sstable_preemptive_open_interval_in_mb disabled value - backward compatibility patch by Ekaterina Dimitrova; reviewed by Andres de la Pena and Jon Meredith for CASSANDRA-17737 commit 0e3bdeab73ad89b7f48fd366b1b2723f5800e784 Merge: ea03af4605 222f66e093 Author: Ekaterina Dimitrova Date: Thu Jul 28 16:22:06 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 222f66e093b1cf315319b27fd8d0990bc9a9c92c Merge: 3a0936b61f 04901913d6 Author: Ekaterina Dimitrova Date: Thu Jul 28 16:19:47 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ea03af460511442e771ddd704d92568a63557686 Merge: d1f461115c 3a0936b61f Author: Caleb Rackliffe Date: Wed Jul 27 15:08:44 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 3a0936b61f63d8d7ab08bda411dd7d5963cafe0b Author: Caleb Rackliffe Date: Tue Jul 26 16:27:21 2022 -0500 Remove usages of Path#toFile() in the snapshot apparatus patch by Caleb Rackliffe; reviewed by David Capwell and Benedict Elliot Smith for CASSANDRA-17769 commit d1f461115c819d905ca8c2df397d9a8bdde1396f Merge: c1fa8e3c92 1fef492e2b Author: Brandon Williams Date: Wed Jul 27 12:51:03 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 1fef492e2bedc024e994a173222d27c6f8d7f0a3 Author: Brandon Williams Date: Wed Jul 27 11:45:55 2022 -0500 Don't require an echo response in TpStatsTest Patch by brandonwilliams; reviewed by maedhroz for CASSANDRA-17771 commit c1fa8e3c92d63b8dc1bb8214b4147f001fc8c2e9 Merge: 1f06705153 8533a6764b Author: Stefan Miklosovic Date: Wed Jul 27 15:12:45 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 8533a6764ba4f7eec700161dab08a0d7fb50b8f6 Merge: 1c70149ef5 6388fd3a16 Author: Stefan Miklosovic Date: Wed Jul 27 14:51:53 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 1f067051537618804ca9d0c66b903be74b942b22 Author: David Capwell Date: Tue Jul 26 11:36:01 2022 -0700 When doing a host replacement, -Dcassandra.broadcast_interval_ms is used to know when to check the ring but checks that the ring wasn't changed in -Dcassandra.ring_delay_ms, changes to ring delay should not depend on when we publish load stats patch by David Capwell; reviewed by Brandon Williams, Caleb Rackliffe for CASSANDRA-17776 commit e0a6b83a02804bf976fdc43718001f23818ee53d Author: David Capwell Date: Mon Jul 25 12:26:35 2022 -0700 When bootstrap fails, CassandraRoleManager may attempt to do read queries that fail with "Cannot read from a bootstrapping node", and increments unavailables counters patch by David Capwell; reviewed by Sam Tunnicliffe for CASSANDRA-17754 commit a57eae67e5d73f8ab3fd0ab172262380c8dc0280 Author: Josh McKenzie Date: Fri Jul 22 13:34:41 2022 -0400 Improve javadoc on CQLTester and GuardrailTester assertion methods Patch by Josh McKenzie; reviewed by Andres de la Pena for CASSANDRA-17772 commit d2d6afb8e928755b3ddcb06e0d3b254fa858fec9 Merge: 523424b9ff 1c70149ef5 Author: Ekaterina Dimitrova Date: Mon Jul 25 10:15:30 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 1c70149ef512212d9cf8f57cb703fce92bf3dafa Author: Ekaterina Dimitrova Date: Thu Jul 21 20:30:48 2022 -0400 Fix Settings Virtual Table to update paxos_variant after startup and rename enable_uuid_sstable_identifiers to uuid_sstable_identifiers_enabled as per our config naming conventions patch by Ekaterina Dimitrova; reviewed by Brandon Williams for CASSANDRA-17738 commit 523424b9ffa7d9fbdba81feeb6c3eadad051f0db Merge: 8451acc8d8 15b7d678e9 Author: Stefan Miklosovic Date: Sat Jul 23 22:35:17 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 15b7d678e96dd84511e0946b3db6eafaa34b1f05 Merge: 962247a493 a1ac1efbb7 Author: Stefan Miklosovic Date: Sat Jul 23 22:27:49 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 8451acc8d8dcfee20d692d1e70ae11b60d2f004e Merge: 91b86487fe 962247a493 Author: Brandon Williams Date: Fri Jul 22 11:30:28 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 962247a49333e4fd3c93d13e9ae501f99fe947a9 Merge: 9beeaf87bd a53bb7ab82 Author: Brandon Williams Date: Fri Jul 22 11:30:13 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 91b86487fe5389765711b3266097fb3faa84e5dd Author: Josh McKenzie Date: Thu Jul 21 14:59:36 2022 -0400 Add guardrail to disallow DROP KEYSPACE commands for non superuser accounts Patch by Josh McKenzie; reviewed by Aleksey Yeschenko for CASSANDRA-17767 commit 66177be58715cb6bef777165e979cad32c5dfe4a Merge: ab9ab903fa 9beeaf87bd Author: Ekaterina Dimitrova Date: Fri Jul 22 09:45:56 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 9beeaf87bd2bbf590520a34b35d3bd961dc041b2 Merge: b4e58d83fc 860ec627a1 Author: Ekaterina Dimitrova Date: Fri Jul 22 09:27:53 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ab9ab903fa590409251e97fe075e02a64c8aa4f3 Merge: f251e5052e b4e58d83fc Author: Ekaterina Dimitrova Date: Wed Jul 20 17:59:49 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit b4e58d83fc72f8babf61367aad23a9499df69881 Author: Ekaterina Dimitrova Date: Mon Jul 18 16:37:53 2022 -0400 Fix flaky test TopPartitionsTest patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe and David Capwell for CASSANDRA-17649 commit f251e5052e42dd845932760e87988d88f0a86ad2 Merge: adcff3f630 bc40f0c17c Author: Mick Semb Wever Date: Mon Jul 18 22:31:18 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit bc40f0c17c5842028645eff811a9645097e34fcc Merge: 38abed3410 c5747547fa Author: Mick Semb Wever Date: Mon Jul 18 22:30:32 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit adcff3f630c0d07d1ba33bf23fcb11a6db1b9af1 Merge: 55f094a6d2 38abed3410 Author: Mick Semb Wever Date: Sat Jul 16 13:01:08 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 38abed341034e0e605954abf4b3eb65faec0ed99 Merge: 46d9cbe2f6 65ef0c0c7f Author: Mick Semb Wever Date: Sat Jul 16 13:00:56 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 55f094a6d2c288b74e6ea86edaa7c45cd494c208 Author: Stefan Miklosovic Date: Fri Jun 24 12:42:12 2022 +0200 Remove ephemeral snapshot marker file and introduce a flag to SnapshotManifest patch by Stefan Miklosovic; reviewed by Paulo Motta for CASSANDRA-16911 commit 1c2cd301258f446747b673e3037135a0809335bd Merge: 889ca60edc 46d9cbe2f6 Author: Ekaterina Dimitrova Date: Fri Jul 15 12:35:51 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 46d9cbe2f66b02cedf1b7961c4c1194488e1323b Author: Ekaterina Dimitrova Date: Wed Jul 6 12:18:34 2022 -0400 min_tracked_partition_size_bytes parameter from 4.1 alpha1 was renamed to min_tracked_partition_size patch by Ekaterina Dimitrova; reviewed by Berenguer Blasi for CASSANDRA-17733 commit 889ca60edc8afd85b4a594442883c1b5efdf3c6b Merge: 315a1a72a6 9bd32771e3 Author: Caleb Rackliffe Date: Thu Jul 14 11:34:54 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 9bd32771e3983efdb0b135699a036c91e89b52dd Merge: 60f742c739 d51f90201b Author: Caleb Rackliffe Date: Thu Jul 14 11:26:59 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 315a1a72a68e145d7a3a4c9bf28907a19ebdf710 Merge: 520fb2bb24 60f742c739 Author: Benedict Elliott Smith Date: Thu Jul 14 17:20:52 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 60f742c739ba4830a5d459bf3efa0783a9717bb6 Author: Benedict Elliott Smith Date: Thu Jul 14 13:50:15 2022 +0100 Followup to CASSANDRA-17530: Fix synchronisation of ClassTransformer, and make compilation deterministic patch by Benedict; reviewed by Blake Eggleston for CASSANDRA-17530 commit 520fb2bb24cef6891ec9912690b0e94d2ff2016c Merge: dfd152cfa0 27a31f4cfa Author: Andrés de la Peña Date: Thu Jul 14 15:48:08 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 27a31f4cfabb3faf65efc38ef866f7c1f78779ed Author: Andrés de la Peña Date: Tue Jul 12 15:31:21 2022 +0100 Fix flaky BatchUpgradeTest by waiting for clean system.batches patch by Andrés de la Peña; reviewed by Brandon Williams and Berenguer Blasi for CASSANDRA-17651 commit dfd152cfa0c4f38b9b7d214f2520081abab98ab7 Merge: 20815a3108 14fc06a3ce Author: Bereng Date: Thu Jul 14 11:41:39 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 14fc06a3ce007b0cbf3bc88c3d8c247da65fc036 Author: Bereng Date: Fri Jul 1 10:14:40 2022 +0200 Raise test timeouts patch by Berenguer Blasi; reviewed by Michael Semb Wever for CASSANDRA-17729 commit 20815a31089447b0d4d1808ed8d6e522f068fae7 Merge: 5aec5a9103 ada958eaf7 Author: Bereng Date: Thu Jul 14 06:59:24 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit ada958eaf7bd37c38cda55ef469c87f1e7e958bb Author: Bereng Date: Fri May 27 09:29:00 2022 +0200 org.apache.cassandra.db.commitlog.GroupCommitLogTest tests failing on trunk patch by Berenguer Blasi; reviewed by Andres de la Peña for CASSANDRA-17232 commit 5aec5a9103590400ff8483f5dd1f44706d7ccc78 Merge: 239b3e5079 c410247c07 Author: Mick Semb Wever Date: Wed Jul 13 11:26:01 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit c410247c072d0447b2e4bf03793f40c86fc67896 Merge: cdc0ce1fa3 ec476e0e25 Author: Mick Semb Wever Date: Wed Jul 13 11:25:49 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 239b3e50798d80fe81ceb56db3c5c869d675a8e3 Merge: 8468d33a31 cdc0ce1fa3 Author: Benjamin Lerer Date: Mon Jul 11 17:43:36 2022 +0200 Merge branch cassandra-4.1 into trunk commit 8468d33a31346a49123873019aa3f18efe6f4247 Merge: 6fcaed8fd6 067121da63 Author: Benedict Elliott Smith Date: Tue Jul 12 12:29:12 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 067121da63c2a8ead48aeb9a4241af5306b14a37 Author: Benedict Elliott Smith Date: Tue Jul 12 12:26:33 2022 +0100 Paxos v2 Linearizability Violation patch by Benedict; reviewed by Blake Eggleston for CASSANDRA-17530 commit cdc0ce1fa33756f091785b58d4cc8f50fa9e7d0e Author: Blake Eggleston Date: Tue Jun 14 14:31:37 2022 -0700 Prevent premature initialization of paxos metadata files during tests Patch by Blake Eggleston; review by Benjamin Lerer and Berenguer Blasi for CASSANDRA-17489 commit 6fcaed8fd635f84acf402dbf86fe94af0a7406ca Merge: cc28e1a85e f50862b434 Author: Benjamin Lerer Date: Mon Jul 11 16:50:31 2022 +0200 Merge branch cassandra-4.1 into trunk commit f50862b434736ea33360559c7539a1a7bbcce2b2 Merge: 1d80dd0086 12b588da08 Author: Benjamin Lerer Date: Mon Jul 11 16:49:49 2022 +0200 Merge branch cassandra-4.0 into cassandra-4.1 commit cc28e1a85e2e5760436030ca57667fa69d6de233 Merge: 89f3978dcd 1d80dd0086 Author: Bereng Date: Mon Jul 11 09:47:29 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 1d80dd008677e0084b1530295207a568853badee Author: Bereng Date: Thu Jun 9 08:53:01 2022 +0200 Fix flaky test - org.apache.cassandra.cql3.validation.operations.InsertUpdateIfConditionTest.testConditionalUpdate patch by Berenguer Blasi; reviewed by Benjamin Lerer for CASSANDRA-17653 commit 89f3978dcde958fbad191b8cf628fd89ace64d7a Author: Caleb Rackliffe Date: Mon Jun 27 18:40:40 2022 -0500 Add a virtual table that exposes currently running queries patch by Chris Lohfink; reviewed by Caleb Rackliffe and Benedict Elliott Smith for CASSANDRA-15241 Co-authored-by: Chris Lohfink Co-authored-by: Caleb Rackliffe Co-authored-by: Benedict Elliott Smith commit 2355ef08a2bf048f4db9d8b24e348ec36a680fcc Merge: 481fdde885 31eb6852e1 Author: Mick Semb Wever Date: Fri Jul 8 12:25:56 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 31eb6852e17bcb36d8276867c75394af5da6e490 Merge: f77a7a29e8 5072bde5fb Author: Mick Semb Wever Date: Fri Jul 8 12:25:11 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 481fdde885e971733c1e72513b3ada90a6392cc1 Merge: b29ad8823a f77a7a29e8 Author: Bereng Date: Fri Jul 8 09:19:49 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit f77a7a29e8bbfc647dd69d42d2fa33ff6de765a5 Author: Bereng Date: Tue Jun 21 11:34:51 2022 +0200 Update the CQL version for the 4.1 release Patch by Berenguer Blasi; reviewed by Benjamin Lerer for CASSANDRA-17570 commit b29ad8823a20d910a0b9b984b06421f390cc0a17 Merge: ab0a9b5f5c 50e7a3f5df Author: Benjamin Lerer Date: Thu Jul 7 14:11:03 2022 +0200 Merge branch cassandra-4.1 into trunk commit 50e7a3f5df1e8934ddfa56e8c3b00a79ea5a14c8 Merge: a250126f0f c378874a9f Author: Benjamin Lerer Date: Thu Jul 7 13:57:19 2022 +0200 Merge branch cassandra-4.0 into cassandra-4.1 commit ab0a9b5f5ca9b5189da2834698897961aac39e87 Merge: 5c7a47b384 a250126f0f Author: Yifan Cai Date: Wed Jul 6 17:41:16 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit a250126f0f277b43a18cb665ccd02a105271bc33 Author: Francisco Guerrero Date: Wed Jul 6 17:34:25 2022 -0700 Remove commons-lang dependency during build runtime patch by Francisco Guerrero; reviewed by Dinesh Joshi, Yifan Cai for CASSANDRA-17724 commit 5c7a47b384acf0f2cc38f9ebe8eff289afe48028 Merge: 5616675b1f 15bdf2e8e7 Author: Caleb Rackliffe Date: Tue Jul 5 18:46:28 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 15bdf2e8e7e7a66dd85bf3f86cc58576a13657be Merge: 91d08e496e 924cd8f52c Author: Caleb Rackliffe Date: Tue Jul 5 18:40:36 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 5616675b1fa7d5b979c0d345ea5f3a3182f6c2d7 Merge: 33a9093c5c 91d08e496e Author: Andrés de la Peña Date: Tue Jul 5 12:19:42 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 91d08e496ee439a2179e617f4db8804ab07c301a Author: Andrés de la Peña Date: Wed Jun 8 17:12:58 2022 +0100 Fix MixedModeAvailability upgrade dtests patch by Andrés de la Peña; reviewed by Berenguer Blasi for CASSANDRA-17307 commit 33a9093c5cc2f8fcf913d1931415b697e52ec108 Author: Eduard Tudenhoefner Date: Fri Jul 1 08:35:04 2022 +0200 Allow sstableloader to specify table without relying on path patch by Eduard Tudenhoefner; reviewed by Stefan Miklosovic, Brandon Williams for CASSANDRA-16584 commit 7119cf8be1c32fa6ac015456965ae27a17cd828c Author: Jyothsna Konisa Date: Thu Jun 30 09:52:50 2022 -0700 Fix TestGossipingPropertyFileSnitch.test_prefer_local_reconnect_on_listen_address patch by Jyothsna Konisa; reviewed by Jon Meredith, Yifan Cai for CASSANDRA-17700 commit 9c411d3f1ae6617380786cd5423151ee645ec209 Merge: 21591cf830 5e4eeb2d0c Author: Jon Meredith Date: Thu Jun 30 13:37:07 2022 -0600 Merge branch 'cassandra-4.1' into trunk commit 5e4eeb2d0c69b920eadd83559f6ab771c7b6a8dc Merge: a9725b681b 008bf8607c Author: Jon Meredith Date: Thu Jun 30 13:34:03 2022 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 21591cf8305d6f473ec512ef31abe4a11da06f61 Merge: f2faec9ff2 a9725b681b Author: Yifan Cai Date: Wed Jun 29 18:05:14 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit a9725b681b948f2122f3d48b96a5c4e7403d2c39 Author: Francisco Guerrero Date: Wed Jun 29 11:15:10 2022 -0700 Fix AbstractCell#toString throws MarshalException for cell in collection patch by Francisco Guerrero; reviewed by Caleb Rackliffe, Yifan Cai for CASSANDRA-17695 commit f2faec9ff2107b3df9a920b02a1d1ddea38ba74a Merge: e966c45afc b75d56fd38 Author: Brandon Williams Date: Wed Jun 29 10:15:01 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit b75d56fd387a3af2706f75c0536cf112fdf6b788 Author: Brandon Williams Date: Thu Jun 23 13:18:33 2022 -0500 Add new compactionstats output behind --vtable Patch by brandonwilliams; reviewed by edimitrov and dcapwell for CASSANDRA-17683 commit e966c45afcf8bef47df245ccb851386e5ce60505 Author: jacek-lewandowski Date: Fri Nov 6 14:59:56 2020 +0100 ByteComparable API Provides an API for converting all values of types that can be used in primary keys to byte sequences that can be compared lexicographically by unsigned byte value (i.e. byte-comparable sequences) and back. patch by Branimir Lambov, Dimitar Dimitrov and Jacek Lewandowski; reviewed by Caleb Rackliffe, Dimitar Dimitrov, Jacek Lewandowski and Aleksey Yeschenko for CASSANDRA-6936 commit 4e8c83118963c1e142bd29eb997be73308a942c7 Merge: 8ffaf4af62 60b54425ed Author: Branimir Lambov Date: Wed Jun 29 12:12:02 2022 +0300 Merge branch 'cassandra-4.1' into trunk commit 60b54425edc0a328bc1baf00c2e5bf111d4b9da8 Author: Jakub Żytka Date: Tue May 11 10:01:12 2021 +0200 Fix commitLogUpperBound initialization in AbstractMemtableWithCommitlog so that it is always available when writeBarrier gets initialized. Harden Memtable API so that it is apparent that getting commitLogUpperBound is valid only after it is fully established. patch by Jakub Żytka and Dan Jatnieks; reviewed by Dan Jatnieks, Jeremiah D Jordan and Caleb Rackliffe for CASSANDRA-17587 commit 8ffaf4af62b87e2b7a70aea88a850242eb522f9a Merge: 63c6467746 07e5876c43 Author: Ekaterina Dimitrova Date: Tue Jun 28 18:17:51 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 07e5876c43998fe7384df115c65749d668183130 Author: Akın Tekeoğlu Date: Tue Jun 21 00:10:59 2022 +0300 Add toBytesInLong to IntKibibytesBound to support widening to long in getBatchSizeFailThreshold patch by Akin Tekeoglu; reviewed by Ekaterina Dimitrova and Andres de la Peña for CASSANDRA-17650 commit 63c6467746d144287e6d90190addb8c58160d7ef Merge: 07d97d7905 495d609c89 Author: Ekaterina Dimitrova Date: Tue Jun 28 14:42:22 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 495d609c89618e2c74cddd8bf6246a833852a64a Author: Ekaterina Dimitrova Date: Mon Jun 27 10:57:24 2022 -0400 Fix widening from mebibytes to bytes in IntMebibytesBound commit 07d97d7905a441ce3c14f4a5ba40d8fedfcc0ba8 Merge: f8207fe287 dcb4768345 Author: Ekaterina Dimitrova Date: Mon Jun 27 19:35:23 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit dcb4768345e0fe96420c4f4597e2eb822040d15a Author: Ekaterina Dimitrova Date: Fri Jun 24 13:35:13 2022 -0400 nodetool clientstats columns should be under a separate flag - --config-options patch by Ekaterina Dimitrova; reviewed by Brandon Williams for CASSANDRA-17715 commit f8207fe28748a168a76f73db37dec8d42dd67a7a Merge: 39e89fd636 33730d9d06 Author: Andrés de la Peña Date: Mon Jun 27 15:16:03 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 33730d9d0603cc110fb8da68d9663b4d52e9b7e6 Merge: 36d0c4e9ae 90e2502c6c Author: Andrés de la Peña Date: Mon Jun 27 15:15:21 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 39e89fd636ee4343eb2201820da87881cbc749e2 Author: Savni Nagarkar Date: Mon Apr 18 16:50:19 2022 -0400 Add guardrail for maximum replication factor Patch by Savni Nagarkar; reviewed by Andrés de la Peña, David Capwell and Josh McKenzie for CASSANDRA-17500 commit 92069ec0932774357f5a7babaf3ec28ca1255286 Author: Bereng Date: Fri Jun 3 07:29:09 2022 +0200 testsome target doesn't work with wildcards Patch by Bernardo Botella Corbi; reviewed by Brandon Williams and Berenguer Blasi for CASSANDRA-17083 commit 465547fdeda9c89199724ec2198f864cef413e9e Author: Attila Homoki Date: Fri May 20 09:58:11 2022 -0400 Increment CQLSH to version 6.2.0 for release 4.2 Patch by Attila Homoki; reviewed by Brandon Williams and Ekaterina Dimitrova for CASSANDRA-17646 Co-authored-by: Brad Schoening <5796692+bschoening@users.noreply.github.com> commit a4796b65e366f4713dc29710504570c3278afdfd Merge: 081d42472d 36d0c4e9ae Author: Brandon Williams Date: Tue Jun 21 09:47:10 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 36d0c4e9ae195518df72d17fef97b420f451ded3 Merge: 1d39a388ba 48c1b7f9b4 Author: Brandon Williams Date: Tue Jun 21 09:46:56 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 081d42472d5ba7446fd7e2f6cecd4b0ec6d98214 Merge: c7202f9d84 1d39a388ba Author: Bereng Date: Tue Jun 21 11:07:16 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 1d39a388ba2866b77a9fb262b5953ade851f4160 Author: Bereng Date: Tue Jun 21 11:06:10 2022 +0200 Revert "Update the CQL version for the 4.1 release" This reverts commit a508ac06e9a24272d2f69f7a7546640626c16fc4. commit c7202f9d84f238cc561c23f98989b76e74916cd3 Merge: 2c7f021c56 a508ac06e9 Author: Bereng Date: Tue Jun 21 08:16:41 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit a508ac06e9a24272d2f69f7a7546640626c16fc4 Author: Bereng Date: Thu May 5 09:53:38 2022 +0200 Update the CQL version for the 4.1 release patch by Berenguer Blasi; reviewed by Benjamin Lerer for CASSANDRA-17570 commit 2c7f021c56d700a7a6acde837ec86cb066b8f697 Merge: 53490acf65 b283f0602e Author: Ekaterina Dimitrova Date: Mon Jun 20 21:14:00 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit b283f0602ee26fc49113331ebbdb79fddcb485a9 Merge: e50e2e93fb c8481836e6 Author: Ekaterina Dimitrova Date: Mon Jun 20 21:02:10 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 53490acf65fd0bb51ffe0d6d22d6d63f31a44272 Merge: 904f964e37 e50e2e93fb Author: Mick Semb Wever Date: Sun Jun 19 16:41:36 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit e50e2e93fb5a9f14ac1b3f6e0a2cb922095ffa1e Merge: 89df37d4bb 15ecaaed43 Author: Mick Semb Wever Date: Sun Jun 19 16:40:53 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 904f964e37fc55f3bb0b8415b9a3b45f3e1541d5 Merge: df5d8bfea0 89df37d4bb Author: Stefan Miklosovic Date: Thu Jun 16 19:38:14 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 89df37d4bb1541569f1f3ba9bda88a7387d69692 Merge: 96ef1d231c 7da28ef378 Author: Stefan Miklosovic Date: Thu Jun 16 19:27:31 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit df5d8bfea08c22e54d62e00c935187cd2623f385 Merge: c3dc18d698 96ef1d231c Author: Brandon Williams Date: Thu Jun 16 09:01:40 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 96ef1d231c819d1b7c87902eac16b22a2534dbfd Merge: 7f38db348e 51ce71881c Author: Brandon Williams Date: Thu Jun 16 09:01:24 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit c3dc18d6984ee232c9da8aebd4161e5a13e1cb66 Merge: 4b1996f5c1 7f38db348e Author: Brandon Williams Date: Thu Jun 16 06:01:35 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 7f38db348e62dbfb64944a2a78ad88485b2190a6 Merge: b12a9083a9 c3998ef601 Author: Brandon Williams Date: Thu Jun 16 06:00:57 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 4b1996f5c1b69db9744891f771591c42d277f0d1 Merge: fca2502e1d b12a9083a9 Author: Sam Tunnicliffe Date: Wed Jun 15 12:02:53 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit b12a9083a97b0eaa3fafb3ef71cf1a79bde14da6 Merge: 92aacd6673 28690e0ebc Author: Sam Tunnicliffe Date: Wed Jun 15 11:58:49 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit fca2502e1dc05876640396732ec1728e16f1944d Merge: 557b8e9982 92aacd6673 Author: Bereng Date: Tue Jun 14 07:00:51 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 92aacd6673ce78cb891255e54653cbc9cbc19e32 Merge: 67632edc51 bb5b2d2889 Author: Bereng Date: Tue Jun 14 06:59:45 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 557b8e9982ad0964191abde810ef5c77a536f70a Author: Jyothsna Konisa Date: Mon Jun 13 11:05:22 2022 -0700 Adding support to perform certificate based internode authentication patch by Jyothsna Konisa; reviewed by Jon Meredith, Yifan Cai for CASSANDRA-17661 commit 56dbc68528c92b9ccaaa34ab19640a7613124828 Merge: 067f4c7461 67632edc51 Author: Stefan Miklosovic Date: Mon Jun 13 10:57:49 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 67632edc51d5fc40e204ba36e11376a89c1077aa Merge: fd547aff55 2531dd1eba Author: Stefan Miklosovic Date: Mon Jun 13 10:41:24 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 067f4c74612698881feec09039d71d12a5936418 Merge: 1b3aec8eef fd547aff55 Author: Brandon Williams Date: Wed Jun 8 11:19:35 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit fd547aff5517d41589a99ad99d7d2cd0e426e5ec Merge: 457e16c27e cd0a40d09e Author: Brandon Williams Date: Wed Jun 8 11:18:59 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 1b3aec8eef6d1348177dc5a1f0ef30d4ab8b79ee Merge: 99d034a224 457e16c27e Author: Stefan Miklosovic Date: Tue Jun 7 22:19:32 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 457e16c27ee65063fa15963c58bea3e9a63c5aa5 Author: Stefan Miklosovic Date: Thu Jun 2 13:58:38 2022 +0200 fix flaky GossipInfoTableTest patch by Stefan Miklosovic; reviewed by Andrés de la Peña for CASSANDRA-17584 commit 99d034a2245c44becb6a730c77ad51ab9340f3a7 Author: Yifan Cai Date: Mon Jun 6 13:15:33 2022 -0700 Option to disable CDC on SSTable repair patch by Yifan Cai; reviewed by Josh McKenzie for CASSANDRA-17666 commit 29fea66c89cc1b378aafbaca8d68d21697e667b7 Merge: 96d0d658a5 9b4784bdb7 Author: Andrés de la Peña Date: Tue Jun 7 13:05:59 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 9b4784bdb7d70bf99c9c290d44b053902b00642d Author: Sathyanarayanan Saravanamuthu Date: Wed May 11 16:21:19 2022 +0100 Fix missed nowInSec values in QueryProcessor patch by Sathyanarayanan Saravanamuthu; reviewed by Andrés de la Peña, Benjamin Lerer and Ekaterina Dimitrova for CASSANDRA-17458 Co-authored-by: Sathyanarayanan Saravanamuthu Co-authored-by: Andrés de la Peña commit 96d0d658a59245994437ac5d975e2228f446373b Merge: 81f2aeb020 f809b6753c Author: Brandon Williams Date: Tue Jun 7 05:45:25 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit f809b6753cbbd27deab40679b99d956c8193fcf8 Author: Brandon Williams Date: Fri Jun 3 19:55:05 2022 -0500 Ensure node2 is down before repairing Patch by brandonwilliams; reviewed by dcapwell for CASSANDRA-17566 commit 81f2aeb02062f7ca527bdd918516692600289e15 Merge: 6247c9d966 3dc30eb45e Author: Bereng Date: Tue Jun 7 09:20:07 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 3dc30eb45ef52368520102f471d53061676e72cc Author: Bereng Date: Tue May 31 10:04:38 2022 +0200 Flaky CompactStorageTest patch by Berenguer Blasi; reviewed by Andres de la Peña for CASSANDRA-17670 commit 6247c9d966d7e3886fef0c7486013578407b37e4 Author: David Capwell Date: Mon Jun 6 13:42:00 2022 -0700 jvm-dtest upgrade tests run all supported pairs of upgrades between from/to but does not actually test all patches from/to patch by David Capwell; reviewed by Jon Meredith, Michael Semb Wever for CASSANDRA-17656 commit 35e05dcebdb7f03166ba94951fbb42661745786e Merge: 30d23f1cc3 d3ce825bf2 Author: David Capwell Date: Mon Jun 6 13:28:48 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit d3ce825bf2b376fd2516e4b594ddb69037c13159 Author: David Capwell Date: Fri Jun 3 14:37:39 2022 -0700 Fix org.apache.cassandra.distributed.test.trackwarnings.TombstoneCountWarningTest.failThresholdSinglePartition patch by David Capwell; reviewed by Caleb Rackliffe for CASSANDRA-17244 commit 30d23f1cc30519501bfcbea2886ad06f399fd5af Merge: 3a55e0cbd0 9ad8493370 Author: Jon Meredith Date: Mon Jun 6 11:09:44 2022 -0600 Merge branch 'cassandra-4.1' into trunk commit 9ad849337063b79308dddc05c2798794354ea035 Merge: 5875cbe839 943184b194 Author: Jon Meredith Date: Mon Jun 6 11:07:14 2022 -0600 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 3a55e0cbd0ac8174a8e231e39cc928c4e597e5ba Merge: ce4ae43a31 5875cbe839 Author: Andrés de la Peña Date: Mon Jun 6 12:23:41 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 5875cbe839f4b5bd8df9a01d7e1b703449a8fd3d Merge: 2eac2590bf 52f6ac4af9 Author: Andrés de la Peña Date: Mon Jun 6 12:19:09 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit ce4ae43a310a809fb0c82a7f48001a0f8206e156 Merge: 063a37699c 2eac2590bf Author: Ekaterina Dimitrova Date: Thu Jun 2 15:33:54 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 2eac2590bfe85b76c5fc1ea7c818082e124f5ed5 Author: Ekaterina Dimitrova Date: Tue May 31 11:01:44 2022 -0400 Revert removal of withBufferSizeInMB(int size) to CQLSSTableWriter.Builder and deprecate it in favor of withBufferSizeIniB(int size);update the docs patch by Ekaterina Dimitrova; reviewed by Michael Semb Wever for CASSANDRA-17675 commit 063a37699c3d5b8548b9b7d926daeab3e08eb7db Merge: 740cec41d2 663e51d928 Author: Caleb Rackliffe Date: Thu Jun 2 12:15:17 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 663e51d92868cbd045a83d7fa53e373bb28721a0 Merge: 133ad50a84 a00d8fd5ba Author: Caleb Rackliffe Date: Thu Jun 2 12:07:38 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 740cec41d2d67783a463bd18f70221de331928df Author: David Capwell Date: Wed Jun 1 08:49:44 2022 -0700 When a node is bootstrapping it gets the whole gossip state but applies in random order causing some cases where StorageService will fail causing an instance to not show up in TokenMetadata patch by David Capwell; reviewed by Blake Eggleston for CASSANDRA-17676 commit 2ae2b5c75fec97a970579cb3149c9c0677a509f4 Merge: 770733367f 133ad50a84 Author: Ekaterina Dimitrova Date: Tue May 31 14:13:23 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 133ad50a840c67a0ae01461e470de7e313ebed7c Merge: a4196cfb30 3d7d064cd2 Author: Ekaterina Dimitrova Date: Tue May 31 14:10:47 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 770733367fd84e6f9cb29083bd8bffb698cb14f5 Merge: c7e7984008 a4196cfb30 Author: Andrés de la Peña Date: Tue May 31 17:13:20 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit a4196cfb30363b5a05d1ca54a5191f82e4ad64bd Merge: a604c0d4de fbad08979e Author: Andrés de la Peña Date: Tue May 31 17:12:35 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit c7e7984008062aa2cf2de5a1fc080674bb2139ff Merge: 044294fe0f a604c0d4de Author: Stefan Miklosovic Date: Sun May 29 09:59:43 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit a604c0d4defb50ffdf7f66c3ac372eb14f39ba3b Author: Paulo Motta Date: Tue May 10 12:23:36 2022 -0300 Remove expired snapshots of dropped tables after restart Patch by Paulo Motta; reviewed by Stefan Miklosovic for CASSANDRA-17619 Co-authored-by: Stefan Miklosovic commit 044294fe0f83eabba887833f167caaffcaf18438 Merge: 2032b2f37e d27ab73641 Author: Mick Semb Wever Date: Sat May 28 13:56:16 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit d27ab73641c532f502b60f82af76c297122cf48a Merge: 1ca7a11dd0 36b73d70fb Author: Mick Semb Wever Date: Sat May 28 13:55:41 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 2032b2f37ee18b6c9d0522afcae938ad3d60edc0 Merge: 1a668322f4 1ca7a11dd0 Author: Mick Semb Wever Date: Fri May 27 23:07:42 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 1ca7a11dd05fcee5e62dfbcd1a86be58540c343a Author: Mick Semb Wever Date: Fri May 27 21:44:06 2022 +0200 Increment version to 4.1-alpha2 commit 1a668322f40d51c672c07f67707930f48b1650a1 Merge: 72d5b4d1b1 0f83fff992 Author: Ekaterina Dimitrova Date: Fri May 27 15:34:46 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 0f83fff99247dd0a4246e1b08ea4a023772ace1c Merge: 0f58db9465 041c33e2d4 Author: Ekaterina Dimitrova Date: Fri May 27 15:31:45 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 72d5b4d1b117a4a86cd578197ecdba667aa56343 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Thu May 12 20:03:25 2022 -0400 add CQLSH command SHOW REPLICAS patch by Brad Schoening; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17577 commit 42806047f7f4b195dc9386e044544e0676b8a6cc Merge: 14fbab15bd 0f58db9465 Author: Andrés de la Peña Date: Fri May 27 11:41:08 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 0f58db94658ee4fa4af49e9499fb2dbe85e8b075 Author: Andrés de la Peña Date: Fri May 20 14:52:07 2022 +0100 Add vnodes option to CircleCI Java dtest multiplexer patch by Andrés de la Peña; reviewed by Berenguer Blasi and Ekaterina Dimitrova for CASSANDRA-17616 commit 14fbab15bd264dd1cf894bf48170cf4f30ada8a0 Author: Josh McKenzie Date: Thu May 19 15:48:20 2022 -0400 Add guardrail to allow disabling SimpleStrategy Patch by Josh McKenzie; reviewed by Aleksey Yeschenko for CASSANDRA-17647 commit 1f83985ef88fe0b85566e248f93a9b09fb388e47 Merge: dcf3d58c4b 577fb37c7e Author: Sam Tunnicliffe Date: Thu May 26 08:34:19 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit 577fb37c7ec41bf35b62cb1f93f35b47c59c0e92 Merge: a849dbb458 1322f03afa Author: Sam Tunnicliffe Date: Thu May 26 08:33:06 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit dcf3d58c4b22b8b69e8505b170829172ea3c4f5c Merge: e9edf16c7f a849dbb458 Author: David Capwell Date: Wed May 25 14:06:27 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit a849dbb458debf9a2d98ef5a08387917bffe6c36 Merge: 00250199c0 68406f07e4 Author: David Capwell Date: Wed May 25 14:03:30 2022 -0700 Merge branch 'cassandra-4.0' into cassandra-4.1 commit e9edf16c7f49cf81a26339470465d46964acb869 Author: Brandon Williams Date: Wed May 25 06:21:15 2022 -0500 Update changes for 17470 commit 39d372539f224eb1eaa3b6209610142abe74712c Author: Brandon Williams Date: Wed Mar 30 14:35:18 2022 -0500 set perms on data/logs to 750 in debian packaging Patch by brandonwilliams; reviewed by bereng for CASSANDRA-17470 commit 80d94245be9806f1e8ae0ab3c6d4d4d6898383f8 Author: Brandon Williams Date: Thu Mar 31 09:33:59 2022 -0500 set perms on data/logs to 750 in redhat packaging Patch by brandonwilliams; reivewed by bereng for CASSANDRA-17470 commit ed3901823a5fe9f8838d8b592a1b7703b12e810b Author: Jyothsna Konisa Date: Tue May 24 10:21:16 2022 -0700 Adding support for TLS client authentication for internode communication patch by Jyothsna Konisa; reviewed by Bernardo Botella, Francisco Guerrero, Jon Meredith, Maulin Vasavada, Yifan Cai for CASSANDRA-17513 commit 8c6b409116881a611623ac1a5a473003b18883f2 Merge: 1275158781 00250199c0 Author: Brandon Williams Date: Tue May 24 06:02:08 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 00250199c0bbfc8c9cae367f835f9075354c6b91 Merge: 951aff25a1 7ea24711cc Author: Brandon Williams Date: Tue May 24 05:59:06 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 12751587818de4ae586a02c7d8568d815a4c0956 Merge: 372d23b1fa 951aff25a1 Author: Josh McKenzie Date: Mon May 23 13:00:01 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 951aff25a1183f41fd146d674136399f3f25b3f0 Author: Josh McKenzie Date: Thu May 19 14:27:12 2022 -0400 Fix OfflineTokenAllocatorTest timeouts Patch by Josh McKenzie; reviewed by David Capwell for CASSANDRA-17291 commit 372d23b1fa2701a74eddb47ad93412f38ec0bfcf Merge: c0b175595d 6f05be4470 Author: Mick Semb Wever Date: Fri May 20 23:01:05 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 6f05be447073925a7f3620ddbbd572aa9fcd10ed Author: Mick Semb Wever Date: Fri May 20 22:03:07 2022 +0200 Prepare debian changelog for 4.1-alpha1 commit c0b175595dcbb6aac65448d182f66cb12db3157f Merge: c2a5c923b8 a67a2a2441 Author: Ekaterina Dimitrova Date: Fri May 20 14:33:36 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit a67a2a2441589e8bf41498c426fddacd7b9719cf Author: Ekaterina Dimitrova Date: Mon Apr 4 18:27:43 2022 -0400 Handle config parameters upper bound on startup; Fix auto_snapshot_ttl and paxos_purge_grace_period min unit validations patch by Ekaterina Dimitrova and Andres de la Pena; reviewed by Andres de la Pena and Caleb Rackliffe for CASSANDRA-17571 Co-authored-by: Ekaterina Dimitrova Co-authored-by: Andrés de la Peña commit c2a5c923b8b077a69c307f0766abb6eb64538427 Merge: 56ce8d1f7f 33b6a3d0f9 Author: Mick Semb Wever Date: Fri May 20 13:56:49 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 33b6a3d0f97b0ed49119ab570580af939340cdb0 Author: Mick Semb Wever Date: Fri May 20 13:55:34 2022 +0200 Prepare for 4.1-alpha1 ref: https://lists.apache.org/thread/hm2ljw4cpwvlb2hf79hpg78s74w39of0 commit 56ce8d1f7f90fa9dd9d1b0e3507dab0cd2dec0cf Merge: 23e5da9358 c1d163abad Author: Andrés de la Peña Date: Fri May 20 11:52:49 2022 +0100 Merge branch 'cassandra-4.1' into trunk commit c1d163abad7279c7d8bb4e76bb39de47c90bbec7 Merge: c635f198f6 5cf6db30f6 Author: Andrés de la Peña Date: Fri May 20 11:52:34 2022 +0100 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 23e5da93580401ea1b3203a64d875d992117ee57 Merge: 733a4a58ef c635f198f6 Author: Ekaterina Dimitrova Date: Thu May 19 14:38:18 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit c635f198f6ceca272e61b857b67fd91f23da3836 Author: Ekaterina Dimitrova Date: Thu May 12 18:13:20 2022 -0400 Fix leak of non-standard Java types in our Exceptions as clients using JMX are unable to handle them Remove useless validation that leads to unnecessary additional read of cassandra.yaml on startup patch by Ekaterina Dimitrova; review by David Capwell for CASSANDRA-17638 commit 733a4a58ef1a285031d0dd62f427327f36f31120 Merge: 26dd119679 0a7084a05d Author: Bereng Date: Thu May 19 10:02:01 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 0a7084a05d17c94879873d91ec95dada8bd285a1 Merge: c47dbf0a47 af24242af9 Author: Bereng Date: Thu May 19 09:59:45 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 26dd119679605bf61ad3caa24a70509e5be5aac9 Author: Yifan Cai Date: Wed May 18 10:27:16 2022 -0700 Add new CQL function maxWritetime patch by Yifan Cai; reviewed by Andres de la Peña, Francisco Guerrero for CASSANDRA-17425 commit 3c034e1bb57df01146158d6e6664de96630ddca9 Merge: d84f78ff34 c47dbf0a47 Author: Brandon Williams Date: Wed May 18 10:12:04 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit c47dbf0a478c7a5a8e0dddc3291d9eee460d4b30 Merge: 21553a5199 6158dccbcb Author: Brandon Williams Date: Wed May 18 10:11:43 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit d84f78ff3455b8d0f580910668ced98fd83238a4 Merge: 7b927eaad3 21553a5199 Author: Bereng Date: Wed May 18 08:38:39 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 21553a5199cc8c12ec520f66eadbaa9447d1fa78 Merge: 8ef34fd2b6 27b5a017ea Author: Bereng Date: Wed May 18 08:30:45 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 7b927eaad3c42657951df688673598b2594514d4 Merge: d6cfa3396b 8ef34fd2b6 Author: Mick Semb Wever Date: Sat May 14 01:31:26 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 8ef34fd2b67991f98aae0e9b7ec1cc92c328ac7f Merge: f451c2b028 4b93dbcf84 Author: Mick Semb Wever Date: Sat May 14 01:31:15 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit d6cfa3396bb1467b7b99ffc52c8777c1bfcc6a3e Merge: b9101d21bd f451c2b028 Author: David Capwell Date: Fri May 13 10:14:30 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit f451c2b02827fd1db34a14e7603a52ffe4fb188e Author: David Capwell Date: Thu May 12 16:25:04 2022 -0700 jvm-dtest unhandled exceptions failing the test no longer work patch by David Capwell; reviewed by Benedict Elliott Smith for CASSANDRA-17549 commit b9101d21bd349be506522e055bc34efe1fba5a16 Merge: e0247d6833 7dbea1a8cb Author: Brandon Williams Date: Fri May 13 06:58:49 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit 7dbea1a8cb92d14f97532615b39d9aa20625b403 Merge: f57c8d2c25 ac0266be69 Author: Brandon Williams Date: Fri May 13 06:54:46 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit e0247d6833f59901e6849e168f6e2363351c6f4c Merge: 9e49a2b21d f57c8d2c25 Author: Ekaterina Dimitrova Date: Tue May 10 21:06:51 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit f57c8d2c25f3a7c2b845ad9205ad25a9b04cca34 Author: Ekaterina Dimitrova Date: Sun Apr 17 15:40:18 2022 -0400 Fix repair_request_timeout_in_ms and remove paxos_auto_repair_threshold_mb patch by Benedict Elliott Smith and Ekaterina Dimitrova; reviewed by Benedict Elliott Smith and Blake Eggleston for CASSANDRA-17557 commit 9e49a2b21d7e7234d580df840d59a4f1e93a5ee9 Merge: 9584887040 89fbb1acf4 Author: David Capwell Date: Tue May 10 15:24:56 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit 89fbb1acf42c97f57ce7301674ab6f9647320c65 Author: David Capwell Date: Tue May 10 12:50:16 2022 -0700 Incremental repair leaks SomeRepairFailedException after switch away from flatMap patch by David Capwell; reviewed by Benedict Elliott Smith for CASSANDRA-17620 commit 95848870403413e216135976a780596b8085b8d3 Merge: 53a67ff2c3 6396562f71 Author: David Capwell Date: Tue May 10 12:41:37 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit 6396562f71316838083618714b142fd982ae0155 Author: David Capwell Date: Tue May 10 09:53:24 2022 -0700 StorageService read threshold get methods throw NullPointerException due to not handling null configs patch by David Capwell; reviewed by Berenguer Blasi for CASSANDRA-17593 commit 53a67ff2c36d90d337aba1409498de29931d4279 Merge: 60ee33ed06 5b3e7b9994 Author: Ekaterina Dimitrova Date: Mon May 9 21:42:02 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 5b3e7b9994d84e8c0e48c179d97d86f2d13e4c3b Merge: 8fc39685fb bbb11086c4 Author: Ekaterina Dimitrova Date: Mon May 9 21:39:39 2022 -0400 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 60ee33ed06ad3dc0ff641fd5c86273569148bd0c Merge: ce515a3d77 8fc39685fb Author: Yifan Cai Date: Mon May 9 15:21:35 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit 8fc39685fbb6c963cf0d86fd458069ab80bbcece Author: Yifan Cai Date: Thu May 5 14:15:49 2022 -0700 Fix testCDCIndexFileWriteOnSync and document cdc index file read edge case Patch by Yifan Cai; reviewed by Josh McKenzie for CASSANDRA-17416 commit ce515a3d77c2042575827e8bdb1da639fc957491 Author: Josh McKenzie Date: Wed Mar 23 12:42:36 2022 -0400 Add guardrail for ALTER TABLE ADD / DROP / REMOVE column operations Patch by Josh McKenzie; reviewed by Jon Meredith for CASSANDRA-17495 commit 11bdf1bf8038fa7f872fe9161a0568d023e6cfac Merge: 013acc641c 9f3bc65727 Author: Alex Sorokoumov Date: Mon May 9 10:58:07 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 9f3bc657273dfa9e20d233636adf662904f01f34 Author: Alex Sorokoumov Date: Tue Apr 19 20:50:58 2022 +0200 Reject oversized mutations on client and internode connection Patch by Alex Sorokoumov; reviewed by Andres de la Peña and Josh McKenzie for CASSANDRA-17456 commit 013acc641c5d487b07be5c082af1e85d26bd127f Author: Bernardo Botella Corbi Date: Mon Apr 18 16:09:21 2022 -0700 Rename DisableFlag class to EnableFlag on guardrails patch by Bernardo Botella Corbi; reviewed by Andrés de la Peña, Josh McKenzie and Yifan Cai for CASSANDRA-17544 commit c48906394e01460382f4070ecc34f6f9754fc567 Author: Yifan Cai Date: Thu May 5 14:15:49 2022 -0700 Fix testCDCIndexFileWriteOnSync and document cdc index file read edge case Patch by Yifan Cai; reviewed by Josh McKenzie for CASSANDRA-17416 commit acb5337e92c3032297c7fb9cfa7af4ef578c7c27 Merge: 2f1c785910 596daeb7f0 Author: Benjamin Lerer Date: Fri May 6 18:15:43 2022 +0200 Merge branch cassandra-4.1 into trunk commit 596daeb7f08e14d69af90fd4f07b9e87f2816681 Author: Benedict Elliott Smith Date: Fri May 6 17:40:27 2022 +0200 Optimise BTree build, update and transform operations Patch Benedict Elliott Smith; reviewed by Branimir Lambov and Benjamin Lerer for CASSANDRA-15510 commit 2f1c7859106031a608394aeb2f9e6d565510b481 Merge: 6f61257889 030831c5f5 Author: Josh McKenzie Date: Fri May 6 11:50:48 2022 -0400 Merge branch 'cassandra-4.1' into trunk # Conflicts: # CHANGES.txt commit 030831c5f5457c37ba78f0539898bb6cbd676823 Author: Josh McKenzie Date: Mon May 2 13:18:51 2022 -0400 Rename truncate_drop guardrail to drop_truncate_table Patch by Josh McKenzie; reviewed by Andres de la Pena for CASSANDRA-17592 commit 6f612578891d8b1bc0cdb0ae0dc3c4c8f5ddb385 Merge: 0541e5b078 c9ed13c425 Author: Brandon Williams Date: Thu May 5 05:57:23 2022 -0500 Merge branch 'cassandra-4.1' into trunk commit c9ed13c4257c7e7f351c3d3e77b70cb62fb210cf Merge: 559b17f71d a524cd4907 Author: Brandon Williams Date: Thu May 5 05:57:06 2022 -0500 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 0541e5b078a2fe6cb0199e72f1a735a6f565ef08 Merge: 6c20fbca4a 559b17f71d Author: David Capwell Date: Wed May 4 09:32:42 2022 -0700 Merge branch 'cassandra-4.1' into trunk commit 559b17f71d3207158d34b1dd2414a531e362af20 Author: David Capwell Date: Tue May 3 14:59:55 2022 -0700 nodetool enablefullquerylog can NPE when directory has no files patch by David Capwell; reviewed by Jon Meredith for CASSANDRA-17595 commit 6c20fbca4a42dc0dde8910ce2b1096861957040c Merge: c6207171f2 3b648ca09e Author: Marcus Eriksson Date: Wed May 4 08:18:49 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 3b648ca09ecfc100d5ad2e3b462d4949dbc03498 Author: Marcus Eriksson Date: Thu Apr 28 15:50:11 2022 +0200 Use TimeUUID serializer for BATCH_REMOVE_REQ Patch by marcuse; reviewed by Benedict Elliott Smith for CASSANDRA-17451 commit c6207171f283d595a8d00f7719a524b2329f413f Merge: 99bf8ba465 523d38095f Author: Ekaterina Dimitrova Date: Tue May 3 21:03:49 2022 -0400 Merge branch 'cassandra-4.1' into trunk commit 523d38095f6c7f33c602616db3a7373bbeb0cde9 Author: Ekaterina Dimitrova Date: Fri Apr 29 20:45:54 2022 -0400 Fix CircleCI MIDRES configuration patch by Ekaterina Dimitrova; reviewed by Andres de la Peña for CASSANDRA-17563 commit 99bf8ba465b15baa59d3dc4e0ed861721dabd191 Merge: 963c24969e 2480a7e7ee Author: Mick Semb Wever Date: Tue May 3 20:46:45 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 2480a7e7ee64791acec6add38e134753b95236fd Author: Mick Semb Wever Date: Mon May 2 17:17:46 2022 +0200 Update trunk to version 4.2 Update jvm-dtest supported upgrade paths. Update circeci to build 4.1 dtest jar patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova for CASSANDRA-17589 commit 963c24969e2d58840a53024d1f405fd6f83a71db Merge: 413f1aed27 62af03dc84 Author: Mick Semb Wever Date: Sun May 1 23:44:19 2022 +0200 Merge branch 'cassandra-4.1' into trunk commit 62af03dc84d3fe6d9dd23567394f8b2b5047794e Merge: 7741eacc54 4bab12c578 Author: Mick Semb Wever Date: Sun May 1 23:44:03 2022 +0200 Merge branch 'cassandra-4.0' into cassandra-4.1 commit 413f1aed27baee6e543ad2992f7a829deaef670a Author: Mick Semb Wever Date: Sun May 1 23:22:08 2022 +0200 Create release branch cassandra-4.1, increment trunk version to 4.2 commit 7741eacc546f80fe4324c7821fcf2c029f64b1f9 Author: Paulo Motta Date: Fri Apr 29 20:20:19 2022 -0300 Add auto_snapshot_ttl configuration Patch by Paulo Motta; Reviewed by Stefan Miklosovic for CASSANDRA-16790 Co-authored-by: fibersel <0583463@gmail.com> commit 31aa17a2a3b18bdda723123cad811f075287807d Author: Paulo Motta Date: Thu Apr 28 19:03:03 2022 -0300 List snapshots of dropped tables Patch by Paulo Motta; Reviewed by Stefan Miklosovic and Brandon Williams or CASSANDRA-16843 commit 458bfd16c7ec759705f920e7ef9a8f2bb5a3f4b5 Author: Jacek Lewandowski Date: Tue Apr 26 14:43:49 2022 +0200 Add information info whether sstables are dropped or not to SchemaChangeListener patch by Jacek Lewandowski; reviewed by Alex Petrov for CASSANDRA-17582 commit e4e19e33faf9ac7cf27a9779c8083a7f5c5b865a Author: Branimir Lambov Date: Fri May 14 16:13:35 2021 +0300 Add memtable API (CEP-11) patch by Branimir Lambov; reviewed by Andrés de la Peña and Caleb Rackliffe for CASSANDRA-17034 commit 77d6bbf25a59d44422f0cbee2631f2fca9170e1a Author: Jacek Lewandowski Date: Thu Apr 28 06:18:19 2022 +0200 Save sstable id as string in activity table patch by Jacek Lewandowski; reviewed by Andrés de la Peña and Ekaterina Dimitrova for CASSANDRA-17585 commit bd36cb9f6b8cc6339f43382bf6625794b641552b Author: Stefan Miklosovic Date: Mon Mar 28 19:46:11 2022 +0200 implement startup check to prevent Cassandra to potentially spread zombie data patch by Stefan Miklosovic; reviwed by Paulo Motta and Brandon Williams for CASSANDRA-17180 commit 7ce140bd1dea311b9f98cdfbcd07dcff9fbd457c Author: Marcus Eriksson Date: Thu Apr 7 14:43:40 2022 +0200 Fail starting when the same parameter exists more than once in cassandra.yaml Patch by marcuse; reviewed by David Capwell and Ekaterina Dimitrova for CASSANDRA-17379 commit da7c66222740f02c6de5b563681dd381d28f8201 Author: Savni Nagarkar Date: Thu Apr 28 13:18:39 2022 -0700 Migrate threshold for minimum keyspace replication factor to guardrails patch by Savni Nagarkar; reviewed by Andres de la Peña, David Capwell for CASSANDRA-17212 commit bb3749f2bb8282f67375c67712d8e3ca1f085879 Merge: 2180a34bbb eac7b1de35 Author: Brandon Williams Date: Thu Apr 28 09:10:47 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 2180a34bbb3443126168c733644b092aac1d731c Author: Stefan Miklosovic Date: Wed Apr 27 21:03:00 2022 +0200 Revert "Add a system property to set hostId if not yet initialized" This reverts commit 33ff36cc42a083311065dee56adc1d72d6866f2d. patch by Stefan Miklosovic; reviewed by Brandon Williams and Sam Tunnicliffe for CASSANDRA-14582 commit bcf56629e821295a22371c2cf178faae0636c68e Author: Marcus Eriksson Date: Tue Apr 26 10:08:19 2022 +0200 Bump timeouts in CasCriticalSectionTest Patch by marcuse; reviewed by Brandon Williams for CASSANDRA-17453 commit 0c37176673ca71a75990a26e421b5991db6eefe1 Merge: 6bd373f5d2 2405d523fa Author: Jon Meredith Date: Wed Apr 27 12:03:44 2022 -0600 Merge branch 'cassandra-4.0' into trunk commit 6bd373f5d27754f80384caa31d1b2a4cfc43bc19 Author: Josh McKenzie Date: Mon Apr 18 14:53:17 2022 -0400 Add guardrail to disallow TRUNCATE and DROP TABLE commands Patch by Josh McKenzie; reviewed by Aleksey Yeschenko for CASSANDRA-17558 commit 381c2a4fa82db80543fd67da6669de022d7576c4 Author: Bhouse99 Date: Wed Mar 30 13:37:22 2022 -0700 add plugin support for CQLSH patch by Brian Houser; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-16456 commit f444c4028680c78b6167161833d6564c3557618f Author: Savni Nagarkar Date: Thu Feb 17 13:29:58 2022 -0600 Add guardrail to disallow querying with ALLOW FILTERING patch by Savni Nagarkar; reviewed by Andres de la Peña, David Capwell and Josh McKenzie for CASSANDRA-17370 commit 545809616c92a91e4c39d1eedfa65800f25a2a93 Author: Marcus Eriksson Date: Thu Dec 3 11:29:38 2020 +0100 Track top partitions by size and tombstone count Patch by marcuse; reviewed by David Capwell and Yifan Cai for CASSANDRA-16310 commit 49cc352916f8dad23533b5328638aeb05ac04ea1 Author: Andrés de la Peña Date: Sat Apr 23 13:52:06 2022 +0100 Make GuardrailDiskUsageTest deterministic patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova for CASSANDRA-17576 commit 3409e5d9b341de2114d6410225922843539d4172 Author: David Capwell Date: Fri Apr 22 18:35:26 2022 -0700 Enhance SnakeYAML properties to be reusable outside of YAML parsing, support camel case conversion to snake case, and add support to ignore properties patch by David Capwell; reviewed by Caleb Rackliffe, Stefan Miklosovic for CASSANDRA-17166 commit 2b90ac1a1671b4071d9aa6f18e852021bc66702d Author: David Capwell Date: Thu Apr 21 14:37:59 2022 -0700 nodetool compact should support using a key string to find the range to avoid operators having to manually do this patch by David Capwell; reviewed by Marcus Eriksson for CASSANDRA-17537 commit 003a96b6a6f649f99138b94c52d28b73c2c3547a Merge: 2723c91878 018c8e0d5e Author: Benjamin Lerer Date: Fri Apr 22 19:28:32 2022 +0200 Merge branch cassandra-4.0 into trunk commit 2723c91878cfd7005a53f6118015c484dacc0f32 Merge: b3842de5cf 2873c91269 Author: Andrés de la Peña Date: Fri Apr 22 17:30:22 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit b3842de5cf1fa1b81872effb4585fbc7e1873d59 Author: Andrés de la Peña Date: Fri Apr 22 16:36:07 2022 +0100 Add guardrail for data disk usage patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova and Stefan Miklosovic for CASSANDRA-17150 Co-authored-by: Andrés de la Peña Co-authored-by: Zhao Yang Co-authored-by: Eduard Tudenhoefner commit c26dc06a28b0e150384474001ac23026ae76e6d5 Author: Tibor Répási Date: Wed Apr 20 22:10:13 2022 +0200 add datapaths subcommand to nodetool patch by Tibor Repasi; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17568 commit 1ad8bf67a9c82cbb5ff38e5cf785f9fe2516d009 Author: Benjamin Lerer Date: Thu Nov 4 16:47:16 2021 +0100 Allow to aggregate by time intervals Patch by Benjamin Lerer; review by Andres De la Pena and Yifan Cai for CASSANDRA-11871 The patch allow to use pure monotonic functions on the last attribute of the GROUP BY clause and introduce some floor functions that can be use to group by time range. A function is pure if: 1. The function return values are identical for identical arguments 2. The function application has no side effects A function is monotonic if it is either entirely nonincreasing or nondecreasing. commit 7db3285e7b745e591dc4c405ae9af6c1cddb0c79 Author: David Capwell Date: Wed Apr 20 15:15:34 2022 -0700 Migrate track_warnings to more standard naming conventions and use latest configuration types rather than long patch by David Capwell; reviewed by Andres de la Peña, Caleb Rackliffe for CASSANDRA-17560 commit d80bb323c9dd8f814fe02e16dec510557c7d5101 Author: ROCHETEAU Antoine Date: Thu Jan 6 13:46:50 2022 +0100 CONTAINS and CONTAINS KEY support for Lightweight Transactions patch by ROCHETEAU Antoine; reviewed by Benjamin Lerer, Berenguer Blasi for CASSANDRA-10537 commit 0d860ec662b0088ea7f77f98051121e198eb5692 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Tue Mar 22 09:41:52 2022 -0400 Add teardown to test_cqlsh_completion Patch by Brad Schoening; reviewed by brandonwilliams and smiklosovic for CASSANDRA-17465 commit dac738d2eba8629d4f482d7cbfd855d2c5b9df47 Author: Ekaterina Dimitrova Date: Tue Mar 22 19:56:52 2022 -0400 Transfer config parameters to the new types; Fix corner case for permissions_update_interval, roles_update_interval, credentials_update_interval;Fix typo in Config annotation; Made Converters type safe and fixed a few cases where converters used the wrong type; o should be provided with unit to DataStorageSpec and DurationStorageSpec; Fix null bug in DataStorageSpec and DurationSpec patch by Ekaterina Dimitrova, David Capwell; reviewed by David Capwell and Caleb Rackliffe for CASSANDRA-17431 Co-authored-by: Ekaterina Dimitrova Co-authored-by: David Capwell commit 03ef67c9d54a8cd52e120ffceb033fc49403dc2b Merge: 74bb6d8496 a46e467737 Author: Ekaterina Dimitrova Date: Sun Apr 17 14:38:37 2022 -0400 Merge branch 'cassandra-4.0' into trunk commit 74bb6d8496d5a2a5b89a843ba764fe37b65ee331 Author: Stefan Miklosovic Date: Mon Apr 11 16:25:15 2022 +0200 Shutdown ScheduledExecutors as part of node drainage patch by Stefan Miklosovic; reviewed by Jon Meredith for CASSANDRA-17493 commit 47cac5c49b93d205fa9b3a57ce55470887c5be45 Author: Josh McKenzie Date: Tue Mar 22 11:35:36 2022 -0400 Provide JMX endpoint to allow transient logging of blocking read repairs Patch by Josh McKenzie; reviewed by David Capwell for CASSANDRA-17471 Co-authored-by: Aleksey Yeschenko Co-aurhoted-by: Josh McKenzie commit 72af1f75fccf877f8996da0a0d8bc1a6adcd30e0 Author: Josh McKenzie Date: Thu Mar 31 11:09:07 2022 -0400 Add guardrail for GROUP BY queries Patch by Josh McKenzie; reviewed by David Capwell and Andres de la Pena for CASSANDRA-17509 commit 634131961af9c1d88b34797c1c45000f71a76dae Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Mon Apr 11 22:32:55 2022 -0400 make pylib PEP and pylint compliant commit 8d324f2aeb0e0871504ecac702b5b37f758dccb0 Merge: b73b9bbd11 d1270c204f Author: Brandon Williams Date: Wed Apr 13 07:11:18 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit b73b9bbd11a9da8321ff4f768f9a58d79eb88292 Author: Brandon Williams Date: Mon Apr 4 14:40:09 2022 -0500 Add procps Debian dependency Patch by brandonwilliams; reviewed by bereng for CASSANDRA-17516 commit a76c7c1f1cd7b4703882c9c6576a52c1201ea9b3 Author: Brandon Williams Date: Mon Apr 4 14:39:46 2022 -0500 Add procps-ng RPM dependency Patch by Dylan Richardson; reviewed by brandonwilliams and bereng for CASSANDRA-17516 commit 2fe1c304835ef39093b70cbb89107383be3c3ee9 Author: Bernardo Botella Corbi Date: Tue Apr 12 11:06:41 2022 -0700 Fix flaky test LocalReadSizeWarningTest#failThresholdSinglePartition patch by Bernardo Botella Corbi; reviewed by Andres de la Peña, Yifan Cai for CASSANDRA-17217 commit 6013f16de7ce79be30fbae29d8562b78e11cd325 Merge: 20175bf77e 1ebdc7e758 Author: David Capwell Date: Tue Apr 12 11:54:09 2022 -0700 Merge branch 'cassandra-4.0' into trunk commit 20175bf77e2c6f72c25240ee445b583805a37630 Author: Savni Nagarkar Date: Tue Mar 29 14:40:55 2022 -0500 Remove guardrails global enable flag patch by Savni Nagarkar; reviewed by Andrés de la Peña and Joshua McKenzie for CASSANDRA-17499 commit e3b3a59c6c2832bcbd3850340f0138e876cfcb5d Merge: 2fde9af74f ce6a65cb29 Author: Jon Meredith Date: Fri Apr 8 14:39:37 2022 -0600 Merge branch 'cassandra-4.0' into trunk commit 2fde9af74f12179fd8b3d592adc9a24d8b2d8f23 Author: David Capwell Date: Fri Apr 8 10:16:41 2022 -0700 Clients using JMX are unable to handle non-standard java types but we leak this into our interfaces patch by David Capwell; reviewed by Alex Petrov, Andres de la Peña for CASSANDRA-17527 commit 537321e9c7d2696cddc35e808a48846cb67ba52a Author: Marcus Eriksson Date: Thu Apr 7 09:06:53 2022 +0200 Remove stress server functionality Patch by marcuse; reviewed by Brandon Williams for CASSANDRA-17535 commit af90c3634007ba9b2967267d6f8383d38cdff85f Author: Caleb Rackliffe Date: Tue Apr 5 15:20:28 2022 -0500 Reduce histogram snapshot long[] allocation overhead during speculative read and write threshold updates patch by Caleb Rackliffe; reviewed by Josh McKenzie for CASSANDRA-17523 commit 910bbb9b5fced249696930c2e38240a9b0f51bb5 Author: Josh McKenzie Date: Tue Mar 29 14:18:10 2022 -0400 Add guardrail to disallow creation of secondary indexes Patch by Josh McKenzie; reviewed by Chris Lohfink for CASSANDRA-17498 commit 105d69fdc531d0882f628cd4bf1e34288280c12b Author: Josh McKenzie Date: Wed Mar 30 14:19:07 2022 -0400 Add guardrail to disallow creation of uncompressed sstables Patch by Josh McKenzie; reviewed by David Capwell for CASSANDRA-17504 commit 01c4404fff8c3dde071d4b9d0327acbb299be688 Author: Josh McKenzie Date: Fri Apr 1 14:34:03 2022 -0400 Add guardrail to disallow creation of new COMPACT STORAGE tables Patch by Josh McKenzie; reviewed by Caleb Rackliffe for CASSANDRA-17522 commit 828d98e9d7643501ee3ac9fa45d46d0b596cfb11 Author: David Capwell Date: Wed Apr 6 11:52:48 2022 -0700 repair vtables should expose a completed field due to lack of filtering options in CQL patch by David Capwell; reviewed by Chris Lohfink, Zhao Yang for CASSANDRA-17520 commit 885836bc492a09bba2f11cba3cddb7ab15a00dc1 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Tue Apr 5 16:10:00 2022 -0400 remove outdated code from cqlsh Patch by Brad Schoening; reviewed by brandonwilliams and smiklosovic for CASSANDRA-17490 commit 9711cd33c4d0d4a69279c8a79d8a8460accf1f12 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Tue Apr 5 15:09:14 2022 -0400 remove support for deprecated version specific TLS in Python 3.6 patch by Brad Schoening; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17365 commit 9d9c6d2750babc746f33f4b54f8f60a11d2ede86 Author: Bereng Date: Wed Apr 6 09:40:13 2022 +0200 Ninja fix changes.txt patch by Berenguer Blasi; reviewed by Berenguer Blasi for CASSANDRA-16916 commit f28dd90feb215db85ac2e510c5657a49edd46e12 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Sat Mar 26 14:47:23 2022 -0400 resolve several pylint issues in cqlsh.py and pylib patch by Brad Schoening; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17480 commit e137c7c34cc10d4f8c9124c6f9af17a44061ab1f Author: Jogesh Anand Date: Sun Nov 14 14:45:35 2021 -0800 Add support for IF EXISTS and IF NOT EXISTS in ALTER statements patch by Jogesh Anand; reviewed by Benjamin Lerer, Berenguer Blasi for CASSANDRA-16916 commit bdde665032679bd197566cfeea34b52538da4f1a Author: Bereng Date: Tue Mar 29 08:13:04 2022 +0200 Adding docs for pre hashed passwords patch by Berenguer Blasi; reviewed by Andres de la Peña for CASSANDRA-17494 commit fa7185ef02c313ff322fcc50d2de03186b9f96a7 Author: Jon Meredith Date: Fri Apr 1 12:58:02 2022 -0600 Streaming sessions longer than 3 minutes fail with timeout patch by Jon Meredith; reviewed by Benedict Elliott Smith, Caleb Rackliffe for CASSANDRA-17510 commit ffa1e9cf5a5646707da282460fe765d56fa78cb0 Author: David Capwell Date: Fri Apr 1 07:53:41 2022 -0700 Add ability to track state in repair patch by David Capwell; reviewed by Zhao Yang for CASSANDRA-15399 commit d70f6e0d597e888b3629c6c01d05685cf79a3d12 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Thu Mar 31 13:16:22 2022 -0400 remove unused 'parse' module Patch by Brad Schoening; reviewed by brandonwilliams and smiklosovic for CASSANDRA-17484 commit d44f527d70f0a8d254b7f76a8aa1b262618da5d5 Merge: 9808a4d8ed c22accc464 Author: Brandon Williams Date: Thu Mar 31 07:22:09 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 9808a4d8ed9c4319dac79dd1f13e2328e64b0a20 Author: Ruslan Fomkin Date: Wed Mar 16 15:17:22 2022 +0100 CASSANDRA-17443 Enable testUnwriteableFlushRecovery of CommitLogTest patch by Ruslan Fomkin; reviewed by Berenguer Blasi, Branimir Lambov for CASSANDRA-17443 commit 1a4dccd3b9f9bfefbccbbe383982306d3aeea1d1 Author: Josh McKenzie Date: Tue Mar 29 12:30:05 2022 -0400 Fix AlterTest.testCreateAlterNetworkTopologyWithDefaults Patch by Josh McKenzie; reviewed by Brandon Williams for CASSANDRA-17496 commit 768bdffe5298b937bfafc2eb42fb93454cfca521 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Thu Mar 10 21:18:30 2022 -0500 change six functions in cqlshlib to native Python 3 ones patch by Brad Schoening; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17417 commit b9d9f9c7dab9b39fb27a4be0b6d9ce2a8718167e Author: David Capwell Date: Tue Mar 29 11:40:33 2022 -0700 Ninja: CASSANDRA-17478 added UpdateSystemAuthAfterDCExpansionTest to the wrong directory which broke CI, moved it to the right directory commit 57ab3afcf16970047d3df4656241cf0705e94bee Author: Caleb Rackliffe Date: Mon Mar 14 17:13:47 2022 -0500 reduce hot-path object allocations required to record local/remote requests against the client request metrics patch by Caleb Rackliffe; reviewed by Jon Meredith for CASSANDRA-17424 Co-authored-by: Marcus Eriksson Co-authored-by: Caleb Rackliffe commit c6cf374c9636020bb6ee428aa88a8736d4a9f115 Merge: 7b58b79fe2 897a2519bf Author: Brandon Williams Date: Tue Mar 29 07:15:59 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 7b58b79fe24c612f8bbf7984c325e88496f482d6 Merge: 813e6ae4a0 0089858819 Author: Caleb Rackliffe Date: Mon Mar 28 15:35:25 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 813e6ae4a09b7172f3be276637ba7e29f6463405 Author: Ekaterina Dimitrova Date: Mon Mar 28 15:59:10 2022 -0400 Revert "change six functions in cqlshlib to native Python 3 ones" This reverts commit 7275443eca804f5dde405f45811441dbc0c1f9fb. commit 3d9365096bc579d10e417278576d650611105120 Author: Josh McKenzie Date: Wed Mar 23 12:42:36 2022 -0400 Disallow removal of a DC from system_auth replication settings Patch by Josh McKenzie; reviewed by Jon Meredith for CASSANDRA-17478 Co-authored-by: Josh McKenzie Co-authored-by: Nachiket Patil commit 8a3b9260a9494af56356f4c9829c4068b7ea182f Author: Andrés de la Peña Date: Fri Feb 11 18:01:43 2022 +0000 Add guardrail for the number of fields per UDT patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova for CASSANDRA-17385 commit 7c1ed6c4092baf34bbfeda5c71b5b2ac25961ff1 Merge: cd5f215c72 d0458abdb3 Author: Stefan Miklosovic Date: Mon Mar 28 14:18:31 2022 +0200 Merge branch 'cassandra-4.0' into trunk commit cd5f215c72dde43d367c065cdc9cd18d940623bd Author: Stefan Miklosovic Date: Fri Mar 25 21:37:45 2022 +0100 ninja: expand user paths for cqlsh.py as part of CASSANDRA-17448 patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-17448 commit dc521cca432a4a7f05cbd3c852c029536c123589 Merge: cd870fe645 4e19c94951 Author: Brandon Williams Date: Fri Mar 25 15:02:15 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit cd870fe645305d934c5f1799c4c7c9e6b8b7d1e3 Merge: 298abb4586 05fa92475c Author: Andrés de la Peña Date: Fri Mar 25 13:13:42 2022 +0000 Merge branch 'cassandra-4.0' into trunk commit 298abb458638a1ec41463e934d3dff2f4124ad69 Merge: e9ca465468 60269bff2b Author: Brandon Williams Date: Fri Mar 25 06:17:17 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit e9ca46546875f0e1737ed88a6c5823b9b127c0bb Author: Stefan Miklosovic Date: Mon Mar 21 23:26:03 2022 +0100 allow users to change cqlsh history location using env variable patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-17448 commit 10103d353c3d63505cb119cda0e38e692cdb1809 Author: Josh McKenzie Date: Thu Sep 30 12:10:46 2021 -0400 Add required -f option to nodetool verify Patch by Josh McKenzie; reviewed by Marcus Eriksson, Berenguer Blasi, and Caleb Rackliffe for CASSANDRA-17017 Co-authored by Josh McKenzie Co-authored by Chris Lohfink commit a64e218d0f2403cb3e5e45eb9e1455138226e5c1 Merge: 0040fea379 4f5feac0df Author: Brandon Williams Date: Wed Mar 23 09:22:47 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 0040fea3797ea3e497691e9d1e2660711c60ac4d Author: Jacek Lewandowski Date: Mon Jan 24 11:51:13 2022 +0100 Implement sstable generation identifier as uuid Patch by Jacek Lewandowski; reviewed by Andrés de la Peña, Benjamin Lerer and Dan Jatnieks for CASSANDRA-17048 commit 30ad754d7e95501ffa916bf986e4cfda1aa5e441 Author: Bereng Date: Tue Mar 1 07:44:19 2022 +0100 Pre hashed passwords in CQL patch by Robert Stupp and Berenguer Blasi; reviewed by Andres de la Peña for CASSANDRA-17334 Co-authored-by: Robert Stupp Co-authored-by: Berenguer Blasi commit 94580a272e77439e4150926a1bd1e723a36236e6 Author: Jon Meredith Date: Tue Mar 22 15:14:00 2022 -0600 Test Failure: org.apache.cassandra.db.lifecycle.LogTransactionTest.testGetTemporaryFilesSafeAfterObsoletion-cdc patch by Jon Meredith; reviewed by Josh McKenzie for CASSANDRA-17286 commit fbd6b047915d9810bf5e3131ba11c9374b3834a3 Author: Josh McKenzie Date: Tue Mar 22 12:41:58 2022 -0400 Log largest memtable on flush at info rather than debug Patch by Josh McKenzie; reviewed by Brandon Williams for CASSANDRA-17472 commit 305cb940c3be63f2eb74a745d60d70abaa0d2e2c Merge: 94c2ad5304 592cee3064 Author: Brandon Williams Date: Tue Mar 22 07:42:08 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 94c2ad5304b687cf78a7ddbc660a2651c9d04dc1 Merge: 6a3edecd21 b465fec770 Author: Brandon Williams Date: Tue Mar 22 07:32:13 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 6a3edecd21eba5001ebe5ffcf5397fc8a6c7c1e3 Merge: 952263dcea a14700bf76 Author: Erick Ramirez Date: Tue Mar 22 11:13:23 2022 +0000 Merge branch 'cassandra-4.0' into trunk commit 952263dcea4f8d8072f03a341a03d001be0962ea Author: Sathyanarayanan Saravanamuthu Date: Sat Mar 19 23:48:48 2022 -0700 removing unused Python packages in pylib patch by Sathyanarayanan Saravanamuthu; reviewed by Brandon Williams and Stefan Miklosovic for CASSANDRA-17449 commit 4ea3e4c5050ba11a5b7897af74bb54e7e8dad068 Author: Caleb Rackliffe Date: Mon Mar 14 16:21:12 2022 -0500 Add native transport rate limiter options to example cassandra.yaml, and expose metric for dispatch rate patch by Caleb Rackliffe; reviewed by Josh McKenzie for CASSANDRA-17423 commit 143a5e8b064e442970182cfb349b4f0826683e85 Author: Andrés de la Peña Date: Thu Mar 3 18:17:38 2022 +0000 Add diagnostic events for guardrails patch by Andrés de la Peña; reviewed by Berenguer Blasi and Stefan Miklosovic for CASSANDRA-17197 Co-authored-by: Andrés de la Peña Co-authored-by: Aleksandr Sorokoumov commit 2f84f492a21ee4e6bfc581c07a5817591a783f76 Author: Bereng Date: Mon Mar 21 13:19:31 2022 +0100 Revert "Pre hashed passwords in CQL" This reverts commit 6946b304e94a8a8d1250680664ddc03b61a139c9. commit 6946b304e94a8a8d1250680664ddc03b61a139c9 Author: Bereng Date: Tue Mar 1 07:44:19 2022 +0100 Pre hashed passwords in CQL patch by Berenguer Blasi; reviewed by Andres de la Peña for CASSANDRA-17334 commit 212a76233ff5a5944faa86b26e976d5694c8abe4 Author: Mick Semb Wever Date: Mon Mar 21 08:33:10 2022 +0100 ninja-fix update .build/build-rat.xml commit 9711f541b33ec2043de04cd4c47cafd122538b4a Author: Mick Semb Wever Date: Sun Mar 6 12:27:05 2022 +0100 Limit GH pull requests to rebases patch by Mick Semb Wever; reviewed by Benjamin Lerer, Berenguer Blasi, Brandon Williams, Ekaterina Dimitrova, Stefan Miklosovic for CASSANDRA-17419 commit 8714f926857ac1507b69957139fe0429e4979c39 Merge: e417fea126 d48df9169d Author: Stefan Miklosovic Date: Sun Mar 20 15:30:08 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit e417fea126f940b6d1f2a2fa22274aa4cf63b24f Merge: d67be0def4 f8be6ea60c Author: Jon Meredith Date: Sat Mar 19 08:03:59 2022 -0600 Merge branch 'cassandra-4.0' into trunk commit d67be0def4085863a039d5d3809a9457e883919b Author: Ekaterina Dimitrova Date: Thu Feb 17 18:51:22 2022 -0500 Add docs about configuration framework added in CASSANDRA-15234 patch by Ekaterina Dimitrova; reviewed by Michael Semb Wever and Caleb Rackliffe for CASSANDRA-17246 commit 34a7eba4b8d1813ad782fe2432cdfd85fa24e6c1 Merge: 96cecdbaa6 e773bbd9c6 Author: Benjamin Lerer Date: Fri Mar 18 15:03:38 2022 +0100 Merge branch cassandra-4.0 into trunk commit 96cecdbaa685b86bd2ac1d10545f9b9a39bbfdea Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Wed Mar 16 14:45:05 2022 -0400 Increase cqlsh version to 6.1.0 Patch by Brad Schoening; reviewed by bereng and brandonwilliams for CASSANDRA-17432 commit 302c6fcf59c5fa9fcd475b2c811ad57ae7c45f49 Author: Caleb Rackliffe Date: Thu Mar 10 17:12:47 2022 -0600 Update SUPPORTED_UPGRADE_PATHS to include 3.0 and 3.x to 4.1 paths and remove obsolete tests patch by Caleb Rackliffe; reviewed by Mick Semb Wever for CASSANDRA-17362 commit f1c1694e4329a1f6dba8b978edaed4312e235116 Author: Doug Rohrer Date: Thu Mar 17 10:26:29 2022 -0700 Support DELETE in CQLSSTableWriter patch by Doug Rohrer, Eric Evans; reviewed by Sam Tunnicliffe, Yifan Cai for CASSANDRA-14797 commit 69138553859277f9beff217a463703d36bca2e29 Merge: 431c692b88 f7009cf340 Author: Stefan Miklosovic Date: Thu Mar 17 12:57:50 2022 +0100 Merge branch 'cassandra-3.11' into trunk commit 431c692b884395b16beadac1d10f7e674b3e7ed5 Author: Jon Meredith Date: Tue Mar 15 16:17:43 2022 -0600 Failed inbound internode authentication failures generate ugly warning with stack trace patch by Jon Meredith; reviewed by David Capwell for CASSANDRA-17068 commit 7275443eca804f5dde405f45811441dbc0c1f9fb Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Thu Mar 10 21:18:30 2022 -0500 change six functions in cqlshlib to native Python 3 ones patch by Brad Schoening; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17417 commit 61be4d836213f708d9a29e59b9ef1df0bebef29a Author: Francisco Guerrero Date: Wed Mar 16 01:31:00 2022 +0100 expose gossip information in system_views.gossip_info virtual table patch by Francisco Guerrero; reviewed by Stefan Miklosovic and Yifan Cai for CASSANDRA-17002 This commit adds a new virtual table that exposes the gossip information in tabular format. The information is the same as the information presented through the `nodetool gossipinfo` command, but the virtual table splits the version and value from `VersionedValue` into two different columns. This is intented to help clients reading the vtable without the need of parsing the version:value information (as it currently stands in gossipinfo). The token value does not have a column. This is consistent with the gossipinfo output which always renders ":" for the Token value. Only the token_version column is available. commit 935bcf1e8732a4138c15205896945c2f02ddb844 Author: Andrés de la Peña Date: Tue Mar 15 13:31:41 2022 +0000 Add guardrails for collection items and size patch by Andrés de la Peña; reviewed by Berenguer Blasi for CASSANDRA-17153 commit 56ec75cc6524e7c514c64a1db839d9da738e42b6 Author: Jacek Lewandowski Date: Tue Mar 15 06:37:16 2022 +0100 Fix file overwriting in PathUtils Patch by Jacek Lewandowski, reviewed by Benedict for CASSANDRA-17439 commit 6f2075acee9ef0041a043917497f59fec194d8ac Merge: 951645a1fc f3123a6538 Author: Stefan Miklosovic Date: Tue Mar 15 10:58:24 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 951645a1fc4486f2b941c4eb72c429780badca07 Merge: 3b1ce69fc1 5f50c797bc Author: Paulo Motta Date: Mon Mar 14 17:21:51 2022 -0300 Merge branch 'cassandra-4.0' into trunk commit 3b1ce69fc1609538be8eac81c4ac4da12af49bff Merge: 0ecd48001f d9bd03552d Author: Brandon Williams Date: Mon Mar 14 14:09:31 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 0ecd48001fab88d2d53c717ef47cd3bc0d1369d0 Author: Andrés de la Peña Date: Fri Mar 11 17:49:17 2022 +0000 Improve guardrails messages Guardrail messages include a specific mention of the guardrail framework and the name of the violated guardrail, so users know what is producing the wanrning or failure. Also, guardrails throw their own type of exception. patch by Andrés de la Peña; reviewed by Josh McKenzie for CASSANDRA-17430 commit 7b91e4cc18e77fa5862864fcc1150fd1eb86a01a Author: Caleb Rackliffe Date: Thu Mar 10 16:19:45 2022 -0600 Remove all usages of junit.framework and ban them via Checkstyle patch by Caleb Rackliffe; reviewed by Marcus Eriksson for CASSANDRA-17316 Co-authored-by: Marcus Eriksson commit a41040ccdcec651bffb4d23843ab9be2d96ba1d1 Author: Andrés de la Peña Date: Fri Mar 11 15:36:39 2022 +0000 Add guardrails for read/write consistency levels patch by Andrés de la Peña; reviewed by Caleb Rackliffe for CASSANDRA-17188 Co-authored-by: Andrés de la Peña Co-authored-by: Aleksandr Sorokoumov commit 2b2c6decfafc6235ad537e72073fab2fd4467e2f Author: Jacek Lewandowski Date: Thu Sep 30 09:50:27 2021 +0200 Refactor schema management to allow for schema source pluggability Patch by Jacek Lewandowski, reviewed by Alex Petrov for CASSANDRA-17044. commit 51e6f8ecc122279ea946b08c528e11105efb40f5 Author: Ekaterina Dimitrova Date: Thu Mar 10 20:38:39 2022 -0500 ninja fix empty line required in test/conf/cassandra.yaml, otherwise cdc and compression unit tests fail commit d2923275e360a1ee9db498e748c269f701bb3a8b Author: Benedict Elliott Smith Date: Mon Mar 30 15:24:12 2020 +0100 CEP-14: Paxos Improvements This work encompasses a range of improvements to Paxos, summarised as: - The introduction of Paxos Repair for ensuring consistency during range movements - The reduction of network round-trips by a factor of 2x for reads and writes patch by Benedict Elliott Smith and Blake Eggleston; reviewed by Alex Petrov, Aleksey Yeschenko and Sam Tunnicliffe for CASSANDRA-17164 commit 8960b4d8513e22c4c11c181c520380342935aee7 Author: Benedict Elliott Smith Date: Tue Nov 30 17:00:45 2021 +0000 [CASSANDRA-17177] List Appends Are Not Linearizable commit 5a0d6154d144b958404a4d7117602d35157848ba Author: Benedict Elliott Smith Date: Tue Nov 30 12:12:32 2021 +0000 [CASSANDRA-17176] Introduce TimeUUID patch by Benedict; reviewed by Blake Eggleston for CASSANDRA-17176 commit eb960b19170e36bd9c52365ef15514a4d1ed9419 Merge: 852d7b667d bdeafa0b79 Author: Andrés de la Peña Date: Thu Mar 10 13:28:47 2022 +0000 Merge branch 'cassandra-4.0' into trunk commit 852d7b667d28e5dbd7f20a3a16234b30da07ff56 Author: Stefan Miklosovic Date: Wed Mar 9 16:51:52 2022 +0100 ninja: move nosetests.xml back to .jenkins/Jenkinsfile nosetests.xml were removed accidentally under CASSANDRA-17397 commit 60675cc2759db0c5629604279e70c51e10dfefd6 Author: Marcus Eriksson Date: Wed Mar 2 11:33:39 2022 +0100 Remove and ban use of Instant.now Patch by marcuse; reviewed by Benedict Elliott Smith for CASSANDRA-17414 commit ca218cc012dd8a345259bd75e945eeebdf77eef1 Merge: da2b35b776 04c1289bb9 Author: Brandon Williams Date: Tue Mar 8 17:23:30 2022 -0600 Merge branch 'cassandra-4.0' into trunk commit da2b35b776fe5304611707b4fe3f65462346755f Merge: c0c0f61869 59ea4f8efd Author: Stefan Miklosovic Date: Tue Mar 8 21:49:26 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit c0c0f6186929068b87b5e0e3018e931f11fff3f1 Merge: 3233c82311 d9766f5ade Author: Stefan Miklosovic Date: Tue Mar 8 17:19:18 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 3233c823116343cd95381790d736e239d800035a Author: Andrés de la Peña Date: Tue Mar 8 11:08:29 2022 +0000 Add guardrail for SELECT IN terms and their cartesian product patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova for CASSANDRA-17187 Co-authored-by: Aleksandr Sorokoumov Co-authored-by: Andrés de la Peña commit ac34f28e31bbc294db80e2b5d06f8904a7f41a46 Author: Bereng Date: Tue Mar 8 11:26:44 2022 +0100 Documentation update of virtual table system_views.clients patch by Tibor Repasi; reviewed by Berenguer Blasi, Ekaterina Dimitrova for CASSANDRA-17344 commit 537d02d25f1953f1907d44106f83874ac73e06b4 Author: Tibor Répási Date: Tue Jan 4 17:33:31 2022 +0100 Expose all client options via system_views.clients and nodetool clientstats patch by Tibor Repasi reviewed by Benjamin Lerer, Berenguer Blasi, Ekaterina Dimitrova for CASSANDRA-16378 commit 27ab63f00501a6afda94e760efab8af3e90c598b Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Fri Mar 4 12:02:25 2022 +0100 remove unused imports in cqlsh.py and cqlshlib patch by Brad Schoening; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17413 commit ae50cbd1ad2120bb070489e7b94d1f8ca393a4b9 Author: Stefan Miklosovic Date: Fri Mar 4 15:11:06 2022 +0100 deprecate property windows_timer_interval patch by Stefan Miklosovic; reviewed by Ekaterina Dimitrova and Berenguer Blasi for CASSANDRA-17404 commit 0d8126dd143898588f4efcdc40b8e2bb10597185 Merge: ac4cd09cce a690f339ab Author: Andrés de la Peña Date: Mon Mar 7 16:30:02 2022 +0000 Merge branch 'cassandra-4.0' into trunk commit ac4cd09cce2886f3e7393cdcd1321152140ec6ec Merge: df9586da33 f1c7bf9a54 Author: Erick Ramirez Date: Sat Mar 5 11:26:21 2022 +1100 Merge pull request #1480 from ErickRamirezAU/CASSANDRA-17361-trunk Fixed title on STCS page, broken links on compaction page (trunk) commit f1c7bf9a54e639666bacdadc28d8665d1ddd13d3 Author: Erick Ramirez Date: Fri Mar 4 08:33:56 2022 +0000 Fixed title on STCS page, broken links on compaction page (trunk) patch by Erick Ramirez; reviewed by Matt Fleming for CASSANDRA-17361 commit df9586da3323bcd817b658ed7cab8dc94a66c20d Merge: c408931aa2 fd5e2ad74e Author: Yifan Cai Date: Fri Mar 4 10:43:49 2022 -0800 Merge branch 'cassandra-4.0' into trunk commit c408931aa262353845c926f367ad06d27d9f164e Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Tue Mar 1 21:18:19 2022 -0500 fix warnings in pylib tests patch by Brad Schoening; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17397 commit 754fc5d075a8c969708551851235901d9db1394a Author: Brandon Williams Date: Wed Mar 2 13:35:00 2022 -0600 Fix bad merge one last time. commit 2f3923a44f662bb9664c34711f847dc9970f10ea Merge: 7daa65675f 0e249fd15e Author: Brandon Williams Date: Wed Mar 2 11:21:13 2022 -0600 Merge branch 'cassandra-4.0' into trunk commit 7daa65675f01a84788af178af5b7c08de14423e3 Author: Brandon Williams Date: Wed Mar 2 08:31:15 2022 -0600 Fix changes commit 3e6bbfb854697791be55c13ed3bfa578e400c9b4 Author: Brandon Williams Date: Wed Mar 2 08:28:35 2022 -0600 fix bad merge commit a8bbd22d3791c5d0b696f8c0ef5cedb4e82f01b5 Author: Dmitry Potepalov Date: Wed Feb 9 11:37:27 2022 +0100 Fix ignored streaming encryption settings in sstableloader Patch by Dmitry Potepalov; reviewed by bereng and brandonwilliams for CASSANDRA-17367 Encryption settings were silently lost in BulkLoadConnectionFactory. Forced the bulk loader test to do some actual streaming besides just handshaking on the native port, the test does not pass without the change in BulkLoadConnectionFactory. commit e87a1e0c0a19c64ed2edc2d340c0f8af16776e2c Author: David Capwell Date: Tue Mar 1 13:15:18 2022 -0800 Expose streaming as a vtable patch by David Capwell; reviewed by Dinesh Joshi, Paulo Motta for CASSANDRA-17390 commit 445e49a0c8e2fb52bf9e4370efd7f0495bc8e00a Merge: e780b5a45b 4fa541705c Author: Brandon Williams Date: Mon Feb 28 07:47:20 2022 -0600 Merge branch 'cassandra-4.0' into trunk commit e780b5a45b829f89049ad358a36be3055dbcd344 Author: Bereng Date: Wed Feb 2 09:52:49 2022 +0100 Error out on noop GRANT/REVOKE patch by Berenguer Blasi; reviewed by Andres de la Peña for CASSANDRA-17333 commit eac9d925621dc1757ac88ac7160dc0b2bd3e0015 Author: Stefan Miklosovic Date: Fri Feb 18 14:40:08 2022 +0100 make startup checks configurable patch by Stefan Miklosovic; reviewed by Paulo Motta for CASSANDRA-17220 commit b83da7d978f1bae5f289f1561dd080d48dc4505b Author: Yifan Cai Date: Fri Feb 25 14:38:43 2022 -0800 Correct size unit to mebibypes commit e40f8163af9dcc889b81265b17e66f36e25fc727 Author: Ekaterina Dimitrova Date: Fri Feb 25 16:07:33 2022 -0500 Fix CHANGES.txt commit 75bea27e5ef673ae7fcfac9007c1d39a68ab0b8c Merge: 12e2c159f8 a8b67e3dd9 Author: Ekaterina Dimitrova Date: Fri Feb 25 15:47:43 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 12e2c159f86b223df1c2018c8a395a805b11956e Author: Ekaterina Dimitrova Date: Fri Feb 11 19:09:30 2022 -0500 Forward port from cassandra-4.0(CASSANDRA-17377) - deprecation of otc_coalescing_strategy, otc_coalescing_window_us, otc_coalescing_enough_coalesced_messages, otc_backlog_expiration_interval_ms patch by Ekaterina Dimitrova; reviewed by David Capwell for CASSANDRA-17378 commit 65fcc79b628a7e415b57c27c596ea1cb5e67211d Author: Ekaterina Dimitrova Date: Mon Feb 14 19:12:26 2022 -0500 Revert removal of deprecated max_streaming_retries into Config, revert name change to user_defined_function_warn_timeout and user_defined_function_fail_timeout patch by Ekaterina Dimitrova; reviewed by David Capwell for CASSANDRA-17378 commit 769f6b3122629d8422b347d13d7f573c56173c55 Merge: 08331f474b 22a4ae16e4 Author: Ekaterina Dimitrova Date: Fri Feb 25 11:50:44 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 08331f474bb03bc87c5fcfb5458ab388d01e0b9d Merge: 7ea0a4aa41 6d0db649ca Author: Erick Ramirez Date: Fri Feb 25 17:43:52 2022 +1100 Added doc for PEM-based SSL support in CASSANDRA-17031 patch by Maulin Vasavada; reviewed by Stefan Miklosovic, Erick Ramirez for CASSANDRA-16950 commit 7ea0a4aa410e9d0d252115607ac5c31d37784357 Author: tejavadali Date: Wed Feb 23 12:42:42 2022 +0000 Add guardrail for number of partition keys on IN queries patch by Krishna Vadali; reviewed by Andrés de la Peña and Berenguer Blasi for CASSANDRA-17186 commit ab7ee5001301925495105d8372cd9a8f965da0a6 Author: Stefan Miklosovic Date: Mon Feb 21 14:54:55 2022 +0100 ninja for CASSANDRA-17293 - moving cqlshlib.xml back commit db3583318258b14b27abe1529f8893da2ff22bf5 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Fri Feb 11 20:35:18 2022 -0500 update Python test framework from nose to pytest patch by Brad Schoening; reviewed by Brandon Williams, Berenguer Blasi and Stefan Miklosovic for CASSANDRA-17293 commit d96c32b0b36d732df898f1d1a732c9398c2d775b Merge: c08baf20c6 5bc9f7c736 Author: Aleksei Zotov Date: Sat Feb 19 13:39:52 2022 +0400 Merge branch 'cassandra-4.0' into trunk commit c08baf20c67cb16eeac00f2bcb23dd87384e7701 Author: Mick Semb Wever Date: Sat Feb 19 00:07:39 2022 +0100 ninja-fix CHANGES.txt so >4.0.0 entries are "Merged in …" entries commit f5f804527e0587216b438b4be60ac87c799e9f1f Merge: adf2f4c83a 85fd49f2cf Author: Mick Semb Wever Date: Fri Feb 18 11:19:07 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit adf2f4c83a2766ef8ebd20b35b49df50957bdf5e Merge: c69e37eca7 f8b3f60dbe Author: Aleksei Zotov Date: Thu Feb 17 23:03:03 2022 +0400 Merge branch 'cassandra-4.0' into trunk commit c69e37eca76d96c331807ea23280945afecad1e9 Merge: 9a8dc6d75f 702824a99d Author: Marcus Eriksson Date: Thu Feb 17 10:39:33 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 9a8dc6d75fd1a4b6e50751ec7ee96114ba5772e9 Merge: 8327744c1a 894d33ce40 Author: Mick Semb Wever Date: Thu Feb 17 09:27:07 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 8327744c1aca0d9471cd8a01a1c7d79365363816 Author: Yifan Cai Date: Wed Feb 16 19:29:35 2022 -0800 Fix improper CDC commit log segments deletion in non-blocking mode patch by Yifan Cai; reviewed by Josh McKenzie for CASSANDRA-17233 commit a725d2c4c8ca6d0e914d3dce3ea90ae495fca2c8 Author: Bereng Date: Tue Feb 15 12:20:16 2022 +0100 Failing test: UNIT org.apache.cassandra.tools.BulkLoaderTest.testBulkLoader_WithArgs1 patch by Berenguer Blasi; reviewed by Brandon Williams for CASSANDRA-17283 commit d70e004975e47de750449557a714b1db799c10a1 Merge: e56bf105a1 b837a586f1 Author: Brandon Williams Date: Wed Feb 16 17:28:09 2022 -0600 Merge branch 'cassandra-4.0' into trunk commit e56bf105a161b1d9677ea4c9c70436f3169edb9c Merge: 81922c5a7b 2687cf1edf Author: Marcus Eriksson Date: Wed Feb 16 11:09:55 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 81922c5a7bcbf9db7564a29922c9d8f6222c7cdc Author: Alex Petrov Date: Tue Oct 12 13:04:11 2021 +0200 Intoduce Harry to the tree Patch by Alex Petrov; reviewed by Caleb Rackliffe and Abe Ratnofsky for CASSANDRA-16262. commit 598d608f0f6b37a4e2ca547d94fc6f78aba0ab8f Merge: 0c87f2668d a87055d56a Author: Mick Semb Wever Date: Sun Feb 13 23:03:10 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 0c87f2668d902caef51a8f281c3eb3ca57bb2924 Author: Ekaterina Dimitrova Date: Wed Feb 9 16:22:46 2022 -0500 Set the config in In-JVM upgrade tests to the old format on trunk because In-JVM upgrade tests do no support per-version config. patch by Ekaterina Dimitrova; reviewed by David Capwell for CASSANDRA-17369 commit 0108dd259390fd6e0c855adac16a91879d2c24d4 Merge: 5cf62c6c02 537ad9e17b Author: Ekaterina Dimitrova Date: Sat Feb 12 20:18:16 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 5cf62c6c02322505db9260d2aa9031386326fc75 Author: Manish Ghildiyal Date: Sat Dec 18 18:26:31 2021 +0100 Add support for string concatenations through the + operator Patch by Manish Ghildiyal; review by Benjamin Lerer, Berenguer Blassi, Brandon Williams for CASSANDRA-17190 commit 9649cb1358a08f5f98413c556ef2f69fb7806c91 Merge: ab0a9fdd3f c60ad61b3b Author: Marcus Eriksson Date: Fri Feb 11 14:09:43 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit ab0a9fdd3f37e31bc335fcc075a4dec0bd721eed Merge: 4094e9ce2c f207460391 Author: Mick Semb Wever Date: Fri Feb 11 11:09:50 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 4094e9ce2c4e7e367a319fe4c6d947340888d819 Author: Yifan Cai Date: Thu Feb 10 17:24:20 2022 -0800 ninja fix CHANGES.txt commit 3a6f6907314670fdb2b316db8f08ffd85da88851 Author: Yifan Cai Date: Tue Dec 7 18:54:58 2021 -0800 Limit the maximum hints size per host patch by Yifan Cai; reviewed by Dinesh Joshi, Francisco Guerrero for CASSANDRA-17142 commit 6d0db649cabc7b177a47c39573fab52fd2570336 Author: Maulin Vasavada Date: Wed Feb 2 17:17:57 2022 -0800 add documentation for CASSANDRA-16950 and CASSANDRA-17031 patch by Maulin Vasavada; reviewed by Stefan Miklosovic for CASSANDRA-16950 commit ce7502a11067ef903cba24ad588cf0a5b9da9257 Author: Michael Burman Date: Thu Dec 23 21:01:09 2021 +0200 Add a virtual table for exposing batch metrics Patch by Michael Burman; review by Aleksei Zotov, Benjamin Lerer and Ekaterina Dimitrova for CASSANDRA-17225 commit 8e1c8f986358aee51dd565a5a2917cb5554ed0c7 Author: Andrés de la Peña Date: Wed Feb 9 18:31:37 2022 +0000 Flatten guardrails config patch by Andrés de la Peña; reviewed by David Capwell for CASSANDRA-17353 commit 543608ba39d5803b963d14821abe193ff0796b4f Author: Francisco Guerrero Date: Mon Feb 7 11:25:45 2022 -0800 Instance failed to start up due to NPE in StartupClusterConnectivityChecker patch by Francisco Guerrero; reviewed by Stefan Miklosovic, Yifan Cai for CASSANDRA-17347 commit f6169d399f79168cbc3580d95082b0e2a99db048 Author: Yash Ladha Date: Sat Feb 5 11:19:15 2022 +0530 add the shorter version of version flag (-v) in cqlsh patch by Yash Ladha; reviewed by Brandon Williams and Stefan Miklosovic for CASSANDRA-17236 commit 6d7b0a10796cfedd11c4a7a1de81bc48bf8ea3cc Author: Yash Ladha Date: Wed Jan 26 09:30:42 2022 +0530 Fix classpath file creation for eclipse Patch by Yash Ladha; reviewed by Benjamin Lerer and Berenguer Blasi for CASSANDRA-17294 When building the eclipse file for development, `.classpath` file was not correctly generated as it was not getting closed. Also there were missing third-party libs that were essential for development. This commit fixes the generation of classpath for `generate-eclipse-files` command. commit 32bdf14ec916404f5fdd4b918b7d80a7bb92aa38 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Fri Jan 28 20:48:45 2022 -0500 Remove obsolete enum34 package Patch by Brad Schoening; reviewed by brandonwilliams and paulo for CASSANDRA-17325 commit f0c97132c969cc8dd028d00c5e68ada8b0b9b9c6 Author: David Capwell Date: Tue Feb 8 13:37:12 2022 -0800 Make vtables accessible via internode messaging patch by David Capwell; reviewed by Aleksey Yeschenko for CASSANDRA-17295 commit 3655b26adf8d3b94095924920d05cc1a16d0f4c0 Author: Maulin Vasavada Date: Fri Oct 8 23:51:47 2021 -0700 Add support for PEM based key material for SSL patch by Maulin Vasavada; reviewed by Jon Meredith and Stefan Miklosovic for CASSANDRA-17031 commit 72302adec8d4ffd8d7fe1c1a3542babe9b117196 Merge: 86de77049b f2816f5a7c Author: Josh McKenzie Date: Mon Feb 7 12:47:57 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 86de77049b3cdf394f8a7d2df3ff96182e2c220b Merge: 210793f943 1923b542d0 Author: Mick Semb Wever Date: Mon Feb 7 15:09:24 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 210793f943dc522161fd26b6192f38a5c83fa131 Merge: e256b98172 66d792adda Author: Marcus Eriksson Date: Mon Feb 7 13:00:57 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit e256b981728153e13fe273c5410e46c66a4c30b4 Merge: 9f56bf4ca7 371add40ae Author: Alex Petrov Date: Mon Feb 7 11:55:39 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 9f56bf4ca7fdb61ad09e5f2ad09b87cd01e0716b Author: Ekaterina Dimitrova Date: Sat Feb 5 17:51:32 2022 -0500 Remove old Duration class in favor of DurationSpec class patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit c51a7c66fc21ca2da08b89ae5f9b4817ee4d8c23 Author: Ekaterina Dimitrova Date: Sat Feb 5 17:15:18 2022 -0500 Bulk change of units around the code to support the move to the new configuration framework patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit 6d5203615f7a9670cb1698b74123666bc25ba471 Author: Ekaterina Dimitrova Date: Fri Feb 4 00:25:14 2022 -0500 Transfer parameters to the newly introduced configuration framework (7) patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit 23138252f20891c26a3692664c6affaf99e86541 Author: Ekaterina Dimitrova Date: Thu Feb 3 23:49:50 2022 -0500 Transfer parameters to the newly introduced configuration framework (6) patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit 1315d0c96f4625a76296f58d431f97669e5178c2 Author: Ekaterina Dimitrova Date: Thu Feb 3 22:28:41 2022 -0500 Transfer parameters to the newly introduced configuration framework (5) patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit ed48f3c017c5e572a523890bcd5b7c798d7eb358 Author: Ekaterina Dimitrova Date: Thu Feb 3 16:43:36 2022 -0500 Transfer parameters to the newly introduced configuration framework (4) patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit 755fd9446b084e659e98bd7336b9e910c2e12577 Author: Ekaterina Dimitrova Date: Thu Feb 3 14:39:48 2022 -0500 Transfer parameters to the newly introduced configuration framework (3) patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit d85f7f7c2dd4b9bbdb44bc96235e6a8bc3ff3967 Author: Ekaterina Dimitrova Date: Thu Feb 3 00:19:28 2022 -0500 Transfer parameters to the newly introduced configuration framework (2) patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit b9e2ab75f8f6dedd45c6ad7a83b3160149869262 Author: Ekaterina Dimitrova Date: Wed Feb 2 12:47:41 2022 -0500 Extend DurationSpec and DataStorageSpec for smallest unit and transfer denylist parameters to the new framework patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit a3258d66bcc9f946304c19d59e75d2721126303e Author: Ekaterina Dimitrova Date: Tue Feb 1 17:14:17 2022 -0500 Transfer parameters to the newly introduced configuration framework (1) patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit 5bb4bab12f8edfef95ed13cbabf8c0f377986065 Author: Ekaterina Dimitrova Date: Mon Jan 31 21:51:49 2022 -0500 DataRate parameters transition to the new framework Fix the DB descriptorRefTest which failed on the previous commit patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit 9c6b382058578ac75b88055a13aa83944901fb88 Author: Ekaterina Dimitrova Date: Tue Dec 14 23:04:43 2021 -0500 Backward compatibility framework for configuration parameters patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit db9f7a67ec4b03413c10034956e2cf18739ca4b1 Author: Ekaterina Dimitrova Date: Tue Dec 14 23:00:56 2021 -0500 Add new custom types and unit tests for configuration patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234 commit da47849b50daa0580f2cb4264bcee8a75140eb05 Author: Stefan Miklosovic Date: Sat Feb 5 15:41:33 2022 +0100 Remove Windows-specific classes and related code patch by Stefan Miklosovic; reviewed by Joshua McKenzie, Bowen Song, Berenguer Blasi for CASSANDRA-16956 commit 28eea6e8cd4055c8d21f872c72f8bd14fd2467ba Author: Caleb Rackliffe Date: Thu Feb 3 18:11:32 2022 -0600 Runtime-configurable YAML option to prohibit USE statements patch by Caleb Rackliffe; reviewed by David Capwell for CASSANDRA-17318 commit df16b3750dc2c1b6b9bcdece6f81dfd3de7ebdfa Author: David Capwell Date: Fri Feb 4 10:15:58 2022 -0800 When streaming sees a ClosedChannelException this triggers the disk failure policy patch by David Capwell, Francisco Guerrero; reviewed by Caleb Rackliffe, Dinesh Joshi for CASSANDRA-17116 commit b968dd49e12617c23ec0a5c019b397aa165cf419 Merge: 6614c08c64 02f58d70d0 Author: Mick Semb Wever Date: Fri Feb 4 14:47:08 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 6614c08c6446079a8bd23eeb98dee1e5ca3debad Merge: 35dbcc2c2d 965fe0decf Author: Jon Meredith Date: Thu Feb 3 14:27:24 2022 -0700 Merge branch 'cassandra-4.0' into trunk commit 35dbcc2c2dbe1c826fd6ecd6e8357f0f5a9bab02 Author: Aleksandr Sorokoumov Date: Sun Oct 31 16:48:52 2021 +0100 CASSANDRA-15215 Use DataOutputPlus.writeBytes in VIntCoding.writeUnsignedVInt In the cases where VInt occupies less than or equal to 8 bytes and the underlying buffer has at least 8 bytes, VIntCoding writes the entire register in a single operation and then adjusts the buffer position. Co-authored-by: Benedict Elliott Smith Co-authored-by: Branimir Lambov commit 8b15f8f22e8126f527ba0d39b9d13b20577c0192 Author: Brandon Williams Date: Fri Jan 28 11:08:07 2022 -0600 Fix line numbering and add final step to readme patch by Brandon Williams; reviewed by Brandon Williams, Ekaterina Dimitrova for CASSANDRA-17319 commit d57d67b536185104e9c9074d08498f5d7f26c9e2 Author: Bereng Date: Fri Jan 28 06:48:59 2022 +0100 Remove Python 2.x circle tests from default cfg patch by Berenguer Blasi; reviewed by Brandon Williams, Ekaterina Dimitrova for CASSANDRA-17319 commit 945a4fc23ac1f60b8380be3b60aef89caf3daba2 Author: Shailaja Koppu Date: Tue Feb 1 09:53:49 2022 -0800 Add a virtual table for exposing prepared statements metrics patch by Shailaja Koppu; reviewed by Ekaterina Dimitrova, Francisco Guerrero, Yifan Cai for CASSANDRA-17224 commit 23e88ad0c80b5910ad5734fe29032ff579956eee Author: nvharikrishna Date: Sun Jan 9 12:35:09 2022 +0530 Remove use of deprecated File classes in tests Patch by Venkata Harikrishna Nukala; reviewed by brandonwilliams and bereng for CASSANDRA-17126 commit ab4919880793e43b361054754b65f750f9f56c2e Author: Brandon Williams Date: Mon Jan 24 12:14:46 2022 -0600 remove py2 circle tests Patch by brandonwilliams; reviewed by bereng for CASSANDRA-17242 commit 1eb777946434a578f2aa673063ba992692c358d4 Author: Brad Schoening <5796692+bschoening@users.noreply.github.com> Date: Thu Jan 20 06:19:45 2022 -0500 remove python 2 code which is now EOL and unsupported Patch by Brad Schoening; reviewed by brandonwilliams and bereng for CASSANDRA-17242 commit d543dae2cd0d6540d95eb3252d79e75393fd993d Author: Jacek Lewandowski Date: Thu Dec 16 08:45:05 2021 +0100 Fix restarting a node when other nodes are down in dtests patch by Jacek Lewandowski; reviewed by Michael Semb Wever for CASSANDRA-17214 commit ecfe7e809baf342b33fae1d6497f32f90c2eda18 Merge: 02dad368f1 67f913a706 Author: Andrés de la Peña Date: Wed Jan 26 11:50:33 2022 +0000 Merge branch 'cassandra-4.0' into trunk commit 02dad368f1dbf38363cfe1a48f089cb2a7f24262 Merge: ff4d63d392 95257c0805 Author: Bereng Date: Wed Jan 26 08:12:46 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit ff4d63d392c89e27ca6e02e38457a5199a24f6a9 Merge: 8ace479455 85248da628 Author: Bereng Date: Tue Jan 25 07:10:41 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 8ace479455baefeadd0f0df7e40b6b9c75967edf Merge: 0a6be7e412 2b63dc7044 Author: Ekaterina Dimitrova Date: Mon Jan 24 12:11:57 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 0a6be7e4128406a0e3e0f85b54c5e8cae3cd7052 Author: Ekaterina Dimitrova Date: Wed Jan 5 21:07:01 2022 -0500 Substitute ant script and scriptdef with custom ant tasks patch by Ekaterina Dimitrova; reviewed by Michaem Semb Wever for CASSANDRA-17224 commit e6b589b833918afea097e746818b7a42dfe3f1ec Merge: 9577fd36d6 e769d0d6d6 Author: Mick Semb Wever Date: Sun Jan 23 13:06:03 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 9577fd36d681266ea2d4758c51ed506e940d0d0e Merge: fac84e0a80 bccb5adf95 Author: Brandon Williams Date: Thu Jan 20 10:09:36 2022 -0600 Merge branch 'cassandra-4.0' into trunk commit fac84e0a8098d562a4c3d59b026dfc925a9da9b7 Merge: add1d1d752 8c8d9b8e57 Author: Brandon Williams Date: Thu Jan 20 10:02:42 2022 -0600 Merge branch 'cassandra-4.0' into trunk commit add1d1d75237d2ef51e7c53b88a66c80bde52550 Merge: 0dc5a289e8 cf9091a12c Author: Brandon Williams Date: Thu Jan 20 09:50:08 2022 -0600 Merge branch 'cassandra-4.0' into trunk commit 0dc5a289e8dd586150253d951e6e229480c0ffc8 Author: Francisco Guerrero Date: Fri Jan 14 16:13:00 2022 -0800 Preserve tests that use BigInt numbers Patch by Francisco Guerrero; reviewed by brandonwilliams and ycai for CASSANDRA-17133 commit e0a61f73b9b9d14db3e68aafb38257a7689557b9 Author: Francisco Guerrero Date: Fri Jan 14 15:36:35 2022 -0800 Revert "Add unix time conversion functions" This reverts commit 8ddcd43b0cfcebfda882a238532d00905fe85eb8. commit 96c80f0b89314dc805714c52dcf1846491c53cdc Merge: df1a2d4c3d 9a1bb62822 Author: Marcus Eriksson Date: Mon Jan 17 14:13:41 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit df1a2d4c3db4ff016d03d5403ac68778a71d5759 Merge: 76f8333e9b 98e798f567 Author: Marcus Eriksson Date: Mon Jan 17 09:13:32 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit 76f8333e9b9c89f029dd586273d5d42fdaf676dd Author: Brandon Williams Date: Tue Dec 7 16:32:45 2021 -0600 Use abolute path when checking file stores Patch by brandonwilliams; reviewed by bereng for CASSANDRA-17084 commit 89c6addb505392ef3434597b61f83da25d13e3e2 Author: Josh McKenzie Date: Wed Sep 15 10:52:46 2021 -0400 Prewarm role and credentials caches to avoid timeouts at startup Patch by Blake Eggleston; reviewed by Sam Tunnicliffe, Jason Brown, and Caleb Rackliffe for CASSANDRA-16958 Co-authored by: Blake Eggleston Co-authored by: Josh McKenzie commit 82edff5907c44efef7262bb48f62cae85e6256d9 Merge: ac1d9bca74 12fc714e2b Author: Ekaterina Dimitrova Date: Tue Jan 11 20:51:09 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit ac1d9bca74533b8e0a852a4d3775a8bb08c1de1e Merge: bc20bddceb aecac82027 Author: Brandon Williams Date: Tue Jan 11 08:26:08 2022 -0600 Merge branch 'cassandra-4.0' into trunk commit bc20bddcebd6a37b14cfbdd50c359be4c9743f73 Author: Aleksei Zotov Date: Mon Dec 20 21:26:41 2021 +0400 Make capacity/validity/updateinterval/activeupdate for Auth Caches configurable via nodetool patch by Aleksei Zotov; reviewed by Josh McKenzie for CASSANDRA-17063 commit 7bd276496264fa0d81ccc6990bf8ca663a0f7612 Merge: 2043cb9fb6 1a05fcf52b Author: Ekaterina Dimitrova Date: Sat Jan 8 15:32:30 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 2043cb9fb6b25ff34afb90467b9476a09acc3933 Author: Yuqi Gu Date: Thu Oct 14 02:47:10 2021 +0000 Upgrade Snappy and JNA version to support Apple M1 Patch-By Yuqi Gu; Reviewed by Dinesh Joshi and Yifan Cai for CASSANDRA-17040 commit 837adcf0a26a704b10ece4e90db267c085021342 Merge: 54aee87c56 57a7b8a70d Author: Jon Meredith Date: Thu Jan 6 14:33:33 2022 -0700 Merge branch 'cassandra-4.0' into trunk commit 54aee87c56a49a0b834b920aead7e0dbed5f5640 Merge: b83d722b99 0aad2e9e86 Author: Mick Semb Wever Date: Wed Jan 5 23:02:59 2022 +0100 Merge branch 'cassandra-4.0' into trunk commit b83d722b99de79d131f58512564b901b11907182 Merge: 9b291f18ab 3ceaed125c Author: Ekaterina Dimitrova Date: Mon Jan 3 20:33:19 2022 -0500 Merge branch 'cassandra-4.0' into trunk commit 9b291f18abfc62ab45e725effe75a8ceb9163760 Author: Kanthi Subramanian Date: Sun Dec 5 13:07:54 2021 -0500 Add startup check for read_ahead_kb setting Patch by Kanthi Subramanian; Reviewed by Paulo Motta and Brandon Williams for CASSANDRA-16436 Closes #1354 commit 481c103951ef8caa74a27dd474c94d9c9e819a46 Merge: 11952fae77 c420d754d8 Author: Paulo Motta Date: Mon Jan 3 11:57:42 2022 -0300 Merge branch 'cassandra-4.0' into trunk commit 11952fae774d51d4e268dae506803fef874c8c50 Author: Benjamin Lerer Date: Wed Dec 15 12:41:29 2021 +0100 Avoid unecessary array allocations and initializations when performing query checks patch by Benjamin Lerer; reviewed by Caleb Rackliffe for CASSANDRA-17209 commit 95522f85d5e14734f9af3096953974a4f48a884f Author: Andrés de la Peña Date: Wed Dec 15 16:31:24 2021 +0000 Add guardrail for list operations that require read before write patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova for CASSANDRA-17154 commit bae92ee139b411c94228f8fd5bb8befb4183ca9f Author: Andrés de la Peña Date: Wed Dec 8 13:28:17 2021 +0000 Migrate thresholds for number of keyspaces and tables to guardrails This adds a new guardrail for limiting the number of keyspaces. It also marks the previous not-guardrail thresholds for keyspaces and tables as deprecated in favour of the equivalent guardrails. GuardrailsOptions is modified to always log updates on guardrails config. A small refactor decouples guardrails from their configuration interfaces. patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova for CASSANDRA-17195 commit f4a2135c5ba442aafd27bb7c12c85b376d5a2b87 Author: Josh McKenzie Date: Tue Dec 14 12:01:47 2021 -0500 Remove strong ref loop in LogTransaction.SSTableTidier Patch by Josh McKenzie; reviewed by Benedict Elliott Smith for CASSANDRA-17205 commit bf1446cd85ca476ca3e6f53ed3e13e18697acfbc Author: Kowalczyk Date: Sat Dec 11 13:17:39 2021 +0000 Add guardrail for query page size patch by Bartlomiej; reviewed by Andrés de la Peña and Brandon Williams for CASSANDRA-17189 commit 75482d0a8ccd0b0d370aeb7ee60c72cd47a191b0 Author: Kowalczyk Date: Mon Dec 6 22:07:13 2021 +0100 Simplify SchemaCQLHelperTest methods Patch by Bartlomiej Kowalczyk; reviewed by Ekaterina Dimitrova and Benjamin Lerer for CASSANDRA-17181 commit 1dd9f55a60e25822e947f04bc0b1906cb3ec922d Merge: a41cdd64b2 aa82e3e427 Author: Brandon Williams Date: Tue Dec 14 13:36:35 2021 -0600 Merge branch 'cassandra-4.0' into trunk commit a41cdd64b217c451b5576abe2f455eaa7ec1f322 Author: Francisco Guerrero Date: Tue Dec 14 10:13:59 2021 -0800 Allow column_index_size_in_kb to be configurable through nodetool patch by Francisco Guerrero; reviewed by Dinesh Joshi, Yifan Cai for CASSANDRA-17121 commit c64ff69bd982e288fd9f19697cee01514ab5f838 Author: Stefan Miklosovic Date: Fri Dec 3 21:19:18 2021 +0100 Emit a metric for number of local read and write calls patch by Damien Stevenson; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-10023 commit e99a8da161ed599c1a22a853c9c7f9caf6c1eb79 Author: Brandon Williams Date: Thu Nov 18 10:58:53 2021 -0600 Don't clean when enabling FQL via JMX Patch by brandonwilliams; reviewed by bereng for CASSANDRA-17136 commit 97b47c3b5f845097181130125752bd6efc1e1e47 Merge: 507c6f7607 e73d05bf85 Author: Joseph Lynch Date: Thu Dec 9 10:30:15 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 507c6f76072fea09d62dd941c929945b2544bba9 Merge: d9460a04da 8cef32ae83 Author: Bereng Date: Thu Dec 9 07:43:26 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit d9460a04daee5fa97639abf2b6e28ff9b29cf636 Author: Yifan Cai Date: Fri Dec 3 12:18:31 2021 -0800 Add non-blocking mode for CDC writes patch by Yifan Cai; reviewed by Josh McKenzie for CASSANDRA-17001 commit 7871a2854217acc89d2f885619243a442cf68e22 Author: Andrés de la Peña Date: Wed Nov 3 10:10:02 2021 +0000 Add guardrails framework prototype patch by Andrés de la Peña; reviewed by David Capwell and Stefan Miklosovic for CASSANDRA-17147 Co-authored-by: Sylvain Lebresne Co-authored-by: Andrés de la Peña Co-authored-by: Aleksandr Sorokoumov Co-authored-by: Zhao Yang Co-authored-by: Jakub Żytka commit cede538e029e14dc6298025794c72cc100118953 Author: Jacek Lewandowski Date: Wed Nov 3 08:59:42 2021 +0100 Relax schema synchronization when opening a keyspace Patch by Jacek Lewandowski; Reviewed by Branimir Lambov and Alex Petrov for CASSANDRA-17071. commit 7fd20bff2b457ac50158db2bf71e1243fe9bb9be Merge: 8e28dc0eba 15eaa95f09 Author: Bereng Date: Fri Dec 3 11:35:56 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 8e28dc0ebac3d80db43acfe76cfb45c0cb17a5c8 Author: Josh McKenzie Date: Mon Nov 29 11:27:17 2021 -0500 Harden resource management on SSTable components to prevent future leaks Patch by Josh McKenzie; reviewed by Caleb Rackliffe and Marcus Erikkson for CASSANDRA-17174 commit 92dc415902654c0e69de47205af62b9bb4532809 Author: Alex Petrov Date: Thu Oct 28 15:35:57 2021 -0400 Tolerate local files in data dir during startup Patch by Alex Petrov; reviewed by Aleksey Yeschenko, Jon Meredith, and Caleb Rackliffe for CASSANDRA-17082 Co-authored-by: Alex Petrov Co-authored-by: Josh McKenzie commit 209c35aa62855e2834e629e8ecb38ebb329aa6ce Author: Benedict Elliott Smith Date: Wed Sep 1 15:35:22 2021 +0100 [CEP-10] Phase 4: Cluster and Code Simulation patch by Benedict; reviewed by Sam Tunnicliffe for CASSANDRA-17008 Co-authored-by: Benedict Elliott Smith Co-authored-by: Sam Tunnicliffe Co-authored-by: Alex Petrov commit eae581a5f07c340594f6af47bb558693ef363611 Author: Benedict Elliott Smith Date: Wed Nov 17 14:34:23 2021 +0000 [CEP-10] Cluster and Code Simulations: Minor improvements - Simplify Semaphore - Future improvements - ScheduledExecutorPlus improvements for simulator compatibility - Debug leaks in Ref or BufferPool - Support use of TokenMetadata without initialising Cassandra - Additional system properties and simulator flags - Permit Clock initialisation within separate ClassLoader - Introduce BallotGenerator patch by Benedict; reviewed by Sam Tunnicliffe for CASSANDRA-17008 commit 31bea0b0d41e4e81095f0d088094f03db14af490 Author: Benedict Elliott Smith Date: Wed Nov 17 14:50:09 2021 +0000 [CEP-10] Cluster and Code Simulations: Minor fixes - fix repair timeout - fix secondary index flushing - fix race condition with Ref - fix resource leaks - fix incorrect exists() - fix repair error reporting when null exception message - fix dtest use of System.nanoTime patch by Benedict; reviewed by Sam Tunnicliffe for CASSANDRA-17008 commit f47e4b294dd0512d8bec5f8cddd3142fa7c1867a Author: Bereng Date: Mon Nov 29 11:52:45 2021 +0100 Flaky GrantAndRevokeTest patch by Berenguer Blasi; reviewed by Brandon Williams for CASSANDRA-17173 commit a06eeda7f20a1afae0a65914562ce0287b45e7d6 Merge: 951d72cd92 0764273608 Author: Bereng Date: Fri Nov 26 07:17:30 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 951d72cd929d1f6c9329becbdd7604a9e709587b Author: Benedict Elliott Smith Date: Wed Apr 14 22:54:53 2021 +0100 [CEP-10] Phase 4: Support CAS Add Operations Co-authored-by: Benedict Elliott Smith Co-authored-by: Sam Tunnicliffe Co-authored-by: Caleb Rackliffe commit a965699c9e0945ad302aa605a659c665f1cc518a Author: Benedict Elliott Smith Date: Mon Apr 26 12:28:48 2021 +0100 [CEP-10] Phase 3: Use paxos variant to specify behaviour Co-authored-by: Benedict Elliott Smith Co-authored-by: Sam Tunnicliffe commit b6916017ca4d7bb2f46355ae28fb7cd40d6eafe8 Author: Benedict Elliott Smith Date: Fri Apr 16 13:50:18 2021 +0100 [CEP-10] Phase 3: Synchronized Legacy Paxos patch by Benedict Elliott Smith and Blake Eggleston; reviewed by Sam Tunnicliffe for CASSANDRA-16934 commit ed8571fe6fd988976a3e7b91ecfc4077f4549c59 Merge: dcd7b0d35f aaffb3b53e Author: Mick Semb Wever Date: Sun Nov 21 19:53:19 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit dcd7b0d35f8d0a8a6182919a6beae7274325d818 Author: David Capwell Date: Fri Nov 19 12:44:44 2021 -0800 repair prepare message would produce a wrong error message if network timeout happened rather than reply wait timeout patch by David Capwell; reviewed by Berenguer Blasi for CASSANDRA-16992 commit c15f530b63a1cd4d5b2835bb418197145beb7bb6 Author: Caleb Rackliffe Date: Fri Nov 19 12:58:02 2021 -0600 Make waiting for session event persistence more reliable in SecondaryIndexTest#test_only_coordinator_chooses_index_for_query patch by Caleb Rackliffe; reviewed by David Capwell for CASSANDRA-17165 commit 4aab2c79b9539e8d8cfe90e4dd700b8da2f9e8a0 Author: Yifan Cai Date: Mon Nov 15 16:12:08 2021 -0800 Increase the buckets count for timer histogram patch by Yifan Cai; reviewed by Caleb Rackliffe for CASSANDRA-17155 commit 8d3fd3a97c5e277c80c846d3aa7b679018e8fab5 Author: Bereng Date: Fri Nov 19 08:32:32 2021 +0100 utests_system_keyspace_directory - more than 500 tests failing on trunk patch by Berenguer Blasi; reviewed by Sam Tunnicliffe for CASSANDRA-17137 commit 10c685222fc415586ae28a01e7896063a3f2f0d3 Merge: 33fd2dc817 1bcfa087f4 Author: David Capwell Date: Thu Nov 18 17:29:23 2021 -0800 Merge branch 'cassandra-4.0' into trunk commit 33fd2dc817f56116af0b4b7d2cd5365ce52948d9 Author: Caleb Rackliffe Date: Wed Nov 17 16:43:00 2021 -0600 Log queries that fail on timeout or unavailable errors up to once per minute by default patch by Caleb Rackliffe and Marcus Eriksson; reviewed by David Capwell and Yifan Cai for CASSANDRA-17159 Co-authored-by: Caleb Rackliffe Co-authored-by: Marcus Eriksson commit 5d781232ae9481bf6680eb644d838546c3f1df8e Merge: a6cfd64bee 74b80154bc Author: Bereng Date: Thu Nov 18 07:46:01 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit a6cfd64bee636ab6ba2017d535d6324f742bb25f Merge: 1c79c6823c 3f2066d150 Author: Mick Semb Wever Date: Wed Nov 17 10:56:24 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 1c79c6823c6eac002d912afb979f6f9fa7fe61e6 Author: David Capwell Date: Tue Nov 16 12:38:38 2021 -0800 Fix test distributed.test.trackwarnings.TombstoneWarningTest patch by David Capwell; reviewed by Caleb Rackliffe for CASSANDRA-17156 commit 670bcff596958839778562d2918b363c03ca5ec9 Merge: be9db0980b 44ace88585 Author: Ekaterina Dimitrova Date: Tue Nov 16 14:35:48 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit be9db0980b9525d67a2853ffb3d2bebaf87ad8f5 Author: Brandon Williams Date: Fri Nov 12 09:57:29 2021 -0600 Rename cassandra-topology.properties to example Patch by brandonwilliams; reviewed by bereng for CASSANDRA-16912 commit f61d817cb5fc4c63d610cab83c5824361c008013 Merge: ad4d2b3a26 b6f61e850c Author: Ekaterina Dimitrova Date: Tue Nov 16 10:33:57 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit ad4d2b3a266535734ef1bdebb4a3545fb3306769 Author: David Capwell Date: Mon Nov 15 08:51:43 2021 -0800 Fix test NetstatsBootstrapWithEntireSSTablesCompressionStreamingTest#testWithStreamingEntireSSTablesWithoutCompressionWithoutThrottling patch by David Capwell; reviewed by David Capwell, Stefan Miklosovic for CASSANDRA-17143 commit b84ec51b4c73d7b9b58ea8a1708ae2f330972970 Author: David Capwell Date: Fri Nov 12 15:47:44 2021 -0800 Refactor normal/preview/IR repair to standardize repair cleanup and error handling of failed RepairJobs patch by David Capwell; reviewed by Marcus Eriksson for CASSANDRA-17069 commit 092bb60ba413b8ef0eb9e0de86ce394a2f939084 Author: Caleb Rackliffe Date: Tue Nov 9 16:57:10 2021 -0600 Take into account starting session size when making assertions about its size after task execution completes patch by Caleb Rackliffe; reviewed by David Capwell for CASSANDRA-17039 commit 97bfde01f805b4c3abd52870e0338440093f5893 Author: Yifan Cai Date: Thu Nov 11 16:40:19 2021 -0800 Log missing peers in StartupClusterConnectivityChecker patch by Yifan Cai; reviewed by Berenguer Blasi, Stefan Podkowinski for CASSANDRA-17130 commit 2ab6e14aea69aad61eaa44bfc7390083f1cfa3af Merge: 28882ec33d 03e83f2070 Author: Aleksey Yeschenko Date: Fri Nov 12 14:41:57 2021 +0000 Merge branch 'cassandra-4.0' into trunk commit 28882ec33d57ee2a48f7073549fc767b6116b289 Merge: d8b2678257 dd6242037b Author: Aleksey Yeschenko Date: Fri Nov 12 11:39:20 2021 +0000 Merge branch 'cassandra-4.0' into trunk commit d8b2678257284e6a714a7ef840563feebaa61d21 Merge: 1049a78761 ddf5c581b8 Author: Alex Petrov Date: Fri Nov 12 07:22:26 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 1049a787618642da6c4287f9cdd5c4134eb2c526 Merge: 9f46a48952 5c5bf7db0d Author: Ekaterina Dimitrova Date: Thu Nov 11 20:40:49 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 9f46a489527c2440d563cef69038aac3e626148a Merge: 4f112af615 79bbb7dac5 Author: Yifan Cai Date: Thu Nov 11 16:23:51 2021 -0800 Merge branch 'cassandra-4.0' into trunk commit 4f112af6156b0ed35eb85124eaa525e5e0b9a059 Merge: e0954fa1a8 4bb6f411d8 Author: Ekaterina Dimitrova Date: Wed Nov 10 16:41:03 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit e0954fa1a80120a066c2a8b9476b175d8186cedb Author: Brandon Williams Date: Wed Nov 10 14:27:43 2021 -0600 Revert "Rename cassandra-topology.properties to cassandra-topology.properties.example" This reverts commit 29aa088ab69f75220162f1282c5f73c9e1993752. commit 77dde2a3c4b40da3d820d4852c572338acbf6dc9 Author: Francisco Guerrero Date: Tue Nov 9 13:32:15 2021 -0800 Introduce separate rate limiting settings for entire SSTable streaming patch by Francisco Guerrero; reviewed by Dinesh Joshi, Marcus Eriksson, Yifan Cai for CASSANDRA-17065 commit 29aa088ab69f75220162f1282c5f73c9e1993752 Author: Bereng Date: Wed Nov 10 11:22:54 2021 +0100 Rename cassandra-topology.properties to cassandra-topology.properties.example Patch by bereng; reviewed by brandonwilliams for CASSANDRA-16912 commit 42b284d5c957675f7b551e36a75a0cc043753530 Author: Andrés de la Peña Date: Fri Nov 5 17:11:09 2021 +0000 Add MV utilities to CQLTester patch by Andrés de la Peña and Zhao Yang; reviewed by Berenguer Blasi for CASSANDRA-17122 Co-authored-by: Andrés de la Peña Co-authored-by: Zhao Yang commit cedde3d991da48039caee2afb421c737777267f0 Merge: 63292292b8 07b908c78c Author: Benedict Elliott Smith Date: Wed Nov 10 11:13:52 2021 +0000 Merge branch 'cassandra-4.0' into trunk commit 63292292b8dbe3bb4f691f82823dcdc0172d2291 Author: Aleksei Zotov Date: Fri Sep 17 17:52:07 2021 +0400 Implement Virtual Tables for Auth Caches Patch by Aleksei Zotov; reviewed by Sam Tunnicliffe, and Benjamin Lerer for CASSANDRA-16914 commit eae7b9c3ade386f28c5f0c7ee015b0d0445388ac Author: Blake Eggleston Date: Wed Sep 15 10:52:46 2021 -0400 Actively update auth caches in the background Patch by Blake Eggleston; reviewed by Sam Tunnicliffe, Jason Brown, and Caleb Rackliffe for CASSANDRA-16957 Co-authored-by: Blake Eggleston Co-authored-by: Josh McKenzie commit 4fb170c5c2c73737d74197dc41d83371c303106b Author: David Capwell Date: Tue Nov 9 07:56:06 2021 -0800 MessagingServiceTest listenOptionalSecureConnection and listenRequiredSecureConnection fail sporadically patch by David Capwell; reviewed by Benedict Elliott Smith, Caleb Rackliffe for CASSANDRA-17033 commit 599294c919df10a4aea592d78364af565d8ff1ed Author: Brandon Williams Date: Mon Nov 8 11:52:54 2021 -0600 Replace System.currentTimeMillis with Clock.Global.currentTimeMillis Patch by brandonwilliams; reviewed by bereng for CASSANDRA-17123 commit 8ddcd43b0cfcebfda882a238532d00905fe85eb8 Author: Kanthi Subramanian Date: Sun Oct 17 17:18:58 2021 -0400 Add unix time conversion functions Patch by Kanthi Subramanian; reviewed by Benjamin Lerer and Brandon Williams for CASSANDRA-17029 commit b29e1037e4da75dfd2a30ad10f8008b24941e37f Author: David Capwell Date: Mon Nov 8 16:03:06 2021 -0800 JVMStabilityInspector.forceHeapSpaceOomMaybe should handle all non-heap OOMs rather than only supporting direct only patch by David Capwell; reviewed by Caleb Rackliffe, Yifan Cai for CASSANDRA-17128 commit d37e256f2ed0105f65ce1753765d3a712be45032 Author: Brandon Williams Date: Mon Nov 8 09:56:12 2021 -0600 Use correct clock source in TimestampSerializerTest Patch by brandonwilliams; reviewed by edimitrova for CASSANDRA-17120 commit a0f950f02d9713a390b1adc05ae5b1f389378a46 Merge: 2fc7206128 c96131035b Author: Sam Tunnicliffe Date: Fri Nov 5 07:58:56 2021 +0000 Merge branch 'cassandra-4.0' into trunk commit 2fc7206128384b2b412042ef17ee86cd32fd595c Author: Benedict Elliott Smith Date: Thu Nov 4 14:33:04 2021 -0700 Forbid other Future implementations with checkstyle patch by Benedict Elliott Smith; reviewed by David Capwell for CASSANDRA-17055 commit 185c5232a693a8e661926dfad3c0536327a4cb79 Merge: 5b4d369266 8ba8f0b841 Author: Bereng Date: Thu Nov 4 08:12:40 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 5b4d3692664172546b25c765f89c94e61400d873 Author: dcapwell Date: Wed Nov 3 15:32:40 2021 -0700 coordinator_read_size had wrong name in cassandra.yaml patch by Ekaterina Dimitrova, reviewed by David Capwell for CASSANDRA-17118 commit 111e94ae13381ede97de190d9e1af9a77cac2b21 Author: David Capwell Date: Wed Nov 3 11:53:36 2021 -0700 commit log was switched from non-daemon to daemon threads, which causes the JVM to exit in some case as no non-daemon threads are active patch by David Capwell, Sam Tunnicliffe; reviewed by Sam Tunnicliffe for CASSANDRA-17085 commit c4b443ab90afd258d4cd616daeff2eba7e847b80 Author: Josh McKenzie Date: Tue Nov 2 10:51:09 2021 -0400 Ninja-fix: CHANGES.txt and NEWS.txt for denylisting (CASSANDRA-12106) commit ab920c30310a8c095ba76b363142b8e74cbf0a0a Author: Josh McKenzie Date: Fri Sep 17 16:34:04 2021 -0400 Add a Denylist to block reads and writes on specific partition keys Patch by Josh McKenzie, reviewed by Aleksei Zotov and Sumanth Pasupuleti for CASSANDRA-12106 Co-authored by Josh McKenzie Co-authored by Sam Overton commit d21e0dd8461e7ab9ce41ad4ee58e75134dc918ab Merge: 37830770d1 6c9d5abbc5 Author: Andrés de la Peña Date: Wed Oct 27 18:19:11 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 37830770d1e54703c4b30a67c259b50317e3d4e3 Merge: 7d0cb2015d 530bc914cd Author: Andrés de la Peña Date: Tue Oct 26 18:24:00 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 7d0cb2015d76a82a9baf1c4769345ebb5194f212 Merge: 8e225c55c4 5aa2fb83e7 Author: Bereng Date: Tue Oct 26 07:44:20 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 8e225c55c49493f00fc9bc0b5809ab026d60c767 Author: David Capwell Date: Mon Oct 25 07:28:08 2021 -0700 v4+ protocol did not clean up client warnings, which caused leaking the state patch by David Capwell; reviewed by Caleb Rackliffe, Jon Meredith, Sam Tunnicliffe for CASSANDRA-17054 commit 60e0da1bd0f8271f7bbba300695d67b9ad0a497f Author: Kanthi Subramanian Date: Fri Oct 22 11:55:23 2021 -0400 Remove duplicate toCQLString in ReadCommand Patch by Kanthi Subramanian; reviewed by brandonwilliams and maedhroz for CASSANDRA-17023 commit b2ccd0f3f588a34cd68222bdacd1914478914ac9 Author: kurt Date: Tue Feb 20 03:49:33 2018 +0000 ensure hint window is persistent across restarts of a node patch by Kurt Greaves; reviewed by Brandon Williams, Mick Semb Wever and Stefan Miklosovic for CASSANDRA-14309 commit 185893256f10c14207bffe49ae733fb1a970aec5 Author: Francisco Guerrero Date: Fri Oct 8 15:05:24 2021 -0700 Allow GRANT/REVOKE multiple permissions in a single statement patch by Francisco Guerrero; reviewed by Benjamin Lerer and Yifan Cai for CASSANDRA-17030 This commit allows GRANT/REVOKE statement to support multiple permissions with a single statement. For example, ``` GRANT MODIFY, SELECT ON KEYSPACE field TO manager; GRANT ALTER, DROP ON ROLE role1 TO role2; ``` commit ec04802e52aa2462756d438581761ddcb133dd25 Merge: fafd88e717 e8d905e636 Author: Brandon Williams Date: Thu Oct 21 07:39:25 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit fafd88e717eec071559c3b141cde8eb9801a4234 Merge: b1c3755f62 4e77cc8928 Author: Brandon Williams Date: Wed Oct 20 11:01:22 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit b1c3755f62a39b091c6c45cba3504eca165d678b Merge: df6ecaae1c 71f7b6cc92 Author: Brandon Williams Date: Wed Oct 20 10:36:52 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit df6ecaae1c3ac3a05c8cd0c6a9c3da58d4e86a98 Author: Benjamin Lerer Date: Fri Oct 8 17:32:15 2021 +0200 Allow to grant permission for all tables in a keyspace Patch by Benjamin Lerer; Review by Andres de la Peña and Ekaterina Dimitrova for CASSANDRA-17027 In some cases it is useful to prevent users to alter or drop a keyspace while allowing them to create new tables. This patch add support for a new DataResource below KEYSPACE but above TABLE. The syntax to grant permission at this level in ALL TABLES IN KEYSPACE. commit 3a950b45c321e051a9744721408760c568c05617 Merge: 5a88bc8cd1 72f3b79018 Author: Andrés de la Peña Date: Tue Oct 19 11:44:40 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 5a88bc8cd14c849850718d646e8c7c404e2f1e89 Merge: d9ca614043 f646a7c312 Author: Brandon Williams Date: Fri Oct 15 10:07:40 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit d9ca61404334f3bd94c08cf66ccd15e8c5287f52 Merge: ebce74ee2d 25f67a75e2 Author: Alex Petrov Date: Fri Oct 15 11:27:39 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit ebce74ee2d186359bbd83c33b96f5b6c3ae1f856 Merge: d389620fee be1684b0ee Author: Benjamin Lerer Date: Thu Oct 14 15:29:37 2021 +0200 Merge branch cassandra-4.0 into trunk commit d389620feebb6798f3269834f567d4b73bcee9f2 Author: Marcus Eriksson Date: Wed Oct 13 09:44:30 2021 -0400 Log time spent writing keys during compaction Patch by Marcus Eriksson; reviewed by Sam Tunnicliffe for CASSANDRA-17037 Co-authored-by: Marcus Eriksson Co-authored-by: Josh McKenzie commit 31b2d133df486e90862ee5e365cbb4a00462e2ea Merge: 460ae341f5 2da274787f Author: Andrés de la Peña Date: Thu Oct 14 12:39:06 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 460ae341f5628bbf65498e693fb61ac77d4d6c5b Author: Aleksei Zotov Date: Tue Sep 28 00:56:56 2021 +0400 Make nodetool compactionstats and sstable_tasks consistent patch by Aleksei Zotov; reviewed by Stefan Miklocovic and Brandon Williams for CASSANDRA-16976 commit 2ce9b13e8ff1d5d54d87e403a25561a82c9490a0 Author: Marcus Eriksson Date: Tue Oct 12 12:28:35 2021 -0400 Add logging and metrics for index summary redistribution Patch by Marcus Eriksson, reviewed by David Capwell for CASSANDRA-17036 Co-authored-by: Marcus Eriksson Co-authored-by: Josh McKenzie commit 5fdadb25f95099b8945d9d9ee11d3e380d3867f4 Author: Sumanth Pasupuleti Date: Mon May 11 16:18:51 2020 +0200 add default_keyspace_rf and minimum_keyspace_rf configuration options Patch by Sumanth Pasupuleti; reviewed by Jaydeepkumar Chovatia, Alex Petrov, Stefan Miklosovic and Alexander Zotov for CASSANDRA-14557. commit fc27042f61a6d78ec998a0186a5e97def90fd50a Author: Aleksandr Sorokoumov Date: Fri Aug 6 15:46:50 2021 +0200 Expose information about stored hints via a nodetool command and a virtual table Patch by Aleksandr Sorokoumov; reviewed by Ekaterina Dimitrova, Stefan Miklosovic and Aleksei Zotov for CASSANDRA-14795 commit 8f2bf50abece27432993f312df7ca7e34aff4a76 Merge: 243d220854 96a5e0feed Author: Caleb Rackliffe Date: Tue Oct 12 16:47:35 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 243d220854895b968e2f169d3650c27d83567bb3 Author: Chang Liu Date: Wed Mar 28 16:31:52 2018 +1100 store FBUtilities.getJustBroadcastNativeAddress() in system.local for rpc_address column patch by Chang Liu; reviewed by Alexei Zotov and Stefan Miklosovic for CASSANDRA-11181 commit 0d4cc2ef0d246df34d534ef2f0df8ad9bb043206 Author: Andrés de la Peña Date: Tue Oct 12 11:55:31 2021 +0100 Add support for type casting in WHERE clause components and in the values of INSERT/UPDATE statements patch by Andrés de la Peña; reviewed by Benjamin Lerer for CASSANDRA-14337 commit c4254663f539069bac96b9314aed58f997724936 Merge: 917d74bd35 e57a8ddbfa Author: Brandon Williams Date: Mon Oct 11 12:07:22 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 917d74bd35712f8374ecefd1d890d02916162daa Author: Bowen Song Date: Sat Sep 25 17:16:53 2021 +0100 add credentials file support for CQLSH patch by Bowen Song; reviewed by Brian Houser, Stefan Miklosovic and Brandon Williams for CASSANDRA-16983 commit 4f09733d28398207bc16ace92cda6e1ffeb99644 Author: Caleb Rackliffe Date: Thu Oct 7 13:34:26 2021 -0500 Skip remaining bytes in the Envelope buffer when a ProtocolException is thrown to avoid double decoding patch by Caleb Rackliffe; reviewed by Sam Tunnicliffe and Jon Meredith for CASSANDRA-17026 commit 5ab15316e1cd83c129f3a66ba89e0fdcb60e4dfd Author: Josh McKenzie Date: Wed Sep 15 16:29:01 2021 -0400 Allow reverse iteration order during resource permissions checking Patch by Sam Tunnicliffe; reviewed by Blake Eggleston, Jeff Jirsa, and Aleksei Zotov for CASSANDRA-17016 Co-authored by Sam Tunnicliffe Co-authored by Josh McKenzie commit 0c444a75e79da8c157813a72d61f2b2f86e187ba Author: Benedict Elliott Smith Date: Wed Jul 28 20:04:03 2021 +0100 [CASSANDRA-16931] CEP-10 Phase 2: Improve DTest @Shared Annotation Functionality commit fe9cff663b48fecdb964caaded2004e83a0c89f4 Author: Benedict Elliott Smith Date: Wed Jul 28 20:03:09 2021 +0100 [CASSANDRA-16930] CEP-10 Phase 2: Improved Configuration For Controlling Determinism Co-authored-by: Benedict Elliott Smith Co-authored-by: Sam Tunnicliffe Co-authored-by: Caleb Rackliffe commit ce2a0a28bc9ca21e1fae29f2a38448a877db06c3 Author: Benedict Elliott Smith Date: Mon Apr 26 12:09:20 2021 +0100 [CASSANDRA-16932] CEP-10 Phase 2: Minor Gossip Fixes * Ensure we apply new states in correct order so as not to lose TOKEN message * Permit replacement nodes to join the ring with lower heartbeat state than node being replaced commit 5b82447098ad634900f8892297ef82083eadb954 Author: Benedict Elliott Smith Date: Thu Jul 29 18:06:32 2021 +0100 [CASSANDRA-17013] CEP-10 Phase 1: in-jvm-dtest-api changes and version bump commit 6812fddb3e6d20e5e8840e57076b97210a9abfe2 Author: Benedict Elliott Smith Date: Wed Jul 28 20:01:18 2021 +0100 [CASSANDRA-16927] CEP-10 Phase 1: Refactor Streaming patch by Benedict Elliott Smith and Sam Tunnicliffe; reviewed by Aleksey Yeschenko and Aleksei Zotov for CASSANDRA-16927 Co-authored-by: Benedict Elliott Smith Co-authored-by: Aleksey Yeschenko Co-authored-by: Sam Tunnicliffe commit aae7e8b07c84476de893b473a13cdc6d9e260573 Author: Benedict Elliott Smith Date: Fri Apr 16 12:01:25 2021 +0100 [CASSANDRA-16928] CEP-10 Phase 1: InetAddressAndPort extends InetSocketAddress patch by Benedict; reviewed by Sam Tunnicliffe, Caleb Rackliffe and Aleksei Zotov for CASSANDRA-16928 commit 6a1d9de3d938d24baa2ba98b5379b0536177c786 Author: Benedict Elliott Smith Date: Thu Jul 29 17:20:18 2021 +0100 [CASSANDRA-16926] CEP-10 Phase 1: Mockable Filesystem patch by Benedict; reviewed by Aleksey Yeschenko, Sam Tunnicliffe and Aleksei Zotov for CASSANDRA-16926 Co-authored-by: Benedict Elliott Smith Co-authored-by: Aleksey Yeschenko commit be1f050bc8c0cd695a42952e3fc84625ad48d83a Author: Benedict Elliott Smith Date: Wed Jul 28 19:47:45 2021 +0100 [CASSANDRA-16925] CEP-10 Phase 1: Mockable Task Execution Co-authored-by: Benedict Elliott Smith Co-authored-by: Sam Tunnicliffe commit e5b92e108851d2be81c43cbf982564f92d28aa6b Author: Benedict Elliott Smith Date: Wed Jul 28 19:24:43 2021 +0100 [CASSANDRA-16924] CEP-10 Phase 1: Mockable Blocking Concurrency Primitives patch by Benedict; reviewed by Sam Tunnicliffe and Aleksei Zotov for CASSANDRA-16924 Co-authored-by: Benedict Elliott Smith Co-authored-by: Sam Tunnicliffe commit 2e2db4dc40c4935305b9a2d5d271580e96dabe42 Author: Sam Tunnicliffe Date: Mon Aug 16 12:08:23 2021 -0400 Verify correct ownership of attached locations on disk at startup patch by Sam Tunnicliffe; reviewed by Chris Earman, Xiaolong Jiang, and Caleb Rackliffe for CASSANDRA-16879 Co-authored by Sam Tunnicliffe Co-authored by Josh McKenzie commit f638f6a7810da970d6f8e8f6d175751be30d868d Merge: b00b82a80f c4a07ae2be Author: Stefan Miklosovic Date: Tue Oct 5 16:31:25 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit b00b82a80f816aa06d876b6c861738847dd6f17b Merge: 24dcc280c2 1aae9468ed Author: Caleb Rackliffe Date: Mon Oct 4 15:14:46 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 24dcc280c2e442eea27e7129c4c948eb6199ed91 Author: Maulin Vasavada Date: Fri May 21 00:43:50 2021 -0700 CEP-9 make SSLContext creation pluggable patch by Maulin Vasavada; reviewed by Jon Meredith, Stefan Miklosovic and Berenguer Blasi for CASSANDRA-16666 commit c3c2c7efecb23d7627aa8ffda72603dc88f1ed37 Merge: 0945002457 3660a58ab4 Author: Alex Petrov Date: Sun Oct 3 23:24:22 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 0945002457738f4c256bc6a262a8d8aaadcdf392 Merge: bcdb575fd3 b22749bbb2 Author: Brandon Williams Date: Fri Oct 1 10:02:40 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit bcdb575fd32bccf196c8b398c7bb9f95f2591b73 Merge: 0ccca8dab2 36266ce257 Author: Andrés de la Peña Date: Fri Oct 1 15:20:32 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 0ccca8dab21719cc9248a87542a504cc057700e1 Merge: 97108ef0a7 339e8b74bf Author: Andrés de la Peña Date: Fri Oct 1 11:59:09 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 97108ef0a70c432ae241286e4b91ed3658700481 Author: Brandon Williams Date: Thu Sep 30 18:37:54 2021 -0500 Revert "Ninja lock setuptools version for thrift" This reverts commit 1907bccc2b9abf3845626d0a157a3f42d9ced0a1. commit 1907bccc2b9abf3845626d0a157a3f42d9ced0a1 Author: Brandon Williams Date: Thu Sep 30 15:35:51 2021 -0500 Ninja lock setuptools version for thrift commit 3cc59017a4aa09ea572f7f2bc7c2c9f9bd9bd09e Merge: b6e400d0c3 59f5e57faf Author: Stefan Miklosovic Date: Thu Sep 30 21:36:06 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit b6e400d0c3c3f74d17261caaa6388e6c70df322f Merge: 9dd0e55017 c0916d67be Author: Andrés de la Peña Date: Thu Sep 30 13:14:45 2021 +0100 Merge branch 'cassandra-4.0' into trunk # Conflicts: # src/java/org/apache/cassandra/db/compaction/CompactionTask.java commit 9dd0e55017f385f0187b478610990ca2590b588a Merge: 6e35f30f66 84ec1dc97d Author: Marcus Eriksson Date: Thu Sep 30 11:24:33 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 6e35f30f665fe73fcb6e6607f1bf303b09638f47 Merge: 15a2fe00fc 42905e94b5 Author: Bereng Date: Thu Sep 30 07:47:40 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 15a2fe00fc9817874f4c1600d56a373c21b1ad1c Author: Benedict Elliott Smith Date: Mon Jan 18 13:36:58 2021 +0000 [CASSANDRA-16923] CEP-10 Phase 1: Mockable System Clock Co-authored-by: Benedict Elliott Smith Co-authored-by: Aleksey Yeschenko Co-authored-by: Sam Tunnicliffe commit 643b9b776f374929128aec66d086eee7e6154e03 Merge: e0777ff5ab 5d457bc683 Author: Aleksei Zotov Date: Wed Sep 29 13:21:10 2021 +0400 Merge branch 'cassandra-4.0' into trunk commit e0777ff5ab1de2a67f9fb3bf881c28f790312e96 Merge: 0218423127 e365908ee4 Author: Stefan Miklosovic Date: Wed Sep 29 08:03:05 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 021842312794a7fc24d49144f9aa7f22bb74a1f9 Author: Brandon Williams Date: Tue Sep 28 12:08:45 2021 -0500 Initialize tmd before each test Patch by brandonwilliams; reviewed by edimitrova for CASSANDRA-17007 commit c7526f943f50e994e94c8287c772c856961833f2 Author: David Capwell Date: Mon Aug 30 10:18:03 2021 -0700 Add soft/hard limits to local reads to protect against reading too much data in a single query patch by David Capwell; reviewed by Caleb Rackliffe and Marcus Eriksson for CASSANDRA-16896 commit ce11fee8b14fad9a84df292b882957d300b546d9 Merge: 41c1eeb4f5 d32253d1ae Author: Andrés de la Peña Date: Tue Sep 28 17:50:27 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 41c1eeb4f52316e8ee825c736eb372c3c13952d7 Merge: 5ee1ba2dfa 984f763939 Author: Benjamin Lerer Date: Tue Sep 28 16:59:10 2021 +0200 Merge branch cassandra-4.0 into trunk commit 5ee1ba2dfaac0f4424322268912f6714e89904ae Merge: 44a004c87d 5f97977dfa Author: Ekaterina Dimitrova Date: Mon Sep 27 15:02:11 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 44a004c87d0ebfbba8cb80f80758c973ebf59e81 Merge: b1e77baa3d 80dccde3cd Author: Brandon Williams Date: Mon Sep 27 09:12:13 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit b1e77baa3d7cef7b417fa787e3005a999a440418 Author: Jay Zhuang Date: Fri Sep 24 15:20:35 2021 -0500 Avoid token cache invalidation for removing a non-member node Patch by Jay Zhuang; reviewed by brandonwilliams and azotcsit for CASSANDRA-15290 commit 01ca6057c0f29c953bcc0cde590e27fadc1070fd Merge: 9f15ec6de1 89f35a49a0 Author: Ekaterina Dimitrova Date: Fri Sep 24 12:54:10 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 9f15ec6de11c57d5fff02fe08639b647fc0749e8 Author: Josh McKenzie Date: Thu Sep 2 13:58:16 2021 -0400 Evaluate consistency levels of auth reads Patch by Jason Brown; reviewed by Matthew Byrd, Sankalp Kohli, and Benjamin Lerer for CASSANDRA-12988 Co-authored by Jason Brown (jasedbrown@gmail.com) Co-authored by Josh McKenzie (jmckenzie@apache.org) commit 81e5abfc3d4976abc26de420206de7595552ab55 Merge: 92baa924d4 77144aa472 Author: Andrés de la Peña Date: Fri Sep 24 11:47:27 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 92baa924d4d2d09a6bae4ddb70add600562f1420 Merge: c648412cf2 e8c675250a Author: Bereng Date: Fri Sep 24 08:36:20 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit c648412cf224144476297901e01a21f1d27c7e3a Merge: 9ed8810183 50e0b40184 Author: Marcus Eriksson Date: Thu Sep 23 11:34:48 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 9ed8810183abdf15cbb72c23ab078b61c14ee2cd Author: Brandon Williams Date: Mon Aug 30 11:18:26 2021 -0500 Add sstable count to compactionstats output Patch by brandonwilliams; reviwed by Aleksei Zotov and adelapena for CASSANDRA-16844 commit ea99f5889bbc92b9b5957c9210b3463e64d72b63 Merge: cb1c8f9d34 d519677072 Author: Brandon Williams Date: Wed Sep 22 11:27:29 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit cb1c8f9d34edfa639096d2d122dfd0ee6d23b479 Merge: 1df2a54eee 053806c320 Author: Brandon Williams Date: Tue Sep 21 09:24:57 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 1df2a54eee24d0d37b699d80b1bfaa8a25178aa9 Merge: e9645cc7a4 86aaf22f15 Author: Marcus Eriksson Date: Tue Sep 21 12:28:17 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit e9645cc7a4e189b58eb0ba20269dfaec9b46da29 Merge: bc348c9985 4018084bf7 Author: Ekaterina Dimitrova Date: Mon Sep 20 17:10:17 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit bc348c99852b210d7ca6d6a09a4d24b274ea1210 Merge: b0855914ef b9d8700355 Author: Marcus Eriksson Date: Mon Sep 20 14:35:46 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit b0855914ef673b183efec0dd132847ddb9d96f5f Merge: e60341823d 9e9dffb704 Author: Marcus Eriksson Date: Mon Sep 20 13:38:05 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit e60341823dee2f4ed394b5649c40fc5a1b5854c7 Merge: 7c3935ced3 e98be8e3ec Author: Bereng Date: Mon Sep 20 09:46:52 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 7c3935ced35f6541dc1d8950b3e001fad8d7c17f Merge: 7faff38826 14af149ed5 Author: Ekaterina Dimitrova Date: Fri Sep 17 17:27:44 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 7faff3882633f93cbdec12e25fdb5883a912a4a5 Author: Aleksei Zotov Date: Fri Sep 17 08:51:30 2021 -0500 Upgrade Caffeiene to 2.9.2 Patch by Aleksei Zotov; reviewed by brandonwilliams and mck for CASSANDRA-15153 commit 06ee0b3f7b51924dd05f364fef576bb347e195a0 Merge: f7c71f65c0 dd3d83a819 Author: Brandon Williams Date: Fri Sep 17 08:49:01 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit f7c71f65c000c2c3ef7df1b034b8fdd822a396d8 Author: Aleksei Zotov Date: Fri Jul 23 21:45:12 2021 +0400 Allow DELETE and TRUNCATE to work on Virtual Tables if the implementation allows it patch by Aleksei Zoto; reviewed by Benjamin Lerer and Chris Lohfink for CASSANDRA-16806 commit e86ae7fbe52c95747d69f7d78b6da6fbd34bd48d Author: Stefan Miklosovic Date: Wed Sep 8 15:40:55 2021 +0200 make creation timestamp consistent for all tables of a snapshot patch by Stefan Miklosovic; reviewed by Paulo Motta and Aleksei Zotov for CASSANDRA-16920 commit 99246fb24ff611ff1ea929867f8d9bf3dc7747ed Merge: 8a91751854 57c1c6183a Author: Stefan Miklosovic Date: Thu Sep 16 14:09:46 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 8a91751854793a47752084b027654ed9c089bd5c Merge: 09c89e5f5f a1790a6a28 Author: Brandon Williams Date: Wed Sep 15 13:41:36 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 09c89e5f5f8604301c233130dfb6e82a36ae30f3 Author: Aleksey Yeschenko Date: Wed Sep 1 12:22:31 2021 -0400 Further restrict schema column drop/recreate conversions patch by Aleksey Yeschenko; reviewed by Blake Eggleston, Sam Tunnicliffe, and Caleb Rackliffe for CASSANDRA-16905 Co-authored by Aleksey Yeschenko (aleksey@apache.org) Co-authored by Josh McKenzie (jmckenzie@apache.org) commit 18b2100bc4239eecfc1cb1ec3c12c87fc58801f5 Merge: 0516aa1be2 05e7cc1553 Author: Mick Semb Wever Date: Sat Sep 11 00:31:50 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 0516aa1be2b30dfe25c213c5ccfed44c9220cccc Merge: b015cbdff6 636ab42bd5 Author: Andrés de la Peña Date: Fri Sep 10 13:28:52 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit b015cbdff6882e20fbc76dcbaf30be982e6a0195 Merge: 08444cbc3f 8b0b22e166 Author: Bereng Date: Fri Sep 10 09:34:21 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 08444cbc3f6378f281a811d74c9cb152c8ad19ca Author: Brandon Williams Date: Thu Aug 26 10:16:42 2021 -0500 Reduce max native frame size to 16MB Patch by brandonwilliams; reviewed by blerer for CASSANDRA-16886 commit b6b3be6d8aa79c5e74b0c0f197075fdc48e38287 Merge: b8f786d387 bc052fa68f Author: Caleb Rackliffe Date: Thu Sep 9 14:09:34 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit b8f786d387ec0b023bd6fe61bc012d8f8dc95301 Merge: dcee430e57 ffc6593d06 Author: Brandon Williams Date: Wed Sep 8 11:50:33 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit dcee430e57f3fd37e4bc6652ad7180361e31444e Author: nvharikrishna Date: Wed Aug 18 15:06:03 2021 +0200 Add support for filtering using IN restrictions patch by Venkata Harikrishna Nukala; reviewed by Andrés de la Peña and Benjamin Lerer for CASSANDRA-14344 commit 4a8c5cc1e06f02f19400c5609ba4b5cbfc02f061 Merge: e04a3a94b9 d90cd518ae Author: Sam Tunnicliffe Date: Tue Sep 7 17:08:12 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit e04a3a94b973969921d720669ac8671ad286e25c Merge: ce2d756c41 9a34ecd4ad Author: Sam Tunnicliffe Date: Tue Sep 7 16:51:55 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit ce2d756c41afdfa74c44db1490f01e7d3ce9730d Merge: 138569b079 4ed28cbf9c Author: Ekaterina Dimitrova Date: Tue Sep 7 10:46:46 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 138569b079b3d17b1020a24463adabecd903b79f Author: Benjamin Lerer Date: Mon Sep 6 13:36:14 2021 +0200 Open java driver connections in CQLTester in a lazy way patch by Benjamin Lerer; reviewed by Andrés de la Peña for CASSANDRA-16918 commit cfc402d26a628bbc9e005c370f5707f4985207aa Author: Alex Petrov Date: Tue Aug 3 14:03:37 2021 +0200 Add a flag to upgradesstables to allow only upgrading sstables older than a certain timestamp and recompress command Patch by Alex Petrov, reviewed by Marcus Eriksson for CASSANDRA-16837. commit 3cb656a471088c138296bd1cbc7098c2228d7097 Merge: 7294210f6b 9c90cf7da0 Author: Stefan Miklosovic Date: Mon Sep 6 10:52:43 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 7294210f6bd5a262a824a261164cd155099a39a7 Merge: 163a4d7137 752160c514 Author: Brandon Williams Date: Fri Sep 3 13:50:17 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 163a4d7137fb67504f2b12cacdc2eb12e8d9a923 Merge: f9aa19e3b1 49e83027e2 Author: Ekaterina Dimitrova Date: Fri Sep 3 10:55:10 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit f9aa19e3b116c0078019e9382d1a6c4bb050f113 Author: Aleksei Zotov Date: Thu Aug 12 14:52:31 2021 +0100 Add nodetool commands to invalidate auth caches Patch by Aleksei Zotov; reviewed by Benjamin Lerer, Sumanth Pasupuleti and Sam Tunnicliffe for CASSANDRA-16404 commit 2d3c21cb61097db7c33777fafc83ae93ccacf142 Merge: 7d59da4ab3 6a4a93a808 Author: Bereng Date: Thu Sep 2 11:09:36 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 7d59da4ab3e446606f0fe0cc73554ad12f674cd5 Author: Jeff Jirsa Date: Mon Aug 23 13:03:59 2021 -0400 Catch read repair timeout exceptions and add metric Patch by Jeff Jirsa; reviewed by Blake Eggleston, Ariel Weisberg, Caleb Rackliffe for CASSANDRA-16880 Co-authored by Jeff Jirsa Co-authored by Josh McKenzie commit 3d4397a6d114db9f0203149e5feaea23b016db06 Merge: 05e0892efe b5e138646b Author: Andrés de la Peña Date: Wed Sep 1 15:34:13 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 05e0892efec5de8d9e52239af829b12d40ef8149 Merge: a74b44ac8a c36c081e5c Author: Marcus Eriksson Date: Wed Sep 1 10:51:46 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit a74b44ac8abab045ee0615db7acd6bc229bbe6b7 Merge: 4e0d8e0d5e 46a740cf61 Author: Caleb Rackliffe Date: Tue Aug 31 16:38:48 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 4e0d8e0d5e0ad7d123ba5af55ad2cd8f857778ca Merge: 057f8e5007 ca4f6b8056 Author: David Capwell Date: Tue Aug 31 08:38:10 2021 -0700 Merge branch 'cassandra-4.0' into trunk commit 057f8e5007d5c00d13e8af500b04b93b1eb7f855 Merge: a5698034c5 af17f136e5 Author: Benjamin Lerer Date: Tue Aug 31 16:59:06 2021 +0200 Merge branch cassandra-4.0 into trunk commit a5698034c5629e46d36bbd1596b2314ef5934b51 Merge: 4bb581f863 78c6279658 Author: Bereng Date: Tue Aug 31 07:25:26 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 4bb581f863da0643ce68cc8ba5a207512ec82571 Merge: b92620c387 9ed0150f82 Author: Mick Semb Wever Date: Mon Aug 30 21:47:13 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit b92620c387d3e8d38f5cf7dfb0d5b89634099ead Merge: ab8705e710 a6ea52118c Author: Mick Semb Wever Date: Mon Aug 30 21:37:15 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit ab8705e7104f76dcea45b2016a6b1c4eb0d26924 Merge: e1016cbb9c 6709111ed0 Author: Sam Tunnicliffe Date: Mon Aug 30 13:48:44 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit e1016cbb9ce8098336365728326e3701ac13fd42 Merge: dfb465c2e0 f59411f1c9 Author: Josh McKenzie Date: Fri Aug 27 15:49:23 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit dfb465c2e08ff158bf9991ec0b9dbfe546c1c8a8 Merge: 33979e3c91 2b6799a394 Author: Josh McKenzie Date: Fri Aug 27 15:11:37 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 33979e3c916b2006e75a042fa4c806364dbbe5c1 Merge: 1b5545beef 585bc69291 Author: Caleb Rackliffe Date: Fri Aug 27 11:54:28 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 1b5545beefb9f0e96e7c6587869f49a39fbc2a89 Merge: 4ec4ab992f 42025fd73a Author: Yifan Cai Date: Thu Aug 26 15:09:04 2021 -0700 Merge branch 'cassandra-4.0' into trunk commit 4ec4ab992f8adc0a60055a60525e9d11a28bc2ae Author: David Capwell Date: Thu Aug 26 13:06:00 2021 -0700 Add client warnings and abort to tombstone and coordinator reads which go past a low/high watermark patch by David Capwell; reviewed by Blake Eggleston, Marcus Eriksson for CASSANDRA-16850 commit ad249424814836bd00f47931258ad58bfefb24fd Author: Abi Palagashvili <0583463@gmail.com> Date: Tue Jun 8 00:23:39 2021 +0300 Add TTL support to nodetool snapshots Patch by Abi Palagashvili; Reviewed by Paulo Motta and Stefan Miklosovic for CASSANDRA-16789 Co-authored-by: Paulo Motta Co-authored-by: Stefan Miklosovic Closes #1046 commit f9b7c1e6984f5b81aae1e3a2191d4e9599db15ae Author: Marcus Eriksson Date: Mon Jan 11 10:55:44 2021 +0100 Allow CommitLogSegmentReader to optionally skip sync marker CRC checks patch by Caleb Rackliffe; reviewed by Josh McKenzie for CASSANDRA-16842 Co-authored-by: Jordan West Co-authored-by: Caleb Rackliffe Co-authored-by: Marcus Eriksson commit 39efc8307acb62f8f5e9459269d193dc6f319037 Merge: 5dd472e943 b824273091 Author: Sam Tunnicliffe Date: Wed Aug 25 17:42:50 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 5dd472e943f237fa86a06d077a27d704c09996db Merge: ee81d8dff4 31ce794d58 Author: Andrés de la Peña Date: Tue Aug 24 18:11:16 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit ee81d8dff43b087029fcb84d600ea0da07ad8385 Merge: ec2be8e574 2dc7501fff Author: Stefan Miklosovic Date: Mon Aug 23 22:52:03 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit ec2be8e57450702fd55a10f8f6809a816846d229 Merge: 4b3f07fc74 ddcf3e6aba Author: Josh McKenzie Date: Mon Aug 23 10:57:27 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 4b3f07fc74089151efeff7a8fdfa9c414a1f0d6a Author: David Capwell Date: Fri Aug 20 12:07:44 2021 -0700 allow blocking IPs from updating metrics about traffic patch by David Capwell; reviewed by Benjamin Lerer, Caleb Rackliffe, Jon Meredith for CASSANDRA-16859 commit d220d24994400d4342f5281f1a51514a6ae8c2fd Author: Caleb Rackliffe Date: Thu Aug 19 10:55:58 2021 -0500 Request-Based Native Transport Rate-Limiting patch by Caleb Rackliffe; reviewed by Benedict Elliott Smith and Josh McKenzie for CASSANDRA-16663 commit b286639eea11b5f6ee709711d00d36aa029bc114 Author: Stefan Miklosovic Date: Wed Jul 28 14:57:02 2021 +0200 implement getauditlog command patch by Stefan Miklosovic; reviewed by Ekaterina Dimitrova, Mick Semb Wever for CASSANDRA-16725 commit 058182025931e6b508d70fa9cfd294c7c860b185 Merge: 304deead2c f80b177f6f Author: Bereng Date: Thu Aug 19 07:38:41 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 304deead2c4a6aa4576783120d12858459d69098 Merge: fd3eb4fd9e 433274baea Author: Benjamin Lerer Date: Wed Aug 18 16:09:44 2021 +0200 Merge branch cassandra-4.0 into trunk commit fd3eb4fd9e930503c7149db5e90644857eb6e4d3 Author: Simon Zhou Date: Sun Aug 1 12:10:37 2021 -0700 Clean up repair code patch by Simon Zhou; reviewed by Ekaterina Dimitrova and Andrés de la Peña for CASSANDRA-13720 commit 5b325b8c514911070ce63f5dafa897f321780a2b Merge: dc77cb8729 cb19b39827 Author: Ekaterina Dimitrova Date: Tue Aug 17 15:15:34 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit dc77cb8729f0cd711479071bd32f0923105ab1a8 Merge: 8d6b1558be af6654cb06 Author: Andrés de la Peña Date: Tue Aug 17 15:19:37 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 8d6b1558bec3a940941fb392c0a555122c72026f Merge: 76f1a64658 ce21eb5fac Author: Andrés de la Peña Date: Tue Aug 17 15:03:14 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 76f1a64658f32dc933449a959f1c48d1d2919683 Merge: 42f6f7ada6 5f755e608a Author: Andrés de la Peña Date: Mon Aug 16 17:59:20 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 42f6f7ada600181fcb83aee6b3ccf470a823b07a Merge: 913d541ee9 f194d1a2ad Author: Stefan Miklosovic Date: Mon Aug 16 16:06:30 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 913d541ee9b9a636c02397f9fd3123e023d5c941 Merge: 39af4a5490 f242ccbedd Author: Bereng Date: Fri Aug 13 08:45:49 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 39af4a54903267656aa3bbefaf30adfc67974919 Merge: 8acbbe042b 979ab72f4f Author: David Capwell Date: Wed Aug 11 13:19:11 2021 -0700 Merge branch 'cassandra-4.0' into trunk commit 8acbbe042b236c6948845ecd7af093c6f0fa3e4b Author: Yifan Cai Date: Tue Aug 10 13:31:06 2021 -0700 Background schedule to clean up orphaned hints files patch by Yifan Cai; reviewed by Brandon Williams, Chris Lohfink for CASSANDRA-16815 commit 6a1b20e58d493925439cc9a67bc6b51bb0be631a Author: Caleb Rackliffe Date: Wed Apr 14 17:27:20 2021 -0500 Modify SecondaryIndexManager#indexPartition() to retrieve only columns for which indexes are actually being built patch by Caleb Rackliffe and Sam Tunnicliffe; reviewed by Benedict Elliott Smith and Aleksei Zotov for CASSANDRA-16776 commit cd36926577729a0ba7e9d710b1dc92a117e5feab Merge: b26a4daf86 112513c72d Author: Ekaterina Dimitrova Date: Tue Aug 10 10:52:52 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit b26a4daf8624f77279b8a98df6fdc7094b6b6da2 Merge: 32d63456a2 a9abccb28b Author: Andrés de la Peña Date: Tue Aug 10 15:23:26 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 32d63456a2b69f5ff378fa5de101fa59c97a44eb Merge: b72ee36a2e 6e169214ef Author: Mick Semb Wever Date: Tue Aug 10 10:07:27 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit b72ee36a2e777b9ab91aaf7def00dd4f1a7e014f Merge: b7c4271b16 148d3730f3 Author: Mick Semb Wever Date: Tue Aug 10 09:59:04 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit b7c4271b16801acff77c020ebf2daf82b1592184 Author: Ekaterina Dimitrova Date: Thu Aug 5 14:30:47 2021 -0400 Fix wrong regex pattern in RingTest patch by Ekaterina Dimitrova; reviewed by Andrés de la Peña for CASSANDRA-16612 commit 6fbe736bc261527af79a57f137c2af59efd81490 Merge: af1b770f1d 762df84f6c Author: Andrés de la Peña Date: Fri Aug 6 19:05:27 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit af1b770f1d019e6f4dcc66d7c76ae0623286a995 Merge: 7d14d3aa90 0685f2dc20 Author: Marcus Eriksson Date: Wed Aug 4 10:36:59 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 7d14d3aa904bb3348915a0894ce22fd7dfd102a7 Merge: 4e2464d44e 8c406b259b Author: Mick Semb Wever Date: Sun Aug 1 15:43:37 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 4e2464d44e72131934d1d223419c9fe5cc8bda5f Merge: e9ab8f3193 d319352fa8 Author: Andrés de la Peña Date: Fri Jul 30 18:12:01 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit e9ab8f3193282a762e338ca637475a389fa157b8 Merge: f66182415b f09ef63310 Author: Benjamin Lerer Date: Fri Jul 30 12:24:17 2021 +0200 Merge branch cassandra-4.0 into trunk commit f66182415b5a778f1420777fd87a345b8942bc3f Author: Jay Zhuang Date: Thu Jul 29 15:13:19 2021 +0200 Batch the token metadata update to improve the speed patch by Jay Zhuang; reviewed by Benjamin Lerer and Brandon Williams for CASSANDRA-15291 commit 592d5e336dca262a420fff385c94073725e7cc33 Merge: 669261ee56 bc51c57b0d Author: Andrés de la Peña Date: Thu Jul 29 12:06:26 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 669261ee5630e08b95fceb0a2ca1aa2e3554d38d Merge: dd369f5a04 7e2a965095 Author: Brandon Williams Date: Wed Jul 28 14:40:02 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit dd369f5a04cf001039fd237e15b599bccba2aa0a Merge: fd5dcc51cb 8b880f3663 Author: Brandon Williams Date: Wed Jul 28 13:35:43 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit fd5dcc51cb09041585fa16c24149d0a66e415b54 Author: xiang.wang Date: Mon Jul 19 17:32:12 2021 +0400 MerkleTrees variables renaming tree -> trees. Patch by Xiang Wang; reviewed by Aleksei Zotov, Benjamin Lerer and Ekaterina Dimitrova commit 236ab2f0875a73fd6e1b4592cb887147a826ccfc Merge: f49c29ee8c 35d8777984 Author: Brandon Williams Date: Fri Jul 23 11:25:12 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit f49c29ee8c7374a05e32be01cb0046236d5309e6 Author: Marcus Eriksson Date: Wed Mar 17 15:28:05 2021 +0100 Reduce the log level on "expected" repair exceptions patch by Caleb Rackliffe and Marcus Eriksson; reviewed by Josh McKenzie for CASSANDRA-16775 commit ad4e7030a095d05cd2a68e5ab4bf46ebffb0ac90 Merge: 818e0baa62 5fec98c583 Author: Ekaterina Dimitrova Date: Wed Jul 21 16:54:38 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 818e0baa6233b7e1182a411df2286c5f5fe3c00e Merge: 29cc615faf 0779160c25 Author: Brandon Williams Date: Wed Jul 21 14:14:30 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 29cc615fafe5f246dbc6668bcf85bfe4467152ee Merge: c1f235f71d b91dcce865 Author: Caleb Rackliffe Date: Wed Jul 21 12:49:12 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit c1f235f71d013318b86bddee3121beb4923ae0f9 Merge: 518b7becf1 cdf68e39ba Author: Caleb Rackliffe Date: Wed Jul 21 12:09:36 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 518b7becf196517aeac3b4d10623f41a4459ced2 Merge: fcea6a5509 e0cecaeec0 Author: Ekaterina Dimitrova Date: Tue Jul 20 15:13:47 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit fcea6a5509eb1b0f83aba3e6eda234383f1b9d11 Merge: e8ba1c3f35 64ec400ab6 Author: Sam Tunnicliffe Date: Tue Jul 20 19:32:39 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit e8ba1c3f354e2f8bbc7f694ad75ae98428de368b Merge: ddbed08087 1853006067 Author: Ekaterina Dimitrova Date: Tue Jul 20 12:29:10 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit ddbed080871b22a71fd18f7391c94fcf6f7d4916 Merge: a9abf5accb fd69375af0 Author: Yifan Cai Date: Mon Jul 19 14:29:48 2021 -0700 Merge branch 'cassandra-4.0' into trunk commit a9abf5accb2265c6394bf8134dc5496c5505375c Merge: 9cc7a0025d a2955d9b34 Author: Brandon Williams Date: Mon Jul 19 07:35:15 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 9cc7a0025d8b0859d8e9c947f6fdffd8455dd141 Author: Mick Semb Wever Date: Sat Jul 17 12:49:47 2021 +0200 Update release doc to include steps to perform on failed vote commit 845077fbc8d7102f303372eb8cb3299a155a436e Author: Yifan Cai Date: Thu Jul 15 12:43:43 2021 -0700 Make JMXTimer expose attributes using consistent time unit patch by Yifan Cai; reviewed by Caleb Rackliffe for CASSANDRA-16760 commit 42ddac3da4a8e97ad4b42ad305908019290cbf38 Merge: e16cacc6e3 5adc3100b2 Author: Benjamin Lerer Date: Fri Jul 16 18:42:30 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit e16cacc6e3ece9ad407c48db51a4204f8256ace1 Merge: 325677b4ef 82e0453901 Author: Brandon Williams Date: Fri Jul 16 07:58:11 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 325677b4eff52ee74fc8bd114303ec587c20e8e3 Merge: d78d2ddcb0 9da947b7bd Author: Caleb Rackliffe Date: Thu Jul 15 15:14:00 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit d78d2ddcb04a982312c2c28960e9b3a23db52b95 Merge: f08fdcf84f fdc2a4aa6c Author: Brandon Williams Date: Wed Jul 14 10:02:04 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit f08fdcf84f0910d4e15acb51ae787a3aaad8c84e Merge: 43a6e5cabd b7454f7ef2 Author: Brandon Williams Date: Wed Jul 14 09:54:05 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 43a6e5cabdb9f3b6be965100cbbd765b378eede0 Merge: e1a2f575f4 75f6167ca3 Author: Mick Semb Wever Date: Wed Jul 14 08:51:21 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit e1a2f575f40adcb03ea456e96dce280cf9dd07bc Merge: 096777d6c6 e0aa3651e2 Author: Brandon Williams Date: Tue Jul 13 10:27:56 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 096777d6c637995a227baa46b1df5e3e57e76039 Merge: f9a220c110 906781ecbb Author: Andrés de la Peña Date: Tue Jul 13 13:25:12 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit f9a220c1100d9f6deaca0ec20e7481576deb5b36 Merge: 25b75dfe64 4dcf7d9ebd Author: Mick Semb Wever Date: Mon Jul 12 22:39:19 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 25b75dfe64e350443afa020cfa9d226dd520a8c4 Merge: 58515c2de6 51f16a30b0 Author: Mick Semb Wever Date: Mon Jul 12 20:36:10 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 58515c2de6d54a6c1994dfa4c78d7452b6b5cd8c Merge: 7cc6331bb2 330cde6610 Author: Benjamin Lerer Date: Mon Jul 12 13:09:24 2021 +0200 Merge branch cassandra-4.0 into trunk commit 7cc6331bb211dbe2bb6b156c66c38ce3b2023446 Merge: f53f3890b1 652eb6c536 Author: Mick Semb Wever Date: Fri Jul 9 00:19:28 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit f53f3890b16e9d17b62731cac42269d0e4d391ed Merge: fbf101072a 80fd439be1 Author: Mick Semb Wever Date: Thu Jul 8 23:44:55 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit fbf101072a65a240eb4831583aeb72e97078737f Author: artsiom Date: Fri Aug 17 17:13:54 2018 +0300 Remove check on gossip status from DynamicEndpointSnitch::updateScores patch by Artsiom Yudovin; reviewed by Benjamin Lerer and Brandon Williams for CASSANDRA-11671 commit 5ef097477c3ffdc5f00b43fc3beb39e5478e6c07 Merge: bf92a67fd3 bff4e54cdd Author: Benjamin Lerer Date: Thu Jul 8 12:49:39 2021 +0200 Merge branch cassandra-4.0 into trunk commit bf92a67fd3010950475e4d85dfaca9fa3d7e2199 Merge: 1207891054 71efaa031b Author: Ekaterina Dimitrova Date: Wed Jul 7 13:49:32 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 12078910549c6c6e9474f8efb3ef274fa7de8209 Merge: cdaa2da91d 68af597e01 Author: Benjamin Lerer Date: Wed Jul 7 18:42:46 2021 +0200 Merge branch cassandra-4.0 into trunk commit cdaa2da91d8f411b7c144a20a0c6697b42c0febb Merge: e6a311f689 afcc0cc5c8 Author: Caleb Rackliffe Date: Wed Jul 7 11:15:29 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit e6a311f6898b1184d7d58826021a82cbda2f9bc0 Author: Andrés de la Peña Date: Tue Jul 6 13:20:34 2021 +0100 Fix AbstractReadQuery::toCQLString not returning valid CQL patch by Andrés de la Peña; reviewed by Benjamin Lerer for CASSANDRA-16510 commit f74f5d001ef0f3cd502dca6111c018fca3ca12b5 Merge: cc3d59af53 ba0f6220b8 Author: Bereng Date: Mon Jul 5 11:11:38 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit cc3d59af53d27136c2b33ae74b0574ddc7dbc434 Merge: c61196643d 8af0f90a52 Author: Brandon Williams Date: Fri Jul 2 13:50:17 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit c61196643df2904a5766dfb4652356d4bd10e76b Merge: 985b46d882 bb3a39ad4d Author: David Capwell Date: Fri Jul 2 11:31:22 2021 -0700 Merge branch 'cassandra-4.0' into trunk commit 985b46d882f884bc7c055e432fc49d3d9d9e7fa7 Merge: d402c4c902 1de960b433 Author: Mick Semb Wever Date: Fri Jul 2 15:46:36 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit d402c4c9025e980d5eb2b3b262c073498757e1e9 Merge: 296f65e8d1 65b42304f5 Author: Mick Semb Wever Date: Fri Jul 2 14:58:03 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 296f65e8d1c25f31a87481843d715f5b7dad9d7b Author: Marcus Eriksson Date: Wed Jun 30 16:15:17 2021 +0200 Log when compacting many tombstones Patch by marcuse; reviewed by Brandon Williams for CASSANDRA-16780 commit 3f6f7be63faf2d07b274caf25a2fa1dcbde14187 Author: Marcus Eriksson Date: Wed Jun 30 16:34:09 2021 +0200 Display bytes per level in tablestats for LCS tables Patch by marcuse; reviewed by Brandon Williams for CASSANDRA-16779 commit d5e8e81993311076b7c9759e864429fc500fe9c6 Merge: dfc3dbc18d d7270725a1 Author: Brandon Williams Date: Thu Jul 1 14:27:34 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit dfc3dbc18d0f4b517847464e7fe5217f04efb119 Merge: 3b0ceec674 2fb3d9e7ac Author: Benjamin Lerer Date: Thu Jul 1 14:08:52 2021 +0200 Merge branch cassandra-4.0 into trunk commit 3b0ceec674b4de1a59c9d39c8ee567899a651512 Merge: 700c290350 c79124d5e6 Author: Mick Semb Wever Date: Wed Jun 30 21:55:38 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 700c290350c86444c81e3ab3fa4c7988050ac82f Merge: 7f691b814d 07803c8cd2 Author: Andrés de la Peña Date: Wed Jun 30 12:24:25 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 7f691b814d4df985af9c0d768634032173ccf60e Merge: 4c4e84fe78 752213c70d Author: Bereng Date: Wed Jun 30 09:36:20 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 4c4e84fe786abedd7b1a622fed55ae26c74fe53c Merge: c3f3d67c05 96884ee705 Author: Brandon Williams Date: Tue Jun 29 16:00:11 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit c3f3d67c05103084e810ff8922027b8e1439d8ff Merge: 4adecb25e0 0fc876894a Author: Mick Semb Wever Date: Sun Jun 27 18:14:30 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 4adecb25e0036a2b8292ca21bf54f6ce4acfca51 Merge: e10ed27cfe 41820921bd Author: Mick Semb Wever Date: Sun Jun 27 17:38:00 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit e10ed27cfef2c79134f9ddd8dd3315a454de51f3 Merge: aac6f7db8c 1a10792b47 Author: Yifan Cai Date: Sat Jun 26 22:13:18 2021 -0700 Merge branch 'cassandra-4.0' into trunk commit aac6f7db8c8f493b8e28842903e6e2cb6838ac75 Merge: d7aaaa5001 6fc0715fe0 Author: Mick Semb Wever Date: Fri Jun 25 13:09:14 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit d7aaaa50013d1e275bc5836a48c64893e1365f43 Merge: 5f23e6d766 26e21bb576 Author: David Capwell Date: Wed Jun 23 13:11:27 2021 -0700 Merge branch 'cassandra-4.0' into trunk commit 5f23e6d766a18782db82d955bf380239990d2c84 Author: Caleb Rackliffe Date: Wed Jun 23 14:25:12 2021 -0500 Add isolated flush timer to CommitLogMetrics and ensure writes correspond to single WaitingOnCommit data points patch by Caleb Rackliffe; reviewed by Yifan Cai for CASSANDRA-16701 commit 05beda90a9206db165a3997a736ecb06f8dc695e Merge: 0c23498afc 340e511d84 Author: Benjamin Lerer Date: Wed Jun 23 10:37:11 2021 +0200 Merge branch cassandra-4.0 into trunk commit 0c23498afc75bd516d06d23c94bcaf597ff154e9 Merge: 3ed4a0f9ca 418bf14016 Author: David Capwell Date: Tue Jun 22 15:26:37 2021 -0700 Merge branch 'cassandra-4.0' into trunk commit 3ed4a0f9ca219378b90e4632342203275f04da1f Merge: 85096e4fba eda1aa2f18 Author: Andrés de la Peña Date: Tue Jun 22 14:57:16 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 85096e4fbacedd8e1202977ce17adcf63dd13179 Merge: eff1c38263 280ef74907 Author: Brandon Williams Date: Mon Jun 21 14:48:36 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit eff1c38263afa750e82b283c268a3fcad2304680 Merge: 968b506722 bc511581af Author: Marcus Eriksson Date: Mon Jun 21 15:52:27 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 968b5067228084ffeab1ecc454c3be479984b0fc Merge: ce422e571b 1af4c425e2 Author: Bereng Date: Mon Jun 21 08:36:18 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit ce422e571b0db8b2cd8c332e233315b9aa266733 Merge: c9130d454c 6c8f6bd663 Author: Mick Semb Wever Date: Sun Jun 20 18:04:02 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit c9130d454c4d43a05498c4fad7a1e3d434784791 Merge: 72dca97af9 d51f9d2655 Author: Benjamin Lerer Date: Fri Jun 18 15:17:52 2021 +0200 Merge branch cassandra-4.0 into trunk commit 72dca97af95c0f2388ce83eba601e41252e802c9 Merge: 9f7482ea00 bd1ffa7617 Author: Mick Semb Wever Date: Thu Jun 17 12:11:49 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 9f7482ea00b01bd0bca71e71f5ea967a360fbbe6 Merge: 5e29de6091 2f355c9462 Author: Caleb Rackliffe Date: Wed Jun 16 14:24:04 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 5e29de6091c6ea5fac3bcc981ca68dc162496dd0 Merge: c3a468c669 ad227d7ce9 Author: Brandon Williams Date: Wed Jun 16 13:53:03 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit c3a468c6691a127b3d611270edea0b218bedc3b5 Merge: db7f15dcd5 2573b08726 Author: Andrés de la Peña Date: Wed Jun 16 17:56:43 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit db7f15dcd5e7770a2959ca8989409b6629d10450 Author: Ben Slater Date: Tue Jun 15 21:51:37 2021 +1000 Clarify in the documentation that "Cassandra must be stopped" applies to the source cluster for sstableloader patch by Ben Slater; reviewed by Stefan Miklosovic for CASSANDRA-16746 commit e1f7691bd52085d312bd78a59145076429a8af2e Author: Stefan Miklosovic Date: Wed Jun 16 10:41:04 2021 +0200 ninja - move entry in CHANGES.txt for CASSANDRA-16669 from 4.0-rc2 among 4.0 fixes commit 597558f4e3743c8887434b526377900de3ba3f3f Merge: 5d3b698c65 379976d8e6 Author: Brandon Williams Date: Tue Jun 15 14:53:17 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 5d3b698c65e4fe4f41bc727b7828e59e3ad7a412 Merge: 58e198273a 6d21bd92c1 Author: Stefan Miklosovic Date: Tue Jun 15 14:38:29 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 58e198273a857d27d686fb514c94acb162222c77 Merge: eca21d9aea 4816ad257f Author: Brandon Williams Date: Mon Jun 14 11:00:44 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit eca21d9aea7b3437368a10a9121fcd0b46805878 Merge: a978754691 e2c3efe3d5 Author: Ekaterina Dimitrova Date: Sat Jun 12 14:05:26 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit a978754691770f588f062e057c4861cab71203e6 Merge: aaf653700f d93e43ed9b Author: Mick Semb Wever Date: Sat Jun 12 11:54:31 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit aaf653700febee2b6375a13387afe2dd23a349c3 Merge: d0a05b9ed7 7d4ee38acf Author: Benjamin Lerer Date: Fri Jun 11 10:57:30 2021 +0200 Merge branch cassandra-4.0 into trunk commit d0a05b9ed727a4a228d50c37eb6b366817998082 Merge: 698078fecf 0aa3a7a181 Author: Benjamin Lerer Date: Fri Jun 11 10:02:52 2021 +0200 Merge branch cassandra-4.0 into trunk commit 698078fecf3914b2a5f9d2ea344868f677f5afb2 Merge: 67cab368bc 1676add406 Author: Bereng Date: Fri Jun 11 08:50:44 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 67cab368bc89da6cab0efbf019f666f6d647ebb1 Merge: 6c30261bc8 c9734e3211 Author: Benjamin Lerer Date: Thu Jun 10 11:14:49 2021 +0200 Merge branch cassandra-4.0 into trunk commit 6c30261bc8d696f599a1e9ebea8a5bffbd0ac9ed Merge: 922872b38f dfd5a0b4a5 Author: Ekaterina Dimitrova Date: Wed Jun 9 15:30:51 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 922872b38f91cdf9d70c4612281a2cb5290bf127 Merge: f364a17881 c7795ee62c Author: Sam Tunnicliffe Date: Wed Jun 9 16:09:58 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit f364a17881ea96ab7ae352a75551aa0ab1bca1f1 Merge: a56bd764dc ddccb3fbbb Author: Brandon Williams Date: Wed Jun 9 09:37:30 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit a56bd764dcb3c3fbb87a9570b94aaa2518f32cfe Merge: 3b97e4bd7b 558d4ed50d Author: Bereng Date: Wed Jun 9 08:48:05 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 3b97e4bd7bc2f5f14bdf82e80a1d2e83bb50add5 Merge: 33ff36cc42 7d1c0131f6 Author: Ekaterina Dimitrova Date: Tue Jun 8 11:14:04 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 33ff36cc42a083311065dee56adc1d72d6866f2d Author: fibersel <0583463@gmail.com> Date: Wed May 26 23:04:57 2021 +0300 Add a system property to set hostId if not yet initialized patch by Abi Palagashvili; reviewed by Stefan Miklosovic and Paulo Motta for CASSANDRA-14582 commit 4fd21245727f4bd3b8a7cd6b8bf5a4a5cb14a328 Merge: 6f2807b9f5 f83f2a3570 Author: Ekaterina Dimitrova Date: Fri Jun 4 18:42:22 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 6f2807b9f5e3141b53309bdc40136e5118ff2b6d Merge: 51c6669676 520489791f Author: Benjamin Lerer Date: Fri Jun 4 13:23:36 2021 +0200 Merge branch cassandra-4.0 into trunk commit 51c6669676c1ce825930145115e542032ed447a5 Merge: 2227057ac3 f74ce55dab Author: Ekaterina Dimitrova Date: Thu Jun 3 20:51:47 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 2227057ac3dacba9bebd82d7cc3319c98d62cff4 Merge: 4acfd3bdf1 e6946e7ddb Author: Caleb Rackliffe Date: Thu Jun 3 13:30:44 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 4acfd3bdf1acdb6b28059a49dd39823d7ea0689d Merge: bedf6ca998 3875fd26ea Author: Ekaterina Dimitrova Date: Thu Jun 3 12:16:27 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit bedf6ca998744207bc0ded4fc3a2b80e0e1e8b86 Merge: 2316a9b7f1 aef535cae1 Author: Marcus Eriksson Date: Wed Jun 2 10:27:33 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 2316a9b7f116546d1c839bdcd77ef2f2c7288e24 Merge: 3af9b0a796 6ad259238d Author: Bereng Date: Wed Jun 2 09:33:22 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 3af9b0a796d0f87e3596103e4b90d8c5d0c5fad0 Merge: 4697e119af 0f25d1e31c Author: David Capwell Date: Tue Jun 1 20:51:34 2021 -0700 Merge branch 'cassandra-4.0' into trunk commit 4697e119afa42fd6c25a79e1700b4443535e63ca Merge: b978cb1d12 2dcb454caf Author: Andrés de la Peña Date: Tue Jun 1 16:27:33 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit b978cb1d12b88964bc5f894e3e5364b9cb22cc80 Merge: c29bdeed88 326802e583 Author: Yifan Cai Date: Mon May 31 11:19:42 2021 -0700 Merge branch 'cassandra-4.0' into trunk commit c29bdeed883ab6b840876ec70ee0bc0a432ac2fe Merge: 3b553d8e13 80ea99dc1d Author: Blake Eggleston Date: Fri May 28 16:28:49 2021 -0700 Merge branch 'cassandra-4.0' into trunk commit 3b553d8e13dbdbe59119de9c917d9aacc440741e Merge: 7e1afa1fe6 0d3ead9f55 Author: Andrés de la Peña Date: Thu May 27 16:35:36 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 7e1afa1fe60242d2252485b6225b7a59762c3741 Author: Brandon Williams Date: Wed May 26 16:50:14 2021 -0500 ninja fix six typo to be six >=1.12.0 commit f8f94c2f674fe61a280a4e72bcc84a283430632d Merge: 855dcad75b 55b4b8dd3b Author: Ekaterina Dimitrova Date: Wed May 26 16:45:45 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 855dcad75b39dbe9a1db42c75274d1f81cc53c7e Merge: 1e19a611b6 f4ab8cce1e Author: Brandon Williams Date: Wed May 26 13:57:40 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 1e19a611b60dd9dd6d31b4cbf3ab32f6550397c0 Merge: 36b140288f 6c8ab5b6b4 Author: Ekaterina Dimitrova Date: Mon May 24 16:34:17 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 36b140288f57c7d01d59444ce81c3c919a312766 Merge: 1936f8e6fb 1816f30206 Author: Ekaterina Dimitrova Date: Fri May 21 20:17:47 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 1936f8e6fba986222a00c2183fcda8b899bf82b4 Author: Ekaterina Dimitrova Date: Fri May 21 18:48:34 2021 -0400 ninja fix: temporary circle ccm fix commit 07504522c42ccd13ddd6fb3225400d84b1aa5777 Merge: 9f5886a4b8 d69ea28216 Author: David Capwell Date: Fri May 21 16:55:19 2021 -0700 Merge branch 'cassandra-4.0' into trunk commit 9f5886a4b8894c733e53c34457adae82034f9380 Merge: 1112cdf83c 658f3e11a6 Author: Ekaterina Dimitrova Date: Fri May 21 14:44:41 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 1112cdf83c36cca8cbab4712b4a709e0ba8ffe42 Merge: 9a432418f2 8cd02afce9 Author: Ekaterina Dimitrova Date: Fri May 21 14:22:51 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 9a432418f2277c40a1fe4b64049688d6354ecdca Author: Alex Petrov Date: Wed May 19 18:40:13 2021 +0200 Ninja: fix incorrect import for guava lib in row util commit fccbdae3b1a8fb63b145e6b7aac25926c6c81288 Merge: 2e9ce1b88f ae84406670 Author: Brandon Williams Date: Tue May 18 14:23:56 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 2e9ce1b88f8719787bf1ee69d5b491f0f2941135 Merge: 5d2c2f5ee3 77a370c424 Author: Ekaterina Dimitrova Date: Tue May 18 11:56:47 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 5d2c2f5ee39d7cd0a9bf419ce577735e25914424 Merge: d997b49a7c 2ca525b6b7 Author: Bereng Date: Tue May 18 06:41:22 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit d997b49a7c19d589c3988015b6e85378b9d034ed Merge: 9422d92603 0301fc6cea Author: Brandon Williams Date: Mon May 17 14:56:06 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 9422d92603591002d9d928e68832019512792244 Merge: 8a507d04b4 326d61e835 Author: Ekaterina Dimitrova Date: Mon May 17 09:10:33 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 8a507d04b41762da723092a1f2a37bdc6f53c632 Merge: 9d57d216f5 d35f36cd05 Author: Benjamin Lerer Date: Mon May 17 10:40:09 2021 +0200 Merge branch cassandra-4.0 into trunk commit 9d57d216f5530e5b80adb753e5daa0d469ae0944 Merge: f91418bbbe 9783d47b9f Author: Ekaterina Dimitrova Date: Fri May 14 18:04:42 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit f91418bbbedf4e0d5396becf14a5e884ac7f0d3b Author: David Capwell Date: Thu May 13 12:43:06 2021 -0700 Move CASSANDRA-14559s bootstrap_test.py::TestBootstrap::test_node_cannot_join_as_hibernating_node_without_replace_address into a jvm-dtest patch by David Capwell; reviewed by Stefan Miklosovic for CASSANDRA-16662 commit f637198484c74b71429b1bc884d3fe9a86a4a926 Merge: 8fb105f269 98b449f38a Author: Andrés de la Peña Date: Wed May 12 11:32:04 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit 8fb105f26966ba5d9ec1af5cab8e4b1a933c218c Merge: 8975a4f031 8a6dc6a723 Author: Bereng Date: Tue May 11 07:05:01 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 8975a4f031e56affb263604219e37385fcb5d7ff Merge: 2c3c3e639a 62e1d74701 Author: Ekaterina Dimitrova Date: Sat May 8 10:06:20 2021 -0400 Merge branch 'cassandra-4.0' into trunk commit 2c3c3e639aec8b7515db0e59b606adbea411626a Merge: abf4738bbb fb76baa608 Author: Andrés de la Peña Date: Fri May 7 17:59:14 2021 +0100 Merge branch 'cassandra-4.0' into trunk commit abf4738bbb11042d3cb83b10d3d8de2836e3c5c0 Merge: affd3b6eb3 7917ab8121 Author: Benjamin Lerer Date: Fri May 7 11:11:16 2021 +0200 Merge branch cassandra-4.0 into trunk commit affd3b6eb378d287f049d4080fb1e4953839f140 Merge: 089509922e 34a1e5f57a Author: Brandon Williams Date: Thu May 6 10:28:47 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 089509922e95249310575bdfc8be7c88b811bed4 Merge: 5fc811210b a867abf030 Author: Benjamin Lerer Date: Thu May 6 14:33:29 2021 +0200 Merge branch cassandra-4.0 into trunk commit 5fc811210b70889084b2be5a6a3577963fd5f3c4 Author: Aleksei Zotov Date: Sat Apr 24 14:25:56 2021 +0400 Refactor NodeTool commands tests patch by Aleksei Zoto; reviewed by Benjamin Lerer for CASSANDRA-16629 commit 8437c76e51a9443e9cdf0ac004441af6146935d2 Merge: e087afee05 42138f12b7 Author: Benjamin Lerer Date: Thu May 6 12:16:17 2021 +0200 Merge branch cassandra-4.0 into trunk commit e087afee057b132927795864efbc625e4a9e1561 Merge: 2a1e537855 c4fab930ca Author: Benjamin Lerer Date: Thu May 6 11:46:21 2021 +0200 Merge branch cassandra-4.0 into trunk commit 2a1e53785593571fd7da7a47534b25a1f318e76c Merge: f55baa4dbf 7e4c77c968 Author: Bereng Date: Thu May 6 10:02:17 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit f55baa4dbf868b2ef6284d5a887f6d76e62944fd Merge: 642e95a4a8 b67d04b6fe Author: Mick Semb Wever Date: Wed May 5 18:07:12 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 642e95a4a80e2ce43272e8fa8a02306ad7a267b8 Merge: 930e9a1c8c a3db11831a Author: Brandon Williams Date: Wed May 5 07:22:36 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 930e9a1c8cde725275c8614b21321fe9fd22c581 Merge: 74a2ef3654 a75b9b8533 Author: Brandon Williams Date: Tue May 4 16:18:35 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 74a2ef3654b551b518de619f6fdcfdf8b24b4f6f Merge: bdf0a0bc26 d2fcdacfa1 Author: Brandon Williams Date: Tue May 4 16:11:06 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit bdf0a0bc2671e038b68100e772e28587283f2ba8 Merge: 7e673e0bab 327d7c1803 Author: Brandon Williams Date: Tue May 4 11:53:41 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 7e673e0bab1abe506bca67dd02fe9cb0427ee3dd Author: Brandon Williams Date: Tue May 4 11:31:30 2021 -0500 Fix CHANGES for 4.1 commit 0cbe920ebd4becf6e023c36b4b51f6710f2a3d3d Merge: 58727f7184 0c7833a52b Author: Mick Semb Wever Date: Tue May 4 18:21:24 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 58727f71841d36020328602c0e64f0af361c5491 Merge: 48570e91b1 288bb9d902 Author: Brandon Williams Date: Tue May 4 08:46:06 2021 -0500 Merge branch 'cassandra-4.0' into trunk commit 48570e91b1f328650b2c791b09a22367ee0a86cf Merge: 5bd8207cd7 b69297ebb2 Author: Bereng Date: Tue May 4 09:44:27 2021 +0200 Merge branch 'cassandra-4.0' into trunk commit 5bd8207cd78d247b8110a49567c4787025f6cc4f Author: David Capwell Date: Mon May 3 20:49:33 2021 -0700 Fix flaky test testHasVersion3Nodes - org.apache.cassandra.gms.GossiperTest patch by David Capwell; reviewed by Ekaterina Dimitrova for CASSANDRA-16651 commit d2b57fe463ebb6a0921f5c6efa87efd5f36263c5 Author: Mick Semb Wever Date: Sat May 1 12:13:59 2021 +0200 Create release branch cassandra-4.0, increment trunk version to 4.1 commit 1a801c68707a29455dfe20c0a33b75b051d0d4f4 Author: Jeremiah D Jordan Date: Fri Sep 15 11:08:07 2023 -0500 Revert "increment version to 4.0.12" This reverts commit 2584f4f0709df2c1031171ac9bbec71376adaba8. commit 05c6c238da4b37032884d1d21321f5d4ed3bf84f Author: Jaroslaw Grabowski Date: Wed Sep 13 11:22:34 2023 +0200 CNDB-7490 fix IndexParallelBuildTest by unloading sstables gracefully with proper ref releasing. (cherry picked from commit 0aa0d87e81dfaba85458f14ac34c2a0be6447535) commit 64aa3dad33f7d55ed869ce465f9b22683afe74e5 Author: Jaroslaw Grabowski Date: Wed Sep 13 08:54:48 2023 +0200 CNDB-7490 fix compaction tests by properly mocking getDataFile method. The method is now used to keep name to sstable map in View. (cherry picked from commit 9450ca03aa664dc0ef9099969bcd6882a37e6459) commit e8d38da264175290b13342b052f536a4473e6cf4 Author: Jake Luciani Date: Tue Sep 12 14:16:01 2023 -0400 Move properies to proper place and default to show all system views (cherry picked from commit 1aab8585e025481d9160e3a43e64556aea8c0968) commit ec6afe7009509452e9236f1ee40e6b0a2b369337 Author: Jaroslaw Grabowski Date: Tue Sep 5 13:42:57 2023 +0200 CNDB-7490 more flexible system_view content SystemViews content is managed by clumsy system properties. (cherry picked from commit 6ac04f9b3f3e7ce45a658366b60b5c2940de2d34) commit 9b2ff6dd4ac3981b93e8110fa19f3b7338111745 Author: Jaroslaw Grabowski Date: Tue Sep 5 12:07:05 2023 +0200 CNDB-7490 sstableByFilename map Surprisingly, sstable.getFilename() doesn't return file's name. The method returns absolute file's path. Use file name instead. (cherry picked from commit fe001ef63b3d17de82221c7cdf71def4551e4f50) commit b2cbaedf1353ef89a1f0a59efdaea86e94943b64 Author: Jaroslaw Grabowski Date: Mon Sep 4 15:43:25 2023 +0200 CNDB-7490 fixup identifier for graph labels (cherry picked from commit 7dfba72be14b444d8e54ff4ff4618c4257613d50) commit f67ec7d4ad79a7c6bcf61e50a71e077260c6312f Author: Jaroslaw Grabowski Date: Fri Sep 1 12:40:19 2023 +0200 CNDB-7490 move ReplicationStrategy out from StrategyAdapter (cherry picked from commit 88e5e04c3a03394bc335990d79067fd134d01d34) commit 3e449784079ea2fabcebee155a3dc3a97e1542b3 Author: Jaroslaw Grabowski Date: Thu Aug 31 15:28:44 2023 +0200 CNDB-7490 make evaluateReplicatedOwnership public for CNDB (cherry picked from commit 863eb9e90049f56ef4c190969b7e74e5e4e25d6f) commit f21eb46819094dcfe5657316c7c700e5d9153d66 Author: Jaroslaw Grabowski Date: Thu Aug 31 13:48:39 2023 +0200 expose SSTableReader getLiveSSTable(String filename) for CNDB (cherry picked from commit db01e9ddd0eaeff5ddd3450ec1d027c1d378c394) commit e4a23e80d278484811ad466fafa409016df33975 Author: Jaroslaw Grabowski Date: Thu Aug 31 13:15:27 2023 +0200 make SIM.buildIndexesBlocking public for CNDB (cherry picked from commit a0fa6f4fbc9c74b83e3da68e4fefc187f7c91619) commit a33c65bea125d8803062af1a760a2c94b272ad1f Author: Jaroslaw Grabowski Date: Thu Aug 31 10:25:46 2023 +0200 port: CNDB-7385: Fix withMigratedKeyspaces in SerializationHeader (#20838) (cherry picked from commit ee96a481c15d7e6768e81931f9c9dbad2408de36) commit 004d203e938010e66a95e4e2874641eb2a23721f Author: Jake Luciani Date: Wed Aug 30 21:25:44 2023 -0400 Refactor TokenAllocation.StrategyAdapter and visibility of index building (cherry picked from commit 03c336ac8dc83df88f82ddfa5d8ac07fe84072eb) commit 71947ba550574c58c73c927f9ad6c591e16385a1 Author: Jake Luciani Date: Wed Aug 30 18:12:04 2023 -0400 Change visibility of TokenAllocation.StrategyAdapter (cherry picked from commit b09c790c796a4506c5e62c0fa35694078dfedb01) commit d0c96b2282c5428952e79a881e9c0b8ad0a6a6e3 Author: Jake Luciani Date: Wed Aug 30 17:47:50 2023 -0400 Port: cndb-7057-followup: call maybeUpdateSelector() in getCompactionArenas() (cherry picked from commit 79a53d842777283a6bd900600d93bb9a647a2ffb) commit beb767698fc1549787fb08b76da463eb28633b1e Author: Jake Luciani Date: Wed Aug 30 17:44:22 2023 -0400 CNDB-7057 Fixed regression causing index builds for flushed sstables. (cherry picked from commit ca50456b081635108c634403cfa2159b17f3cbeb) commit af857168d230ee7bba0de978c3ac22c973952e56 Author: Jake Luciani Date: Wed Aug 30 17:12:51 2023 -0400 Port: CNDB-7057 Support building indxes on the Compactor and optionally disable that in Writers (cherry picked from commit 0452d6bcdfab0b1b4aaaa476cdd8f815ff9eb9af) commit 674ab1476f85376beadb13ace820254796ba4420 Author: Jake Luciani Date: Wed Aug 30 13:40:56 2023 -0400 Port: CNDB-7336: fix BKD reader to use readVLong for doc count (cherry picked from commit 2974729a58b7701979cd02fc0eb5a5905f4034ba) commit 73bf87d46aeea1a935a36250142d30e189a559ca Author: Jake Luciani Date: Wed Aug 30 13:23:44 2023 -0400 Port: CNDB-7103. Make index build parallel for reloaded sstables (cherry picked from commit d1cc33ea13845c654c10122c7e4fbeea2f83f613) commit f74d3d01a833d4c51602b97f3d8b5ce68e4bff64 Author: Jaroslaw Grabowski Date: Wed Aug 30 14:49:30 2023 +0200 port: cndb-7151: add SSTABLE_COUNT to IndexesSystemView (cherry picked from commit 89c2de8363585b2c9ca2ad8b40eb88a1c11413ac) commit 64133e72e1ca509ab5969940e646aa2843627893 Author: Jaroslaw Grabowski Date: Wed Aug 30 14:43:13 2023 +0200 port: CNDB-6965: make hints replay rate limiter configurable - Allow update hints dispatch concurrency in runtime (cherry picked from commit 5d81e2a210ce5e584d7829fec2f6d5e913ab9990) commit f2a2f45b59e1f870e89d85bc5c2cacc71cc52d31 Author: Jaroslaw Grabowski Date: Wed Aug 30 14:27:02 2023 +0200 port: CNDB-7126 Make index build async for remote reloads (cherry picked from commit 391efeb40a90d89055da954e271f7ed96c60a892) commit ce728a963ef50a333623bf26ecb275003c3a0f0a Author: Jaroslaw Grabowski Date: Wed Aug 30 13:20:31 2023 +0200 port: CNDB-6939 Fix Tables#withNewKeyspace and add the possibility to copy a KeyspaceMetadata by transforming TableParams for all tables/views. (cherry picked from commit 6bc77e580e3770654b33cc66334018024291ccd4) commit 4a3171f544d78b2c4591174486cb8e66ce505fe1 Author: Jaroslaw Grabowski Date: Wed Aug 30 11:32:08 2023 +0200 port: CNDB-5291 open token allocation API Token allocation StrategyAdapter is now a part of the API. It will allow CNDB to force single replica per rack allocation with custom token filtering (by service dc, rack and generation). (cherry picked from commit 449642476b0b052cb7d736c79f736bc8ef4798b3) commit cdd736011533bb7bdee007761d40207cb94461ba Author: Jake Luciani Date: Tue Aug 29 13:56:38 2023 -0400 Fix compile (cherry picked from commit 53891719dc467922099499b4c6c0cf70ed52d061) commit 92caa3ef9d7f7c596e6bc890ac86456bedfed06e Author: Jake Luciani Date: Tue Aug 29 12:43:07 2023 -0400 Port: CNDB-2665 Add coordinatorReadSize metric. (cherry picked from commit 803f0bdcc3c17c077e652457698f3072b6ff4663) commit fd76e996cc2cf360ed506c3f1097cdf32bdc9727 Author: Jaroslaw Grabowski Date: Tue Aug 29 10:04:35 2023 +0200 port: DSP-23333: Limit row size during deserialization (#20725) Do not allow reading more bytes than the declared row size in the row preamble when reading from the sstable. This is achieved by wrapping the input source with TrackedDataInputPlus which has now a new feature of limiting the read data. When the sstable is corrupted, we can try to read much more data for a row than we should. In some cases, it can lead to OOMs when we read malformed cell counts and sizes. (cherry picked from commit a51f37a20b6303b19fc3cfdf04f1f9724f64d31e) commit ffbb194b546aed82f6571adc834fac8a6df44924 Author: ethan-brown2022 <107228208+ethan-brown2022@users.noreply.github.com> Date: Wed Sep 6 15:59:17 2023 -0700 Revert "CNDB-7135 Adaptive UCS Metrics in CNDB (#684)" (#708) This reverts commit 5fe9c507ae97ffdbee204e6adeff511ccf72f5f7. (cherry picked from commit 59c7827d4c7862ebb565c21e9f35f70691559cbe) commit f638dc0390d565e036563dcebe4bec3c37057623 Author: Jakub Żytka Date: Wed Aug 30 15:02:25 2023 +0200 STAR-1364: fix RecoveryManagerTruncateTest::testTruncate (#700) RecoveryManagerTruncateTest::testTruncate incorrectly expected commit log replay to replay some mutations. The condition has been changed to expect no mutations being replayed for the keyspace in which the truncated table resides (cherry picked from commit befebc4a1542ba2368a122cf6a425ff84d3db188) commit b42b6b8b1187b9d8f7729511a0534d312c44c450 Author: ethan-brown2022 <107228208+ethan-brown2022@users.noreply.github.com> Date: Mon Jul 24 16:27:06 2023 -0700 CNDB-7135 Adaptive UCS Metrics in CNDB (#684) - Added MetricsNotification as another implementation of INotification - Added ColumnFamilyStore.publishMetrics() that is called every X minutes by CompactionManager for each cfs - Added Tracker.publishMetrics which creates a MetricsNotification and calls handleNotification for each subscriber - Added system property cassandra.publish_metrics_interval_minutes to enable or disable publishing metrics (disabled by default) (cherry picked from commit 5fe9c507ae97ffdbee204e6adeff511ccf72f5f7) commit 005f7ebc0e9177e3569fa4081f3f6422fb7f22e8 Author: Daniel Jatnieks Date: Fri Sep 1 16:21:23 2023 -0700 CASSANDRA-18366 Fix FailingRepairTest by making sure gossip is started before each test. commit 9ba3407d3372c076edec01625372c547e1179f4d Author: Daniel Jatnieks Date: Tue Aug 29 16:03:18 2023 -0700 Fix SonarCloud reported bug in CassandraTableRepairManager and re-interrupt thread after catching InterruptedException commit 73a7b931843e7e5de90911955e94ab151f833bc6 Author: Daniel Jatnieks Date: Tue Aug 22 16:04:07 2023 -0700 Fix IllegalAccessError in jvm-distributed tests by removing @Shared annotation from NativeLibraryWrapper and subclasses commit 58301651fa85cb7073f2a9e22e9c4e59ea401bab Author: Branimir Lambov Date: Thu Jul 6 15:46:40 2023 +0300 Switch to uppercase K for kilo in line with what's used in other places in the OSS codebase (cherry picked from commit 50c4c16947864c39c4d8ec01f0523a1f5be2ad8f) (cherry picked from commit 718370bf957f564205c8bc248f06c33250dac85f) commit 646ef97ba95f79529d14c196f0638e543c91aecb Author: Branimir Lambov Date: Thu Jun 22 16:54:40 2023 +0300 STAR-1880: Support reserving threads for level or below for UCS Change the default reservations to max in level_or_below mode. This should work well enough to make configuring reservations unnecessary for most if not all users. (cherry picked from commit 44cac92a4e7bac3b51ab759637d38f4b544c601b) (cherry picked from commit 799b1ec174c4de34ed9c4711accde0dbef99b32f) commit bd98d0759e6dce7c6bb1affb5391936ae4818217 Author: Jaroslaw Grabowski Date: Thu Jul 6 12:34:50 2023 +0200 CNDB-6968 open UCS api (#650) - make splitInNonOverlappingSets public, it will be CNDB to parallelize GC work (cherry picked from commit c26b250943de382cf3a461b8ac1ed3d8b04a1542) (cherry picked from commit 240b2f82faae106ca8c669ae77381348b033f791) commit de5cfb69978ffb634dd69b047faca6411f644288 Author: Jaroslaw Grabowski Date: Tue Jul 4 07:54:40 2023 +0200 CNDB-7007 return expired tables level from getLevels (#656) (cherry picked from commit f2380881df754e6a25189e89eae87bbacc587c5f) (cherry picked from commit f35a00b42e8b333d6f555b3790049cfce29aebf6) commit 8f799ad68ee74fbfbd4e5b3acb182d306a749569 Author: Branimir Lambov Date: Tue May 30 19:20:01 2023 +0300 STAR-1878: Add additional configuration parameters and tests for UCS density - Add sstable_growth and reserved_threads_per_level - Add support for V1-compatible mode - Prioritize lower levels - Test and fix shard number calculation for multiple nodes/disks - Remove l0_shards_enabled - Deal with sstable repetition in non-transitive modes - Extract Overlaps class - Improve documentation - Add a lot more tests (cherry picked from commit 10a2a31ae846da3518bf9bbb8e67ee383c5eaa44) (cherry picked from commit bdce696ee2699e9fdec80cb62c5ad8b3ba9bf6e2) commit e28c206dde6c0d80261f4d48e25bb9428d3b49ef Author: dan jatnieks Date: Wed Jun 21 16:57:48 2023 -0700 CNDB-7129 Remove guardrail check for disallowed write CL's from SelectStatement (#675) (cherry picked from commit 2b22984fe144cd0d5b964d8d701132c9e96005cb) (cherry picked from commit 662afa606808715f187c22000d158409207105e1) commit 17c92002d71a1e51adb01ae9fb432b39fbc2c5a9 Author: ethan-brown2022 <107228208+ethan-brown2022@users.noreply.github.com> Date: Wed Jun 21 10:02:26 2023 -0700 changed logging from warning to info when there is no saved controller config file (#674) * changed logging from warning to info when there is no saved controller config file * added checks to not store controller config files for system tables * Separate logging for the case where there is no file (debug) and the case where the file is present but unreadable (warn) (cherry picked from commit d5ffa002385b735e2cebb9aac445dba56762380a) (cherry picked from commit 90b9436325003e845956bbd848e0b5c878321443) commit d7f3ca4bf06da0db7d12a4263b3e3b6048729da5 Author: ethan-brown2022 <107228208+ethan-brown2022@users.noreply.github.com> Date: Thu Jun 15 08:55:27 2023 -0700 Star 1864 (#671) * Removed keyspaceName and tableName from parameters in Controller.fromOptions (cherry picked from commit e9ddd5f0d9b9c3943c1fb738591ecf8053c32d72) (cherry picked from commit 61ce2240e42fefc63fd2920ffd5d8e7a4973c9dc) commit 720a61c4858edeae275bcf889e159c9d89ca532e Author: Matt Fleming Date: Tue Jun 13 17:15:50 2023 +0100 CNDB-7037: Check that memtable exists before flushing and avoid IOOBE (#664) (#666) For offline services such as the compactor it's possible to not have live memtables. Account for this during flushing after removing indexes to avoid triggering an IndexOutOfBoundsException: Error happened while updating the schema java.lang.IndexOutOfBoundsException: Index: -1 at java.base/java.util.Collections$EmptyList.get(Collections.java:4483) at org.apache.cassandra.db.lifecycle.View.getCurrentMemtable(View.java:106) at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:1115) at org.apache.cassandra.db.SystemKeyspace.forceBlockingFlush(SystemKeyspace.java:552) at org.apache.cassandra.db.SystemKeyspace.setIndexRemoved(SystemKeyspace.java:576) at org.apache.cassandra.index.SecondaryIndexManager.markIndexRemoved(SecondaryIndexManager.java:837) at org.apache.cassandra.index.SecondaryIndexManager.removeIndex(SecondaryIndexManager.java:420) (cherry picked from commit 4252ad472d594fcd49af31ec858a6cbdb1769667) (cherry picked from commit 1c9d70a992307117549328175fb3a2358cbd8a7c) commit 2cc0558429bc08a9de5e2154bf5deb491a38af3c Author: ethan-brown2022 <107228208+ethan-brown2022@users.noreply.github.com> Date: Tue Jun 13 08:41:55 2023 -0700 STAR-1876 Fix schema inconsistencies between static and adaptive UCS(#658) - AdaptiveController no longer uses STARTING_SCALING_PARAMETER. - AdaptiveController now recognizes and uses SCALING_PARAMETER_OPTION or STATIC_SCALING_FACTOR_OPTION (deprecated) allowing initial scaling parameters to be set for each level instead of one scaling parameter applied to every level. This also fixes potential schema inconsistencies when switching from static to adaptive. (cherry picked from commit f8421e911bd896e05027f03b145bdd13ad475c1b) (cherry picked from commit 378f2dc0d71afa8db2d6e4c04909fea8dc1a8d0c) commit ded52a2419838eaaf77b9151d18b45984d6f3c7d Author: ethan-brown2022 <107228208+ethan-brown2022@users.noreply.github.com> Date: Tue Jun 6 15:16:00 2023 -0700 STAR-1864: Storing Scaling Parameters on disk to keep configuration when a node restarts (#615) Store current flush size and current scaling parameters for each table in a file to keep configuration if a node restarts. (cherry picked from commit 8a7a6d9c4fc8d1230b068f498a33073bf27f4fa6) (cherry picked from commit 974081e9160842a996b65db92a47d78d55d6faa2) commit dbcb203dacfa323a079b3b213057e4346bb069b3 Author: dan jatnieks Date: Tue Jun 6 13:08:40 2023 -0700 CNDB-6974 Remove the table and keyspace thresholds from cassandra.yaml (#651) (cherry picked from commit 6c6622ab5a96f9f21f7500643bd530cbaaa6ca0b) (cherry picked from commit 392b0bd2c608f678b2832127f8752ddfa40b3b1b) commit 707c92a9ea0973ccf12175608102e99e98209495 Author: dan jatnieks Date: Sun Jun 4 20:08:06 2023 -0700 CNDB-6974 Remove table and keyspace thresholds in CC (#646) Table and keyspace thresholds are deprecated in C* 4.1 with CASSANDRA-17195 and planned to be removed. (cherry picked from commit 645ddc47c327bec6eaa7650e7412d106ea2d4c08) (cherry picked from commit 36dc86ab76f1d6d94e8a5a0a3496b3a82bb272b8) commit 2be477df7deb087717649a7226c85c5daafa1aba Author: Piotr Kołaczkowski Date: Thu Jun 1 16:22:33 2023 +0200 CNDB-6976 Set SEVER_VERSION and PRODUCT_TYPE Converged Cassandra needs to set SERVER_VERSION and PRODUCT_TYPE in the OPTIONS message returned to drivers on connection. The drivers use those OPTIONS to know they are talking to astra and to change the default CL to LOCAL_QUORUM for writes. (cherry picked from commit 1f9179002a8b9fc6c65f1c2a40b84be5c47985b6) (cherry picked from commit 8bf048175fc3a0c9c68489ec7320a3c7ef409f1f) commit 9958751ed1f08f6c2a0aaa790837f16158c96111 Author: Jaroslaw Grabowski Date: Thu Jun 1 21:18:49 2023 +0200 CNDB-6968 adjust UCSv2 to CNDB (#638) - expose bucketMax - adjust types in getLevel (cherry picked from commit af4c83aef4ff4f2e9056e0ed44af64f75cb3081a) (cherry picked from commit dc34c2e3b5dda4379a5cfb00af7f5847ec0f530e) commit aac142a02a79763b813faf33612fbf5e10b06a61 Author: Andrés de la Peña Date: Wed May 31 11:38:16 2023 +0100 Add method to get all the registered pre-built functions of NativeFunctions (cherry picked from commit 23c2891e7a0f9a52ac99ee0eab9602503499820a) (cherry picked from commit bcbc9b8f2aa99974f84397f517650e01962593ba) commit e0cc82a780d904a10cba600c69a4bc7667a6bb72 Author: Branimir Lambov Date: Tue May 30 12:17:10 2023 +0300 STAR-1569: Fixes in limit calculation (cherry picked from commit 86df78a0f69780e2d3675ee3bffd0a319bfb03fa) (cherry picked from commit f385eb3fd05d12bf5231b56ca1cbbcf8b9f95b7a) commit cb3c3c01ae337661f53404dfc55b8df490b8360b Author: Jonathan Ellis Date: Tue May 30 10:13:31 2023 -0700 backport FunctionFactory for vector similarity functions (#630) * cherry-pick 6da9e336 * add TimeUUID instanceof check to CQLTester * r/m tests exercising maxwritetime, which doesn't exist in CC yet (CASSANDRA-17425) * r/m tests exercising sub-cell writetime, which doesn't exist either * clean up merge --------- Co-authored-by: Andrés de la Peña (cherry picked from commit a3b866174620880b32384131732d80b901859aec) (cherry picked from commit bf490bcb2d3c3450d55d8b1fc77f31cb0983cac4) commit 96bb45a7bd01ce6199f4923fb3be82f9241a675b Author: Branimir Lambov Date: Tue Jul 12 19:10:57 2022 +0300 STAR-1569: Density-based Unified Compaction Strategy Changes UCS to rely on density (i.e. size divided by token coverage) rather than size + fixed sharding. This allows it to assign levels to sstables before defining shards and thus to increase the number of shards as the density grows (i.e. as data moves to higher levels of the compaction hierarchy). UCS now uses a target SSTable size to determine splitting points and can correctly understand the layouts produced by legacy STCS and LCS, which enables efficient upgrades. Instead of relying on fixed sharding points, it will now evaluate overlap and form buckets based on sections of the token space where sstables overlap. See UnifiedCompactionStrategy.md for more details. (cherry picked from commit 6b474341fc4fd20ee32b82cce343a99f8d8a9ce3) (cherry picked from commit c66cc08cef21ff045cb37a0ce276d272e4dd2b84) commit e01c5f8e6012603349b501ff3a63ca99e9837520 Author: Branimir Lambov Date: Thu Feb 9 15:31:45 2023 +0200 STAR-1569: Density-based Unified Compaction Strategy preparation - Generalize prettyPrint, add parseHumanReadable and test Introduces prettyPrintBinary (with 2^10 multiplier, e.g. MiB) and prettyPrintDecimal (with 1000 multiplier, e.g. ms) that cover the full range of double, and a corresponding parsing method that can read both. - Switch to more friendly scaling parameters Replaces the integer W with levelled/tiered + fan factor indicators, e.g. L10 (levelled, fan factor 10) for -8 and T4 (tiered, fan factor 4) for 2, N (neutral) for 0, in all scaling parameter printouts. Input is accepted in either form, including T2 and L2 in addition to N for 0. - Fix null CommitLogIntervals - Rename increaseSlightly to nextValidToken (cherry picked from commit 0f9b8afa2fcca6b09e615e9ab8f761982246e50d) (cherry picked from commit 0405665715ce4e6a24beca0c226c18ac43348e1e) commit 236c2ab10fa7c7cc0465f01fd62b4caa810d32bb Author: Jaroslaw Grabowski Date: Wed May 24 11:58:41 2023 +0200 CNDB-6522 port riptano/bdp/pull/20676 (#610) (cherry picked from commit acd10a38657e366fd3dc0c8e538a4a742d270bba) (cherry picked from commit 0f8731c868523ee22e55730a6ec510758e528b14) commit 4c535fa81743e063b162fb9babd0338a4778316d Author: Jacek Lewandowski Date: Mon May 22 09:27:34 2023 +0200 CNDB-6857: Add a method to override keyspace in serialization header Note that, although we provided a test to verify that, that test did not fail when run without that additional operation. The fix is added just to ensure compatibility with DSE. (cherry picked from commit 91c3a329578f3cf2f162e2e9337b826e99c9a1e9) (cherry picked from commit c0663c45f544f21f7edc924316dc8a14c720929d) commit 745f0d832407b6a3543bcefbe9b216e6ccefb762 Author: ethan-brown2022 <107228208+ethan-brown2022@users.noreply.github.com> Date: Wed Apr 26 12:54:08 2023 -0700 CNDB-6577 sstable-cleanup fails verification tasks due to invalid UCS property name (#606) * added 'Void' type in Timer.java * - Either "static_scaling_parameters" xor "static_scaling_factors" is accepted - Changed default readMultiplier to 1.0 instead of 0.5 for adaptive compaction cost calculations * added coverage for 'static_scaling_factors' option * renamed 'static_scaling_parameters' to 'scaling_parameters' * fixed failing tests (cherry picked from commit 89e9cb7c69feba7bd5ba55a58e06078a01eb7b7b) (cherry picked from commit 3ba08bd6e75092b41f865db00607033ad827486e) commit e24c80993ec222875070570a668af21ef0e4c9af Author: ethan-brown2022 <107228208+ethan-brown2022@users.noreply.github.com> Date: Wed Apr 26 10:04:00 2023 -0700 STAR-1864 changed default readMultiplier from 0.5 to 1.0 (#609) * added Void type in Timer.java for testing * changed default readMultiplier to 1.0 instead of 0.5 * removed changes to Timer.java (cherry picked from commit 57013da1d2d4bd0bee33f8a9778671c2eaa8297a) (cherry picked from commit f521a6a9e3eb71a13798a1866528ad242c248611) commit 44995b751e32925c54f7cd554e1189f494cc1015 Author: ethan-brown2022 <107228208+ethan-brown2022@users.noreply.github.com> Date: Wed Apr 5 12:20:05 2023 -0700 Star 1864 (#601) - Changes to W are applied to level 0 first, and applied to higher levels on future calls if the choice remains the same. - Added max_adaptive_compactions option that sets a limit on the number of concurrent "adaptive compactions" (compactions triggered by changing W). - Updated read costs to include all sstables checked regardless of hit. - Made readMultiplier and writeMultiplier configurable options (costs_read_multiplier and costs_write_multiplier) and changed default readMultiplier to 0.5. - Compaction costs are only updated if inputDiskSize > 0. (cherry picked from commit 7f34b7c48df3aa7dca14229f1aac7797b63ea234) (cherry picked from commit 3b26eeb6ebf9db9537cca904e5724c50c66a6e43) commit cac3062aaefc25c1471bc6f422998aa34884d7b5 Author: Sergio Bossa Date: Thu Mar 23 12:53:38 2023 +0000 CNDB-6178 (#599) * CNDB-6178 Introduced new Timer abstraction to schedule timed tasks efficiently via Netty's hashed wheel timer. * CNDB-6178 Caching Replica objects in SystemReplicas with daily expiration to avoid accumulating "dead" replicas, which could happen on Serverless where replica churning is way more common. * CNDB-6178 Introduced IO stage to schedule potentially IO-bound tasks. * CNDB-6178 Introduced the following to facilitate Logged Batches implementation in Serverless: * Allow to filter batchlog endpoints by preferring the local rack, to reduce network cost. * Minor code visibility and interface related changes. Co-authored-by: Massimiliano Tomassi (cherry picked from commit c25a43d626965cddbb66e3c7a84ae40c9aaeea0a) (cherry picked from commit 560d28e48f90ef96e84f2761742456bd5f969ec4) commit 7befa098e76f7d61ea1c0f5fe677b938a5bee78e Author: Jaroslaw Grabowski Date: Fri Mar 17 18:08:46 2023 +0100 CNDB-6522 add NativeMemoryMetrics (#605) - remove aligned allocation metrics - CC doesn't have aligned allocations (see CNDB-6521) - add allocation counting to MemoryUtil that is used for native memory allocation, expose the value in NativeMemoryMetrics - point bloomFilterMemory to correct CC class counting bloom filter used memory - Workaround Java surprising behaviour around empty direct buffers. ByteBuffer.allocateDirect(0) allocates bytes, this breaks our expectations about memory capacity / memory used in NativeMemoryMetricsTest. See MemoryUtil for details. --------- Co-authored-by: Massimiliano Tomassi (cherry picked from commit fc0fcf98226fa5e5300e247c81f0a44300e18f7d) (cherry picked from commit 84e4f36e354d371ee6dfdaf5528b31392f55e27d) commit 82e6ae7ddcb09b934f15ee8fcb2845ecf00d8857 Author: Jaroslaw Grabowski Date: Wed Mar 15 14:40:36 2023 +0100 CNDB-5691 port #5205 table metrics aggregation (#600) * port #5205 configure table metrics aggregation via table extensions The default aggregation is set via cassandra.table_metrics_default_histograms_aggregation, the default is INDIVIDUAL (no aggregation). CNDB will default to AGGREGATED. The setting may be set per table via ALTER/CREATE statements. The custom value is stored in the table schema to survive restarts. From the original commit message: An extension has been added with key HISTOGRAM_METRICS and binary value either 0x00 or 0x01, with 0x00 meaning aggregated keyspace histograms, and 0x01 meaning individual keyspace histograms. Unfortunately extension payloads must be binary, which is why I chose to use a single byte rather than an encoded string. These values are grouped into an enum, the MetrcsAggregation enum in TableMetrics. * port #5205 metric aggregation This patch provides the infrastructure required to reduce the cardinality of table metrics. Tables can either use individual metrics, as before, or keyspace metrics. This is controlled by a table metadata extension. A system property determines if, in the absence of this extension, tables should use individual or keyspace histograms by default. The default of this property is individual histograms for C*, but CNDB services will set this property to switch to keyspace histograms by default. TableMetrics.Table[Meter|Timer|Histogram] classes were modified to work without table metrics when TableMetrics#metricsAggregation == MetricsAggregation#AGGREGATED. The classes forward update calls to parents as usual, but skip table metric if it's missing. When asked about the current metric value they return either table or aggregated keyspace metric depending on TableMetrics#metricsAggregation. Additionally, an equivalent class was added for LatencyMetrics - TableMetrics.TableLatencyMetrics. It serves the same purpose as other Table* wrappers, it either forwards the calls to parent metrics and self (via LatencyMetric class) or just to parents. Also, coordinator*Latency metrics were added to keyspace metrics, this allows to aggregate coordinator* table metrics. Lastly, the table metrics are reloaded on table extension property change. * port #5205 global aggregates for tables are optional Global aggregates for table metrics may be disabled with -Dcassandra.table_metrics_export_globals = false. (cherry picked from commit a9fac9c482ff8bbf077e4b2cc6e918495dff51ec) (cherry picked from commit 01446a2c7f690019069884d8f7e7e7eed26c502a) commit de0cb3694dcbf6c7728b34e5035b810dc3c3718e Author: Massimiliano Tomassi Date: Tue Mar 14 16:38:08 2023 +0100 CNDB-6142 Don't directly use ClientMetrics in AbstractMessageHandler (#604) (cherry picked from commit 80ad8266f22a6eb9f1c704adfebd1a0d5570ff64) (cherry picked from commit 5bf0d3ab713d2873c2581f57914b77426a5627a1) commit 399680f16d5861a6002f53dfd3e52a6bfc2e95a3 Author: Jaroslaw Grabowski Date: Mon Mar 13 13:34:20 2023 +0100 CNDB-6523 (CASSANDRA-18311) fix BufferPool.memoryInUse counter (#602) The counter was incorrectly decremented by the size of the unused portion of the provided buffer. It is now decremented by the number of bytes actually returned to the pool (that may be different than "size"). The number is calculated as a difference between original and resulting buffer capacity. (cherry picked from commit cbea20addbfa41d3ccb448271ccfa9ac48aab4c7) (cherry picked from commit a7a74243a3a43c71f1b2e87fe8047ce5bbd30a0f) commit e12bd916bb39eaa99008c9049c8bba10ff87b6ea Author: ethan-brown2022 <107228208+ethan-brown2022@users.noreply.github.com> Date: Tue Mar 7 10:07:22 2023 -0800 Star 1597 (#598) Added periodic reports for compaction log with the following options: 'log_all': 'all' to enable periodic reports 'log_all': 'events_only' to only log on events 'log_all': 'none' to disable logging 'log_period_minutes': '5' to set interval for periodic reports to be 5 minutes (cherry picked from commit 584a3bfc16f007992082b172c810066f598de4a2) (cherry picked from commit 962bf99587f4c42766c0f9f105e5f5dfdc4b91cb) commit 2b7b6cd1634b913f823a4fd7e4ddd989941856f4 Author: Daniel Jatnieks Date: Fri Aug 25 19:03:51 2023 -0700 Use OSS handling of die in DefaultFSErrorHandler over changes made in STAR-13 and STAR-1502 commit 095b63469bed2750727ae5efd3b088c936436366 Author: Jaroslaw Grabowski Date: Mon Mar 6 15:57:19 2023 +0100 CNDB-5690 Extract global handler and make it configurable (#597) It is ok to move fn.accept at the top of inspectThrowable. Both existing fns may be invoked before the rest of the method. Other fns are JVMStabilityInspector::inspectDiskError JVMStabilityInspector::createDefaultCommitLogErrorHandler (cherry picked from commit 5713aaffa51a971bca32ccf413ab9d9ea1ea8313) (cherry picked from commit d8d81d6a6f8c1e6ba2ea00d3b7b8d39eb5f4ae67) commit fdf1932d3ecdf7b6a7a01fbb255a1d55de61c345 Author: Piotr Kołaczkowski Date: Fri Nov 4 17:23:36 2022 +0100 STAR-1779: Fix failing assertion and improve diagnostics The test tests for the number of received SYN messages in the shadow round. Unfortunately the original assertion required one SYN from each seed during the shadow round plus at least one more message from the non-shadow gossip round. The assertion would fail if executed before the first non-shadow gossip round. This commit fixes the assertion to check only for the shadow round messages. (cherry picked from commit b065fc4896cdbe9c1ed64ce5874840e8e5f8482d) (cherry picked from commit a44de691fd60908ebc0e02cd76264d89f8b43fdf) commit 273cc87d7d3c24d62af246b784413991c44c086e Author: Mike Adamson Date: Fri Feb 3 10:34:09 2023 +0000 STAR-1844 Fix legacy index CQL queries failing with NPE (#596) (cherry picked from commit 63d15d6de408cf373e8745c447ce697624adfdf9) (cherry picked from commit c3ef1fb3c798308fdebfc80f3a85fa1709afd493) commit a76587caa18cbd81739d0c7d52b1aa5ca16e4ee5 Author: Jaroslaw Grabowski Date: Thu Jan 26 08:28:21 2023 +0100 Fix compilation of MessagingService (#595) Fixes: [javac] cassandra/src/java/org/apache/cassandra/net/MessagingService.java:219: error: cannot find symbol [javac] static Map versionOrdinalMap = Arrays.stream(Version.values()).collect(Collectors.toMap(v -> v.value, v -> v.ordinal())); (cherry picked from commit 4904d4f2fd199f18e63c39590481fe4123c9c3d9) (cherry picked from commit 2303a0bd598ae86d333ea7d4b3c7e09da65b4d16) (+1 squashed commit) Squashed commits: [17aee9223e] CNDB-6235 messaging metrics impl is configurable (#594) * messaging metrics impl is configurable * internal latency gets `from` address Overriding cndb method will use it to record metrics per "from-dc", similarly to internodeLatencyRecorder (cherry picked from commit 142bb94d0336a700041bc7b11d98ab21473045bc) (cherry picked from commit 35fe294ce15c0c3c2e70cd090208f70b6ddb1c78) commit b2c8e4eb0faff9895a7380b16ac532319f6be345 Author: Jake Luciani Date: Tue Jan 10 11:21:32 2023 -0500 Add Missing version (cherry picked from commit e67dc9264bd07d06e0c2ade263e5313325f7034b) (cherry picked from commit 68058f02723c7975c76162c38e7aa6cf3f780e7c) commit 3df4bf53545d3bf166ce84e9e664db5162d5dc93 Author: Jake Luciani Date: Mon Dec 19 16:05:12 2022 -0500 Add Mutation Serialization Caching Patch by T Jake Luciani; Reviewed by Josh McKenzie for CASSANDRA-17998 Co-authored-by: T Jake Luciani (jake@apache.org) Co-authored-by: Mike Adamson (mikeatdot@gmail.com) Co-authored-by: Robert Stupp (snazy@apache.org) (cherry picked from commit 08b5857c790c95f538fc63dc17cf93cc7b86abab) (cherry picked from commit 12bafd76d9567a353471a2e7ab4297ff533eaa23) commit 59529a6e313e7d3b8b9261881e54f265916590c5 Author: Jake Luciani Date: Mon Jan 9 21:22:18 2023 -0500 Revert "STAR-1641 Cached mutation serialization (#536)" This reverts commit 9c133a2e3ad13a3077ecd67751025ed0fe24c6f4. (cherry picked from commit 5ba8c6ac71f909a40c39f3c942cf47bc26521f2e) (cherry picked from commit 1b55c2c831b9191f7c1ba627d1c16ccb06919e25) commit c09a5f6303cf894cda31f41592bc6e9ad30ca3e3 Author: Branimir Lambov Date: Thu Jan 5 13:21:23 2023 +0000 STAR-1826: Fix invalid reuse of metadata collector during flushing This was causing the cardinality information of an sstable to include more keys than it actually has, resulting in grossly overestimated key counts when picking a bloom filter size; the latter resulted in 4-5 times larger bloom filters on L1 than they should be. (cherry picked from commit b036f2d6210c5ba9f2e0d4651cee209b8e690dd7) (cherry picked from commit dade7d4227e4233107635aef63dae43bff20930b) commit 6454a4bf2400b2ef2c36044e6304e065071f4b0d Author: kijanowski Date: Wed Jan 4 07:37:32 2023 +0100 On a writer failure, wait for a third response to let speculative retries fulfill the read request (#589) * Always speculate, wait for 3rd response on failures * Review fixes * Add logging * Revert logging * Reformat code, add comments * Only add 1 node to the fail fast check if we actually have a chance to speculate * adjust test to not let it time out * Lower number of cycles to not time out * Lower number of cycles to not time out * Delete ReadFailureTest (cherry picked from commit 7d7bb9dfbf92b05155710f377e172e2fb390e4cb) (cherry picked from commit b121922050e3ca13955678c9eb0cfa1fec44d1fd) commit d603be9015c8ffed945b478c8f700abec26aa40d Author: Mike Adamson Date: Thu Dec 22 14:17:12 2022 +0000 STAR-840 Sharded TrieMemoryIndex to reduce write contention in SAI (#298) * STAR-840: Sharded TrieMemoryIndex to reduce write contention * Refactor IndexContext construction to single constructor * Add selective search of index shards based on key range * Test and comment PrimaryKeysMergeReducer * Fix ShardBoundaries.getShardsForRange to correctly identify ranges representing the whole ring * Fixed range intersection for partial bounds * Randomize MemtableIndex tests and filter in-memory equality queries (cherry picked from commit 2bdda4c4f07e23d8f1370c33f2d159e00ce27f6a) (cherry picked from commit 1572bd49496f41f8fb2dd228334acc623fe738d3) commit 650033160bfc278c0e523f86f8e573ba1c37d2f1 Author: Zhao Yang Date: Tue Dec 20 19:05:13 2022 +0800 CNDB-5791: Add Cross-region LWT support (#587) (cherry picked from commit 351ea78f66456f81181bc9d25c1314b203d1317e) (cherry picked from commit b83d44ebca18aa434d9bd7d63ae2ef7371e97313) commit ab947c34ca33c4bc5bfb56a3dda5555341b6ed02 Author: Zhao Yang Date: Fri Dec 16 21:22:22 2022 +0800 CNDB-5791: add EACH_QUORUM support (#580) * make fields in AbstractWriteResponseHandler accessible * Add Mutator#mutateCounterOnLeader for counter leader * Moved mutateAtomically() into the Mutator interface. (cherry picked from commit 12d9e96f29ef4d81b3b96f30b2457a21b96b4838) (cherry picked from commit 7caba0cd219a1e4162cff8b5d7a9be1efb1de7da) commit 0d2508080016584061be651b7447b11f7a442407 Author: Mike Adamson Date: Mon Nov 28 14:20:47 2022 +0000 STAR-1812 Stop collection guardrails spamming logs (#588) * STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc69b0abad4f25bb13e4c5c2da61d86751e5) (cherry picked from commit 8aa875bc69cde6807ac110f1d1781582c808e030) commit d9bd7576df8aeb329842c72c7bee7fdc051c3713 Author: Mike Adamson Date: Fri Nov 11 17:01:04 2022 +0000 STAR-1759 - Create new local and peers tables if load fails (#585) (cherry picked from commit d0e210914c2aaf23e2b5a870426787e1b72c378e) (cherry picked from commit e69df1d0c9574e120821dccc652967b0b2705ac1) commit 5fed7786ba969e0fc373080a69d31b6259d55e59 Author: Mike Adamson Date: Fri Nov 11 16:59:54 2022 +0000 STAR-1795 Fix JDK8 compilation error in CommitLogReplayer (#586) (cherry picked from commit c9f0cc4937b5048baea73ed485d6c629fafe1842) (cherry picked from commit ad17429e7153a74fff6b38521c6b008fc6138691) commit 64fecfc797f4ec0b9676f87b491a81d04fa57437 Author: dan jatnieks Date: Thu Nov 10 12:58:23 2022 -0800 STAR-1705 Port CNDB-4974 changes to add Micrometer BufferPoolMetrics (#562) Add POOL_ALLOCATIONS representing non-overflow buffer pool allocations; Use simple Meter to back hits and misses instead of a registered metric. Co-authored-by: Stefania Alborghetti (cherry picked from commit 2d5fdc54a33fbcd23522e6b71b3ba66a2c2ce8bf) (cherry picked from commit aeb8226d4ec713e4943ae8d2a9ee287b88bf849d) commit 7b41151bb0834c3e19dedfef301e5922e789e89c Author: Mike Adamson Date: Tue Nov 8 13:07:00 2022 +0000 STAR-1690 Stop message processing after MessagingService shutdown (#581) * STAR-1690 Stop message processing after MessagingService shutdown The RequestCallbacks class now stops adding callbacks after it is shutdown by the MessagingService when that is shutdown. * Make new log statements trace (cherry picked from commit b9da90a98e3896b48521319a2aa6121462c16b98) (cherry picked from commit 0ba3a39761c570a5e371898d43a461f01e5f5f7e) commit e4551d19e18a8176237a7819724a1e015cb74e9b Author: Piotr Kołaczkowski Date: Thu Nov 3 16:42:12 2022 +0100 STAR-1587: Fix incorrect assertion in random data generation proc The original assertion `mid < fakeUb` might fail if `fakeLb == fakeUb`, which could happen if `canonical` collection contained two or more consecutive integers. This situation is very unlikely, but not impossible. Fixes: java.lang.AssertionError at org.apache.cassandra.utils.LongBTreeTest.randomKeys(LongBTreeTest.java:681) at org.apache.cassandra.utils.LongBTreeTest$RandomTree.select(LongBTreeTest.java:440) at org.apache.cassandra.utils.LongBTreeTest.lambda$testRandomSelection$17(LongBTreeTest.java:359) (cherry picked from commit 882cea0b2abdcdfd6033840c18a8340952e17074) (cherry picked from commit a3c3c87c648d46ae04bdd8906fa63259d445b7c8) commit cffb85aa256d457765e31daa5d69299f7c6b26d9 Author: Jacek Lewandowski Date: Thu Nov 3 16:19:39 2022 +0100 STAR-1748: Fix RepairRequestTimeoutTest (cherry picked from commit 28021551fb215893351aa1658d5a9a4e8c2c17bb) (cherry picked from commit 2997ae5f9b007031b3996bcebef08a41f810f4c4) commit c4e04bd4f11a24d7ab83498cc0bce33215d4a7c4 Author: Jacek Lewandowski Date: Thu Nov 3 15:49:04 2022 +0100 STAR-1590: Fix Splitter (invalid number of splits produced) (cherry picked from commit d4bb82428b32a05be5d0725b8c2925fcaff090fc) (cherry picked from commit 2e7b99a97ce1024d5fc27df405c740bf5e7d4eda) STAR-1590 Fix SplitterTest conflict with CASSANDRA-18013 (cherry picked from commit 71bbfb6259b6aafeb322393b16cea9d22229c633) commit d11ab043756d30e320198c2341c587ec26cae3cf Author: Jakub Żytka Date: Thu Nov 3 13:13:53 2022 +0100 reduce number of iterations etc in LongBTreeTest (#554) 018c8e0d5e8bc55fc51d3361fcb27c3c1fd189f6 increased perThreadTrees by factor of 100 subsequent bd41794af4cb29204e19ec360b7eea4830a0dac6 decreased it by 10 in one of the tests here, we're decreasing it by 100 everywhere, to get down to the original number (and avoid timeouts in CI) (cherry picked from commit 289ebdb806c84aef7246f6f3602bfc3718c35d8a) (cherry picked from commit d816d10b01ab3e5947b6dfd73274986f12358143) commit 4f9f0f20e74edd516065b4e3c5e04d12728b9f2d Author: Jacek Lewandowski Date: Fri Oct 28 13:39:55 2022 +0200 STAR-1316: Increase timeout in ProxyHandlerConnectionsTest (cherry picked from commit c194c10afbaeb4513f6fe83e6f521123af25ded2) (cherry picked from commit ce81d6b2cf8a6b94fd2140034615c0c5ef64e0b7) commit f41478c3d8180277d12fe9157c94d64f59997e66 Author: Sergio Bossa Date: Fri Oct 28 11:17:04 2022 +0100 Port of CNDB-2744: add API to MutationInitiator to skip replayed mutations. (#574) Co-authored-by: Zhao Yang (cherry picked from commit d941d66d1b91778f43989f240da01add0ff8be27) (cherry picked from commit 2bf113f15552458a42aed4653c79dbe4c107e0c0) commit 135f9b99f0b23ec59202776d4ccd5fa7b452d280 Author: Jacek Lewandowski Date: Wed Oct 26 12:49:18 2022 +0200 STAR-1604: Add the "repetition" property ...which makes the build put the test results in the repetition index based directory and add a suffix to the test name (cherry picked from commit 74809bc0d13e06f8f5f884ce59c00f2ce4c59f7e) (cherry picked from commit 043ee6cd1c1f8b8fee1ce4586fa1d1c9334c18b7) commit 194fd32b8bb1916d63d2af4ca47b3eb929379818 Author: Jacek Lewandowski Date: Tue Oct 25 15:04:14 2022 +0200 STAR-1772: Fix BatchTest (cherry picked from commit 14d5229f5a4268e858ef1ee92de381c0bec53926) (cherry picked from commit 8a10f2cffc1478ddc78f7dbb2e7babef81f137c9) commit 71258a4ec1899c40de54615673d9a1e7edf97e0a Author: Jacek Lewandowski Date: Mon Oct 24 08:53:01 2022 +0200 STAR-1764: Fix SortedLocalRanges.isOutOfDate Replaced CompactionRealm.getKeyspaceReplicationFactor to .getKeyspaceReplicationStrategy (cherry picked from commit efc3dd6935f28113dacc750992cb108cbf039bf9) (cherry picked from commit 44153a3607777165d3e7746e62b280d4e78dbfd2) commit badce52d8c5d52bd1a9cfa73fbef015c0b2e02a5 Author: Piotr Kołaczkowski Date: Wed Oct 19 18:47:37 2022 +0200 STAR-1739: Add a way to check the number of cached file chunks (#570) Needed by CNDB tests. (cherry picked from commit a3322ce3e4f723479f4f6a37dc411a6d37acc0a5) (cherry picked from commit 60151be19950b27188aa76b2563177691d96fe0a) commit f18a78e43341a62e5ab4bf24fb2c0ec2fb1bd4e0 Author: Jacek Lewandowski Date: Tue Oct 18 15:37:48 2022 +0200 STAR-1745: Upgrade byteman and jacoco (cherry picked from commit 3a41bb17cfd5ffe3df8d8013676f4d5828af753c) (cherry picked from commit 21d7e7938c9fe6c1f82ababa357c8456dc902728) commit 832ea820956dac7f93740b59377ec562738cb79a Author: Mike Adamson Date: Wed Oct 19 11:05:28 2022 +0100 STAR-1586 Fix SelectiveIntersectionTest (#564) The setup phase of this test was not waiting for indexes to be queryable before inserting data and flushing. This was resulting in a race condition during the flush leading to a random occurence of higher posting reader usage than expected. (cherry picked from commit 10c15efd79023253eedd688fbd6eaadb40992294) (cherry picked from commit 9c49ba28c24b30475e8757b94d4b50aab4b71a99) commit 23295b3c2e6319987c8456c5bfbc07a6dd431b7b Author: Ruslan Fomkin Date: Wed Oct 19 08:55:34 2022 +0200 STAR-1626 Enable BatchTest and UUIDTests to run fully (#563) Enable disabled test in BatchTest, which was broken due to previous name violation in BatchTests. It is fixed by setting batch size guardrail to the same number as in Apache Cassandra. Updated default batch size number in the comment in accordance to actual values. Fix name violation in UUIDTests to UUIDTest, so the tests are executed. (cherry picked from commit 4d537da98009150f46d399d8c60ea0fafda7f039) (cherry picked from commit 1da65460b7f20f4fa6328a7686660441863616ee) commit 0a6eeaee05e34d730ba23d69de720cb2501e9418 Author: Matt Fleming Date: Tue Oct 18 23:32:54 2022 +0100 STAR-1746: Don't throw an exception if we fail to unload a CF (#568) This is a partial port of CNDB-3190 and is required to make the CNDB unit test, TableDataDeletionTest.testDropTableAllDMsUploadFailed, work. The main idea behind CNDB-3190 is that we don't want to allow any exceptions that occur when unloading a CF to interfere with tenant unassignment or make the schema inconsistent. (cherry picked from commit c4845fcfa151634721f359de6522b509622de39c) (cherry picked from commit 67dc99891187e86b5eff6831a33105b89bc9cccc) commit 41dae845a7f9c207664b56767cd561210cbfcbfd Author: Matt Fleming Date: Tue Oct 18 18:50:15 2022 +0100 STAR-1740: Convert RemotePath to local file when taking snapshot (#566) We can't mix RemotePath and UnixPath files since it causes this error: Caused by: java.nio.file.ProviderMismatchException at java.base/sun.nio.fs.UnixPath.toUnixPath(UnixPath.java:198) at java.base/sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:469) at java.base/java.nio.file.Files.createLink(Files.java:1102) at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:188) (cherry picked from commit e9a1980ef47946cc9f3eefa36bb8fcaf0e57328b) (cherry picked from commit e213fe716f672caeb1c82a8155a5814cd469b4bf) commit 56912a95545a7da3cde81513b95598c7024a28c1 Author: Jacek Lewandowski Date: Tue Oct 11 13:54:49 2022 +0200 STAR-1693: Post-update fixes (cherry picked from commit dfae73c6c3557de07f9af852737ead1a830cdf00) (cherry picked from commit 32d499dd32d03f18942f4935e0eff7c94ed1515e) commit eed739a5383040ce141818273e28edc569c400c3 Author: Jacek Lewandowski Date: Fri Sep 16 10:24:39 2022 +0200 STAR-1639: Use repo.datastax.com/dse as alternative repository (cherry picked from commit 01ab4cca025102eac84aa45b92a83e79bc2fa0ea) (cherry picked from commit 3c68b149bc102a4aad6ebb05375ace36f3709237) commit 2283c1f18780d2e3d858f8c3ed6dfcec0b63e3f6 Author: Jacek Lewandowski Date: Thu Oct 6 08:35:05 2022 +0200 STAR-1660: Fix and prevent direct memory leaks in dtests There are a couple of things here: - `unsafeFree` method in `BufferPool` did not do what it was probably expected to do - that is, the direct buffer was not released properly because when it is allocated in `allocateDirectAligned` the method actually returns a slice of the original buffer, while the only reference to the original buffer is in `attachment` field of the returned buffer. It was mitigated by using a new method to clean, which can release the parent buffer by recursively go through the attachment hierarchy - for in-jvm dtests, releasing of all buffers in buffer pools was added as the very last step of instance shutdown; it fixes memory leaking between the subsequent instance restarts; in production run, we just stop the JVM and the buffers are lost, but in case of those dtest we need to deal with that explicitly (cherry picked from commit 7684e279c48ea7ae6ca14eabfe0d775b77d82aa2) (cherry picked from commit 27bd07a816a1df876e8f8b85fd7451e026e37ff1) commit 710efb1c9ed22c131c5040875ba38291b45576dc Author: Zhao Yang Date: Wed Jul 13 13:32:55 2022 +0800 STAR-1706 Connect back using preferred address Port CNDB-4855 Fixed streaming to connect back using peer preferred address instead of Channel#remoteAddress (cherry picked from commit 308fb9ddcf856836f721ea5b08fa456a0c0c4fd5) (cherry picked from commit ca0c88180c79cb9b571b39e38cd62079f7ad4d40) commit b22858092cf8c089979ded35fae05ad4a8a6a404 Author: Jakub Żytka Date: Fri Oct 14 10:58:51 2022 +0200 CNDB-4898: add ReloadReason.REGION_CHANGED for reloading sstables when region comes online (#551) (cherry picked from commit 1df0020fc8507f61b2f2ba5a3b5324ab28ece80d) (cherry picked from commit 06b1c55cf4c488aec624db19fca933b94b587fc5) commit b5b4d18c59612adbedee3f4047358a5058eada09 Author: Matt Fleming Date: Thu Oct 13 17:05:29 2022 +0100 STAR-1701: Restore SSTables if they cannot be dropped and fix lifecycle txn bug Port BDP part of CNDB-4035: restore sstables if they cannot be dropped and fix lifecycle txn bug so that we SSTables are added back to the live set if we fail to drop them. (cherry picked from commit edfd2fbf15a601534d1f562ad4a03f2faed36891) (cherry picked from commit 2bd1ffd66257418f6f3a656236f668d9dd7f441d) commit 93e198dc0b23754722f360725070f327b7153064 Author: dan jatnieks Date: Thu Oct 13 08:14:49 2022 -0700 STAR-1699 Port CNDB-3654: Streaming cdc-enabled SSTables will fail. Warn about it (#558) Co-authored-by: Matt Fleming (cherry picked from commit e20b1de6d4823f07c327d4db132d6996b57914b8) (cherry picked from commit e383018abbcae926322156518015f45abbb3c23a) commit 52b00b3b1b9414ff17f4b138fb8d18ede4194fd7 Author: dan jatnieks Date: Thu Oct 13 08:07:06 2022 -0700 STAR-1702 Port CNDB-4894 - bypass txn.isOffline() in scrubber (#559) Co-authored-by: Stefania Alborghetti (cherry picked from commit c1f3a296ed4465b2b418e93c8f39b1b33b2f188f) (cherry picked from commit 00c3da859a38f344dc687d4ac2a2660256b7c857) commit 5d69a21a3de84617b12b6c813b25f2135c9cb5e9 Author: Ruslan Fomkin Date: Thu Oct 13 15:09:52 2022 +0200 STAR-1704 Fix DroppedColumn to use CQL string in column name (#560) Port of CNDB-5113 Fixed DroppedColumn#toCQLString by using the CQL String version of the column name, which also double quotes the name if it's in mixed case. Co-authored-by: Massimiliano Tomassi (cherry picked from commit 258fe6fd1eaabf4e272e4ef8652162f43cc90ee4) (cherry picked from commit 67a2262e5514867897f54f6abf27a44d028b529b) commit a8aa4836bbe9512f2c8e2a60eed6171b3d0eeb2a Author: Stefania Alborghetti Date: Thu Jun 9 12:39:30 2022 -0400 STAR-1694 Expose buckets in UCS and accept sstables for direct classification This commit changes the API of UCS as follows: - The Bucket inner class is now public - The method for extracting shards with buckets is now public, and it accepts a custom list of sstables These changes are required for CNDB, so that we can classify all live sstables and visualize their corresponding shards and buckets in a diagnostic tool such as Autobot. The comments for warnIfSizeAbove have been clarified and moved to the method Javadoc. Port of CNDB-4385 (cherry picked from commit c58b47d916b4152afd92a2d27eaf3e68b3464d42) (cherry picked from commit 3e5ef3f046a87471396981c234b6c0db3dace0c2) commit 9120e36badd4217c4fa5cca5923f3c68d74050d6 Author: Stefania Alborghetti Date: Tue Jun 7 17:13:39 2022 -0400 STAR-1694 Configure max compaction overhead This patch adds a way to customize the compaction overhead, i.e. the transient amount of space required by a compaction whilst both input and output sstables are present. In BDP this is just estimated to be the size of the input sstables. It's unclear if we can improve this in CNDB, but I kept the refactoring because initially I got confused thinking that in CNDB we could just waive this requirement since the input sstables are in the file cache. So I think it's good to spell out why we use the input sstable sizes by encapsulating the calculation in a method with javadoc. The patch also adds a warning to the logs: if a compaction cannot be performed because the space overhead is larger than the space available, then the logs now contain this information. Without this, troubleshooing why compaction tasks are skipped is quite hard. This warning was already present in BDP but was missing for CNDB. Port CNDB-4385 (cherry picked from commit 8ab993db6d21f4db08d2d8978139a40cefce3a35) (cherry picked from commit 818fee7af71588dd4d6e2ce58aed54f39690d9ae) commit 2961723d52548d6e98a2577d6a27c8b9952e449e Author: Matt Fleming Date: Wed Oct 12 17:03:26 2022 +0100 STAR-1697: Port keyspace renaming (KeyspaceMetadata::rename) from DB-… (#555) * STAR-1697: Port keyspace renaming (KeyspaceMetadata::rename) from DB-3896 Required by CNDB-3170 and CNDB-4909. (cherry picked from commit f74c1a6a5424c3018f28d9d7f6bb06ee2331d550) (cherry picked from commit 6fb5012781910e965c21ffebae4e8d8fcd475ba2) commit 1d28ad00b2d5e8a5efac998a46c3d041d5165e92 Author: dan jatnieks Date: Mon Oct 10 00:43:35 2022 -0700 STAR-1683 Port remaining pieces of CNDB-4308 (#549) Co-authored-by: Stefania Alborghetti (cherry picked from commit 3039232d07521c0b53879e9603ff63ebbece24a3) (cherry picked from commit 9624082a5680ab08900999ee23c38e691bccfff0) (cherry picked from commit e574fc3f12ecc17f99d5cb6b1caf502c19001202) commit 395bb93ce93b1009e21e388c83f267df5ca09cda Author: dan jatnieks Date: Tue Oct 4 00:16:02 2022 -0700 STAR-1680 Use interface PartitionPartition return type in getFirst and getLast (#546) (cherry picked from commit d5983575fc5c26b2cdc996b5d10180c8a3d3cff8) (cherry picked from commit ea71a01a99d5a158924812a232abdee9b09d6f11) (cherry picked from commit 8e7dc74dc2fb6ddc6c9278a062544e6ac52cc443) commit 9feebfd841e868c2eb78c21c0abc6dd86d1c9770 Author: Jeremiah D Jordan Date: Thu Sep 22 12:03:01 2022 -0500 STAR-1659 Update to latest snakeyaml. (cherry picked from commit aa6b96d0e4dbb123db9fa5bb8811d2823d32ff1b) (cherry picked from commit 3af02c6985ecddb4fe77c5fae357f41eb28017d7) (cherry picked from commit e72575c1bc16c1e7c1b8b23820ef4ef6d4e122f9) commit 7baee9e4ed4daabb132cc6e18a4a4a22a9f07ee7 Author: Ruslan Fomkin Date: Mon Sep 26 17:55:38 2022 +0200 STAR-1509 Make SSTableReader.open visible for tests in CNDB (#545) (cherry picked from commit efc13e23ae46f589c30b13de69ad7ce6e662a990) (cherry picked from commit 73cacda8e9a25a4a9a1cc8b482b174efed69ff86) (+4 squashed commits) Squashed commits: [87ffea017f3] STAR-1509 Add API to expose protected fields for tests in CNDB (#544) (cherry picked from commit 523b62a68d1bdbfbbc7e03ee091a3b8db8dc736e) (cherry picked from commit 2d26e5895be866e1ee65ba456dac273aa5dd311c) [4b1db304f2d] STAR-1509 Add tombstone counter to table metrics (#543) (cherry picked from commit 32ac6cd2471f49f060716903198dd1ac53939b0c) (cherry picked from commit e66ae789b8d4303d2c2ab441da7ca42f1c1f9384) [8a69565d6a1] STAR-1509 Add getFile to CompactionSSTable interface (#541) (cherry picked from commit efdcd173c5406f8dc4febea59dc64c80afbb3558) (cherry picked from commit 216272c177ccb8a455bef1a4a274048fab1864a7) [39d085c0cad] STAR-1509 Make some UCS Shard related API public (#542) (cherry picked from commit 3d816c130f97f2bd67f804a614970a66410bee57) (cherry picked from commit 643f70e5d75f4718b3cca64595702cd3a6b3744f) (cherry picked from commit d6b26c723b424372607734282315067870e7646b) commit dd3cf0d4e06b8aebf03ec88466aea10443a5d9c7 Author: dan jatnieks Date: Mon Sep 26 06:40:44 2022 -0700 STAR-1478 Add Types parameter to DroppedColumn.prepare to avoid keyspace lookup (#540) (cherry picked from commit 96c608fd13e99e6afa29820cd877c36391406986) (cherry picked from commit ecee8149a21f9206242011e6017eee55e4adf1ff) (cherry picked from commit f49c2c4b24b6af52c6f18ff39df33ab3ca672487) commit e54b8adbfb3059a5d9dd8a351770d08ed61ead92 Author: Ruslan Fomkin Date: Mon Sep 26 11:01:58 2022 +0200 Switch artifactory to cloud (#534) Actually it is not used in CC, since Jenkins resolves publishing the artifact in own way. (cherry picked from commit 5eb9648b1fb95c0b3dafde11b2e6470b266382a0) (cherry picked from commit 4c140a8a6afa301318204b8257a0ce46f072700d) (cherry picked from commit 9b77b44d2dd684fce3f6477c323720bebc7be43d) commit 5cdb9a349d4a1f66642565c0007ad835c7fb4464 Author: Mike Adamson Date: Fri Sep 23 15:51:57 2022 +0100 STAR-1641 Cached mutation serialization (#536) * STAR-1641 Cached mutation serialization This introduces a per-mutation cache of the serialized representation of the mutation. * Fix broken test (cherry picked from commit aa0190be12bb7efc4644508cd28b39f7876d2244) (cherry picked from commit 9c133a2e3ad13a3077ecd67751025ed0fe24c6f4) (cherry picked from commit 58dc33043f15ecaf8b24fa6cf536f0a5d6d42e29) commit 65130306fe008a1ba14559aba827c4071621d4a6 Author: Matt Fleming Date: Thu Sep 22 16:20:37 2022 +0100 STAR-1492: Print message when repair syncing starts (#538) This string is expected by CNDB's RegionRepairTest.testRepairWithTruncate() and adding it fixes that test for Converged Cassandra. (cherry picked from commit 0f55457358ab0647cf1731986e0fbc83c19db9e4) (cherry picked from commit 041a3a847973397b3c1cf77486fd26020f840cc3) (+1 squashed commit) Squashed commits: [4f5477634d8] STAR-1492: Track compaction ops during validation (#537) (cherry picked from commit b23358e3edad435d209b5e7c167b050f7b628a70) (cherry picked from commit c54eb73cc90e4b881409815f7ad1f850a8aa36d0) (cherry picked from commit 8941f3a38a53d32fc0e1f597d56463ca786436fc) commit de108feeff2dfef399d6dc39a35dd9964b57c9ea Author: Zhao Yang Date: Wed Sep 21 15:31:04 2022 +0800 CNDB-5555: add accessor for HintMessage#unknownTableID (#535) (cherry picked from commit bf64792f34e393681ee7b0a0edebc5a807ac7acc) (cherry picked from commit 0ca9f1af550460da216bc58502fe405111dd9670) (cherry picked from commit 432eb4e58bb8a2d8848775bfa10b30ae15bff829) commit c33891a002e9cbcaee766210dc44b6ec681a1a1f Author: dan jatnieks Date: Tue Sep 20 07:24:10 2022 -0700 STAR-1491 Use StorageProvider.getLocalPath in PathUtils.rename to ensure local paths when renaming files (#527) (cherry picked from commit 4d8d0b143b54d357223300dab581959c75315aa7) (cherry picked from commit 8779fc26e7c90804b2cc43f1f697d2c078e70e0c) (cherry picked from commit fa7f31d66faaa362fb34bc23458781fe66518a90) commit e7a2b1253404be1ef3be056d2f818505ad4547ca Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Mon Sep 19 12:15:11 2022 +0200 STAR-1443: Fix VerifyTest (#533) Basically there were two tests which verify the same thing, but one was intended to use BTI and the other one BIG format. Since by default we run with BTI, the BIG test explicitly sets the property to use BIG format. However, we run the full tests suite for both BTI and BIG, so when we run with BIG by default, the BTI fails (because it has no explicit configuration and assumes we run with BTI as default). The fix is simple - just have a single test, which just uses the default sstable format, it will use BTI in one test run and BIG in the other. Those squashed tests didn't differ by anything but assertion on the created sstable format. (cherry picked from commit 9c008dfb5818608c7118b743a918c89127e7e0c6) (cherry picked from commit a7520a6c6c7cd1e5a5ec06934c8279a539e7a676) (cherry picked from commit 067432338f5350472f5ff78d708214eff020ae27) commit 6154f930da3954e2e2768ee2e9ed8fa707a7c61e Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Mon Sep 19 11:43:50 2022 +0200 STAR-1195: Fix IPMembershipTest (#532) Added metadata directory to the list of data directories in in-jvm dtests (cherry picked from commit 0d9625e5d110f6f47a818baad52c38549547beb7) (cherry picked from commit 77c01333f08755cd8567543a7cba9a130ca507a3) (cherry picked from commit 4cb9fb88dc0bbcb36b54d4e7aa584a722b27dec5) commit f6f92c50678092f070b4d0ec714a5821b8371cc4 Author: Jacek Lewandowski Date: Sat Sep 17 14:36:42 2022 +0200 STAR-1191: Fix NativeTransportEncryptionOptionsTest TLSv1.1 is now disabled by default in Java (cherry picked from commit 7f00f54d7d1894984e78f0484d61598f6ca61579) (cherry picked from commit d0d5ea3f8c78d79da9ad4c6b64a3a8de60d76fc8) (cherry picked from commit 56310209ca6409b2858753a812ece5e0dac0874a) commit acd25aa904f9f8ddfae8dcd966a8316cf4df6b2a Author: Jacek Lewandowski Date: Sat Sep 17 10:51:11 2022 +0200 STAR-1646: Fix SSTableIdGenerationTest (cherry picked from commit d1564ba5647e2ebbc4a62cfa82d90e7e28300138) (cherry picked from commit 8ca0458db85630ec25d9c272aadc8bb03fb1b25b) (cherry picked from commit 87135171e0c4ff77fed10d4c3ab0ddcfa55b4a79) commit 0d0cadb983ce6625645f378783009c31dfe655a8 Author: Jacek Lewandowski Date: Sat Sep 17 11:17:32 2022 +0200 STAR-1585: Fix MultipleSSTableFormatsTest (cherry picked from commit 3a2044c67ac49ebd0423af08b8e37a308a4f3cfb) (cherry picked from commit 4c7aac30dac3d1646b0289864ca5b369d9279e1b) (cherry picked from commit 51df0518137f19ab37145280ca78f912c0fbddca) commit 8a020d85dd3ad36bbfcdb735d4c67aead2fb6108 Author: dan jatnieks Date: Wed Sep 14 05:54:43 2022 -0700 STAR-1491 Fix name of REPAIR_PROGRESS_REPORTER property to be custom_repair_progress_reporter_class (#526) (cherry picked from commit ec16bdb653066e575d2b2fd507e49d21740735c8) (cherry picked from commit 7972b3fa35ec9017eb97ff14ad1ee4d15b802491) (cherry picked from commit ba58d5d6a0eadfb943082c438c56e8d9d08156ce) commit 5f93409766ff2402e49ab3fcefb649d69897ec23 Author: AditisJain <102972224+AditisJain@users.noreply.github.com> Date: Mon Sep 12 08:09:34 2022 +0100 STAR-1577 Remove redundant calls to isInternalKeyspace() for a better compaction performance (#521) Co-authored-by: Aditi (cherry picked from commit 0ef2f02b9bf55eaa8882ef216ffe683d079104f9) (cherry picked from commit 6cc63be641daf7e73f9e0fc8394ab6cfd1c3f5ca) (cherry picked from commit adb13962591be2aa13c777516a335fb1bc1fd33e) commit 5201992db8841d699627f592c440842fdb966046 Author: Sergio Bossa Date: Fri Sep 9 15:02:50 2022 +0100 STAR-1489 (#524) * STAR-1489 make verb decorator efficient * STAR-1489 Add Mutator callbacks invoked when locally applying mutations. * STAR-1489 Allow to override attributes in create/alter keyspace. * STAR-1489 Added tests. Co-authored-by: Jake Luciani (cherry picked from commit 8fc3807cbd0ab0de2d662d9ec9d40f71ad01dc0d) (cherry picked from commit 0f8abbcf7cde4fccdd17f5377a532b778494bc58) (cherry picked from commit 7211eb627a202e8a30179bc1b75cec52eb2d0431) commit 55ddf02d687940e9af983294f329f94d13bf2d03 Author: Ruslan Fomkin Date: Thu Sep 8 15:20:33 2022 +0200 Add method to update consistency level in query options (#525) Adds public API to update consistency level in query options. Adds a test and fixes name of BatchTest, so it is run in CI. Co-authored-by: sergio (cherry picked from commit c71af493f83ae49ae46ed5edec7c98348f619df2) (cherry picked from commit 95e69bd91664c0d42ab611e69fb991b809b4d648) (cherry picked from commit c0eb9ef68bccb94febd9f13a0a9c7f735008f3d0) commit b641290259588c852422ece497ac16c64c8e6413 Author: dan jatnieks Date: Wed Sep 7 15:00:21 2022 -0700 STAR-1478 Port SYNC_LAG_FACTOR to make the CL sync block delay factor configurable (#522) Co-authored-by: Jake Luciani (cherry picked from commit 0861d30f5fa44d5efd9b6aabea76c00e8c51f75c) (cherry picked from commit b8962646c3f1af6e4894652f6d7f0933f97d504a) (cherry picked from commit de19f5b0b1e1906de204a6948d689d7bd3a3931b) commit 5a1ae75190a1e6350d97b2e81cce9a19c796c175 Author: dan jatnieks Date: Fri Sep 2 12:39:01 2022 -0700 STAR-1611 Port CNDB-2770 to add toggle to disable compaction history (#520) Co-authored-by: Jaroslaw Grabowski (cherry picked from commit f4cff37a2955944d65f7b8b9d9665c105dacefe3) (cherry picked from commit d8cdd1d629f9cd3f7b3fe5a47cd2519329474aee) (cherry picked from commit 36893846f55de8002f4ad409c98ddd668681616d) commit 91b84994ed39d4a515cd18d162661a7b218f40d6 Author: Jake Luciani Date: Fri Sep 2 09:35:57 2022 -0400 STAR-1489 make verb decorator efficient (#519) (cherry picked from commit b2ab847906678d293afedcd90672badf7cf54ff9) (cherry picked from commit 5f8addb301da1645e66a8e8942935d5c59d0ffba) (cherry picked from commit 87c08368fa4e98b032afeb8883d4d6a582979bb4) commit 8ab99866f2039506e41a0b16a75f7283071b42dc Author: Jacek Lewandowski Date: Wed Aug 24 21:34:26 2022 +0200 STAR-1502: Fix a bug in scrubber There was an uncovered case when there is just one item in the index and the data file is completely broken. The scrubber falls into infinite loop trying to move to the next partition using index. But since the index is exhausted, it cannot make any progress and also it was not marking that moving forward makes no sense. (cherry picked from commit cf81e17e067035362bd978d56c1db0919dd46d11) (cherry picked from commit c77d4f0c47a4192f2c55452c41807bade9e57271) (+1 squashed commit) Squashed commits: [b4c9f68a041] STAR-1502: Fix JVMStabilityInspector behavior on CorruptSSTableException The problem here was that JVMStabilityInspector killed JVM in face of CorruptSSTableException when the policy was set to die, regardless what handlers were set. (cherry picked from commit 90d5ba715a003acb3b4e04c4438398cf45b332c8) (cherry picked from commit 04c9a9f21b87aa7e18066c35b364f515094d4f58) (cherry picked from commit 50b565925eaa3b6735a6cd9a5dd103b665052785) STAR-1502 fix conflict with CASSANDRA-18294 commit aebd9db94908065bcd167a49059dd5b497d0559b Author: Matt Fleming Date: Thu Aug 25 13:41:31 2022 +0100 STAR-1477: Port the rest of cndb-1116 to fix billing during tracing (#515) * STAR-1477: Port the rest of cndb-1116 to fix billing during tracing We need to set the tracing keyspace per-statement since this info needs to be passed from the coordinator to the writers in CNDB. Without this change, the writer knows that a query is being traced but not the keyspace/tenant and instead it'll bill the shared tenant. (cherry picked from commit 79364d452ef0f62e438b43ca742582c6f5b8d112) (cherry picked from commit d4eaccef6707d7f846348d6db1e88dddd5e509bf) (cherry picked from commit 71c2cbea00be1b7259766ffa5e2ed512898fe344) commit 73b2b4c3fe573e64e663e2bd253c913dde1f86ba Author: Matt Fleming Date: Thu Aug 25 00:25:26 2022 +0100 STAR-1482 Log message fixes for SingleCompactorTest (#513) * STAR-1482: Partial port of CNDB-946 Port debug logs from DSE that were added in CNDB-946 to help diagnose slowness when dropping a table. These are needed because the SingleCompactorTest CNDB integration test expects to find some of these log messages in the log files. I skipped the log messages for the schema code since it's very different between C* and DSE and because different log messages already exist. Plus, some log statements have already been added as part of other CC porting work. * STAR-1482: Sync compaction controller strings with DSE CNDB's SingleCompactorTest integration test expects specific strings to be in the logs. Make the CC logs match DSE so that tests pass but also because they provide more info for users anyway. (cherry picked from commit 8b3f11684629a33d597b35537d56dbdff8f24cbd) (cherry picked from commit fb1a3ca65ed49ee4eef580e52a2cb4308dae5598) (cherry picked from commit 5850f62f75505ea046833db321c029dc62db290f) commit 59730ae6e0741ab9967aa4f57203f6d579e4e018 Author: Jake Luciani Date: Wed Aug 24 10:05:59 2022 -0400 Add USE_RANDOM_ALLOCATION_IF_NOT_SUPPORTED to TokenAllocator (#516) (cherry picked from commit cc82c955bcf1f2b7f60be98d71d5360d03251739) (cherry picked from commit 383cbe31b6bf0035335aeb9f95822e9a9bfeca09) (cherry picked from commit 07d634371d9dac946fbf7e0fe02e5cea561799fd) commit e049131d56ed8d99083b5398dd746d5ecd410ef0 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Tue Aug 23 21:10:20 2022 +0200 STAR-1602: Remove redundant waiting for read/write barrier when dropping a table (#514) (cherry picked from commit e70219627d692f186cf6bb6c77f6934a04c8c973) (cherry picked from commit ce5173cb2c3562ef5ee691b1d3b7b1d5aa7698bd) (cherry picked from commit 605fe06ab2934309f43261f617f774f09b62a019) commit 56b9f678bf83765a360b68db02f9924fce004a2c Author: Jake Luciani Date: Mon Aug 22 09:57:39 2022 -0400 STAR-1503 Handle edge case of sstable ref leak after prepare before commit (#512) (cherry picked from commit bb30bdd9647669706eeb50a4289df3207bc32370) (cherry picked from commit 11914d479348da6bdba9c8f6da47195374f4f3a1) (cherry picked from commit 2c16b893abbcbee7b364fc5038daf9362a1a2dff) commit ec29d3d81d4df94f9cd2ddfc3854622d87dd70dc Author: Ruslan Fomkin Date: Mon Aug 22 12:51:08 2022 +0200 Replace custom make with FBUtilities.construct (#511) Ports from DSE brought implementations of custom make methods to construct custom instances. There is already an utility method for this: FBUtilities.construct. This commit does number of unifications and fixes in classes, which were affected by replacement of make with FBUtilities.construct: - Replaces custom make methods with FBUtilities.construct to construct instances of custom classes provided through system properties. This removes make implementations and even corresponding classes. - Uses method isPresent for checking if custom property set. - Improves custom property tests for instanciating custom classes. - Removes tests of custom makes. - Removes usage of JUnit version 3 by removing usage of TestCase. - Setting and resetting custom properties is unified into static BeforeClass and AfterClass, since custom properties are read only during the first class usage. (cherry picked from commit 4336a7294776c7496d5444fa0f38d57a4f717d92) (cherry picked from commit 20bc84d055d372ad98407a63b8556a8eb2a96736) (cherry picked from commit 164dadce2427b258cd7d467db30f0e716a64e1ba) commit 4007c545fd1d6548e4ef632cfb6767afd3173768 Author: Jake Luciani Date: Wed Aug 17 15:53:10 2022 -0400 STAR-1493 replayId should be defined per segment manager path (#505) * STAR-1493 replayId should be defined per segment manager path (cherry picked from commit e0f1d91d468e67af01b4f80afe5ed8f77d1039e8) (cherry picked from commit bb004954a7f7b22263363f4bf699c39fb3a4e0c8) (cherry picked from commit bb6d9422ed5d9bfb710efd43135514f8ef587c2f) commit 18a6c72263d44c0b7f4942714bfdedc52a47bb14 Author: Ruslan Fomkin Date: Wed Aug 17 16:42:50 2022 +0200 STAR-1525 Support for hinted handoff (#486) Provides support for hinted handoff for multi-region support: - Adds property for custom implementations of hint verb handler. - Adds hints endpoint provider interface and implements default provider. Allows to provide custom implementation of hints endpoint provider through relevant property. Relevant calls are replaced in hits implementation to use the provider. - Adds property for max hint file size, track data size in file descriptor and prevents hinting if exceeded - Adds hint timeout property and uses it in verb handler - Adds property and skips rewriting left hints if set - Makes needed API public and provide getters - Fixes test coverage (cherry picked from commit 60bdd24518e94efadf68fce43d92cf836edc7597) (cherry picked from commit 52a6c0c9b57facc47954d48cab911e7793c61ef2) (cherry picked from commit ce4275436c78d0fe27b33f6fc146ec5347f7f9a6) commit 2057359a8d45a73256dddf989b6587fdd99505fd Author: Jaroslaw Grabowski Date: Wed Aug 17 16:27:02 2022 +0200 STAR-1598 port CNDB-3932 (#509) If a user creates a prepared statement for a table and then enables the CDC param, the prepared statement must be invalidated. If we don't invalidate the prepared statement it'll be executed using the old TableMetadata object and take the non-CDC write path and go directly to CNDB writers instead of via Pulsar topics. (cherry picked from commit 1bed2e7ff4588a5774fd0a46794209fd14997590) (cherry picked from commit 68da6b20721d24a5449dfc72caebdb1d64d7920d) (cherry picked from commit 68f5f62971581f3868b8b877f7b25cbd71ccccef) commit 264ab5d2619df726154e2ff5132aa79dcf0ee8f9 Author: Matt Fleming Date: Wed Aug 17 15:05:53 2022 +0100 STAR-1480: Port UCS log config option from CNDB-2264 (#510) This option is used by the CNDB compaction integration tests. (cherry picked from commit 65674a6864361fc6ff02efaa24eedee02904d84e) (cherry picked from commit a21340df3aa91d437eadfc4f8461dc6e7d3bd81f) (cherry picked from commit 27970c8deddc5ea78779f8af0ed2981714585e04) commit 0647b76a7af4b1e1a5a6c9270ff2d60003ff22c9 Author: Jaroslaw Grabowski Date: Wed Aug 17 13:46:10 2022 +0200 STAR-1600 DSE changes around schema changes for offline services (#508) * port CNDB-1250 Flushing causes exceptions for offline services. Skip flushing for these services. * port CNDB-2495 (DSE changes only) (cherry picked from commit 135597b58999920d81972ce701700b8b7ea3280f) (cherry picked from commit ad2de6b32341f7bbe93b26c8e6558babf2bd4d11) (cherry picked from commit f965eb1120a11eeea3e1ea95fb321d378974947f) commit 13f2576e74086bfd9e9040c1be94dfd1aab87642 Author: elliott.russo Date: Fri Jul 29 16:04:44 2022 +0100 STAR-1579: Add scaling parameter in the compaction log analysis report Addition to the compaction_report.html file which shows the W value changing over time graphically. In the drop down menu, select W value to see how W changes over your test (more useful for adaptive tests than static scaling parameter tests). Minor change to the graph.html file which automatically scales the negative y-axis of the compaction report The y-axis in the compaction report was fixed to a minimum value of 0. The addition of graphically representing the W value changing over time means having to scale the y-axis manually within compaction_report.html as the W parameter becomes negative in the read only stage. The graph.html change automatically scales the y-axis only if there is a negative data point (meaning it only really effects the W value plot). Final touches to CompactionLogAnalyzer and graph.html (cherry picked from commit a0d37a5444bcb9215bd842655ab60932fd5935c1) (cherry picked from commit 2f632778e97765ea98cfcdf2253eb765f1e1e4ba) (cherry picked from commit d503cb7d32ba8b6cf0c49b454ad9ac025dc51c5c) commit 22eefdcdfc3d521c28d1149949b2b1e6fb08d9e9 Author: Jaroslaw Grabowski Date: Tue Aug 16 10:17:28 2022 +0200 STAR-1341 abstract mutations used by CDC (#458) * abstract mutations used by CDC At first, I tried to refactor StorageProxy methods and WritePerformer interface, but since every now and then we want to rebase on OS C*, I reverted all the changes. Instead, a new abstraction level over various mutate methods is introduced - Mutator. The default implementation proxies the calls to the original (default) methods. The users may provide a custom class Mutator implementation to alter the mutation behavior. This solution was chosen to minimize the rebase conflicts, it's a compromise between code structuring and maintenance cost. In the ideal world we could refactor the mutation methods to separate classes. * add ComplexColumnData.hasCells * make currentTimeoutNanos overridable CDCWriteResponseHandler doesn't use the default timeout taken from DatabaseDescriptor * port 'CNDB-3334 CDC counter support' Add applyCounterMutationWithoutLocks method. The method applies the counter without locks (not needed in CDC as all the writes to the same partition are serialized by CDC Service). The mutation is applied with the given id and clockMicros. The values are precomputed and distributed by CDC Service. Writers receive the mutation with the attached values, thanks to this all the writers may apply the mutation as if they were leader replicas. See the ticket description for details on leader and follower replica concepts used by pure Cassandra counter write path. * make RepairedDataInfo public RepairedDataInfo.NO_OP_REPAIRED_DATA_INFO is used by CDC to denote no repair data info. (cherry picked from commit 5191be202d3361415e220b9aabd6bb48c8a18cfc) (cherry picked from commit 75b8d120863deb997d9e483ff8fa634c7516d297) (cherry picked from commit ba2a0a64f566e0a4ab8252f27013871d6132231e) commit 6b0c581cd7ac7f7af0ce19caee1268629ffed77a Author: Jaroslaw Grabowski Date: Thu Aug 11 11:58:55 2022 +0200 STAR-1575 port cndb-1496 and 1145b3e61f4 from DSE * port cndb-1496 Enables overriding keypace with a prefixed keyspace. * port 1145b3e61f4 from DSE "Add KS override for schema altering statements (#19372)" (cherry picked from commit 2eaf23dd7d2d6c0a47f0fcfd3a8cc932093f64bd) (cherry picked from commit 0ccdffa659ff75cd9298f91a0da4e474075b5a8e) (cherry picked from commit a3962adb78705224fe670b80602001ea35ad05f8) commit a0618beac8d131883b4765e2ad2df1aa222df893 Author: Jake Luciani Date: Wed Aug 10 14:21:06 2022 -0400 STAR-1495 adds keyspace overrides to all types (#499) (cherry picked from commit cf7602d35e751ed16764a03dd20d52e742618195) (cherry picked from commit eb3632fdd23bae0bf06cf84ad023d4a3206c5970) (cherry picked from commit 8c1b0d027b860ca2afc80a5de64db877c6cdb5b1) commit c0992a69387ff5bfeb1fe099450a05b38c3c3918 Author: Jake Luciani Date: Tue Aug 9 11:39:54 2022 -0400 STAR-1494 allow coordinator driver change address lookups to be (#501) redefined (cherry picked from commit 1e221d6fe7af5f1d5704e25ae02e3e0db76231bc) (cherry picked from commit fdad35fd299d221defd4aa726e17857d65c396b5) (cherry picked from commit 74647df259500a2f235a6b567219a4fba5375b90) commit 880dc4219b7ccba47a593a50b873e37fa30c15a3 Author: Mike Adamson Date: Wed Jul 27 12:05:58 2022 +0100 STAR-1567 Initialise virtual tables for distributed tests (#498) (cherry picked from commit 8c8b01238e5e5203b3387df3fe4a134e5c931941) (cherry picked from commit 292063c4851428f5fc20965c63fa8326669009c4) (cherry picked from commit cbfb8fd6f7d83b7f2d33cf71a86c46c457ddbc23) commit 89d23757b1af05419f16c2384f166d4d78dcb4b0 Author: Mike Adamson Date: Tue Jul 26 16:21:40 2022 +0100 STAR-1497 Fixes for CNDB SchemaTest (#497) * STAR-1497 Bring in CNDB-1889 to allow quoting of user type reserved words * Use correct name when comparing table options in table_properties_ignored (cherry picked from commit ad948c80e01f0b268b62f90edb4c9cdbbbd4d887) (cherry picked from commit 6f7c9d73932a01dcfd9e0b47c95ac00ea959613e) (cherry picked from commit 0f144bb256203910f196c2d15a351fa0f85ea28b) commit 9dc941ff48bd0ba39dfb48724f12351c278d778c Author: dan jatnieks Date: Mon Jul 25 17:49:58 2022 -0700 STAR-1485 Ports code from DSE to clear AuthCache entries using a filter (#496) Ports ability to clear AuthCache entries using a filter from DSE; this is used by cndb to clear all permission cache entries for a given user (RoleResource). (cherry picked from commit 0a7e79555bb15129d030dc02eb3591f876a6660e) (cherry picked from commit bbda8a69d08f8b4e26c372c70881209e09ab9e68) (cherry picked from commit 6546d44a119478f5ef3ea6762620c43777b1ec5f) commit 7042a2702fc254618012e12b9f6e7181b5d956eb Author: dan jatnieks Date: Fri Jul 22 16:09:51 2022 -0700 STAR-1538 Add metadata_directory to InstanceConfig (#494) (cherry picked from commit 50c25cfee6e072ae133f1de216029eadb9f2fdab) (cherry picked from commit 9c2e3543683eac10f8c2882a366f9453a8daeb2b) (cherry picked from commit ed2a0f081656d9252582ba4536a627cb27ec25ac) commit 7deb4f0a7addb543da3f35c789f54e7d721c81f1 Author: dan jatnieks Date: Fri Jul 22 16:05:23 2022 -0700 STAR-1364 Use `Mutation.apply` instead of `applyUnsafe` to persist the commit log, so the expected mutations will be replayed. (#493) (cherry picked from commit 2cad10d45664478884e04f476aa5bf8d413b4623) (cherry picked from commit 5bfc22510014f19dc201e9ce93c38d8dd195f3c2) (cherry picked from commit 51fa2beb308ed57bf19c27078b9ac1fda7c5dcde) commit 22f5310284008530b695730c2e2d2d3226c9e61e Author: Jake Luciani Date: Fri Jul 22 10:34:56 2022 -0400 Fix NPE when cloning ResultSetMetadata (#495) (cherry picked from commit 8335dbb74ce2774432689e940f681a5f453c6579) (cherry picked from commit c5a3f30047492c16dce77ce41c0bb56109b12246) (cherry picked from commit 7743b70c81b43887e47d502c9030e9f8e3ba2cc2) commit 12a32e51b39583336090a98f307ca80eca428098 Author: dan jatnieks Date: Thu Jul 21 07:30:13 2022 -0700 STAR-1505 doTruncateBlocking should be called from execute not executeLocally so it can be overridden by alternate implementations. (#487) (cherry picked from commit 6104667bf27f57992b5e96cbbffbd3c8aaada153) (cherry picked from commit 927f5f4b178f4b612885ea9a8ccb4bd3beeffcf4) (cherry picked from commit 13c1b8e38bd9c1b3f7e9cd7d8d5a3d7147a2d505) commit e6b212b2f73246e3f0f938026e6fb1537f73ac4d Author: Mike Adamson Date: Wed Jul 20 17:15:59 2022 +0100 STAR-1487 Changes required for MetadataManagerTest (#491) This includes the following fixes: Allows the index to be queryable with status UNKNOWN Caches component files in Descriptor to avoid a race condition when the RemoteStorage for a tenant is unregistered (cherry picked from commit 9643bef112ea331ea81dca447ed3b189554a9297) (cherry picked from commit ccd3b18745822494428a83544edce3a9baa2282b) (cherry picked from commit 2ebd8832d66e82afd3a71922714cdb2fe0e81c83) commit 220852db9cee54c8cc244c8661d4174310c9cc94 Author: Jake Luciani Date: Tue Jul 19 14:05:05 2022 -0400 Convert SS call to use IFailureDetector (#492) (cherry picked from commit 6aeb2ba04729a44beb3fa4195923a1dfd8684110) (cherry picked from commit 62e1b50870cbde762bd916729ceef4b825577fe8) (cherry picked from commit 6fb43cceb928d5836a626c391a274a9ebd6d5bf6) commit 22716ce2bf3448eb231d790db6dfa7b724cf69f0 Author: Mike Adamson Date: Tue Jul 19 17:01:29 2022 +0100 STAR-1530 CNDB-259 Make default index configurable (#488) (cherry picked from commit 68041583798023c23bfa68ddcef4abae5856b577) (cherry picked from commit 83bc62548f5e25e19bb16cac1b54bfad8eca6d3c) (cherry picked from commit b2cb657672a7de69a35a9b2b262725a3208e2680) commit 913619bdbc442502bffa079956696a3029aacd77 Author: Mike Adamson Date: Tue Jul 19 16:59:13 2022 +0100 STAR-1531 CNDB-1861 Replayer throws error when intercepting a table drop schema change (#489) (cherry picked from commit 32cb396f62943282cb9b647e2206f20686cb26c4) (cherry picked from commit ef3241c24f67974d08ca6e0eb3d50bb2d8bcf21c) (cherry picked from commit b9e5a6a887d318ec4fe304dfec1c06f9072b0f96) commit bff4ccf3d04bf20a288b28394e4b2bd0b5b31311 Author: Mike Adamson Date: Tue Jul 19 10:18:41 2022 +0100 STAR-1376 Restrict which SystemViews are loaded by system property (#467) * STAR-1376 Restrict system_views tables by system property (cherry picked from commit 0c5a16b60a7b7f509011b3c18898bce8cc91daa1) (cherry picked from commit 441767276b2e3ddc307567cd04aa4a7abd240279) (cherry picked from commit bd5568437404a2e4948f204f074ffefe98c46dd3) commit 8081f7d1a6268bb1a10cf514a1ef39f229985e38 Author: Mike Adamson Date: Wed Jul 13 14:49:41 2022 +0100 STAR-1424 Make UCS the default for all tables Includes the following additional changes This also sets the correct list of table options allowed Stop UCS filtering SSTables during anti-compaction Make sure SSTables are correctly grouped into their compaction arenas Authors: Mike Adamson & Branimir Lambov (cherry picked from commit a4d370a19776fb962b10ce01dbcb13fb74e3f62f) (cherry picked from commit 1263eb977fa1ba48c01405ee7932ba1adaa5352e) (cherry picked from commit 07cc2e12fa77a11022839c123fb2b7d1e6d9e5bd) commit bbfa33ebdfd19bf05d78b01e661ba522b02ef553 Author: Mike Adamson Date: Tue Jul 12 10:06:06 2022 +0100 STAR-1329 Implement storage persistence and virtual tables for Nodes (#445) STAR-1329: Use on-disk storage for Nodes Replaces the Nodes implementation with the implementation from 6.8 and replaces local, peers and peers_v2 tables with virtual tables (cherry picked from commit 46da723e4bbe09efc622ec3502ad6dfd28cfe615) (cherry picked from commit 01e3c1b3f30366a3682b3d04d76a2b93cedcfe22) (cherry picked from commit 83fd2915bb95c8335c0df0b7d59e946d2de83235) STAR-1329 Fix call to Nodes.Peers.getPreferredIP conflict with CASSANDRA-18200 (cherry picked from commit c6765cdb47174a29dbc2994cda881eb59dcc6770) STAR-1329 Fix conflict in CommitLog with CASSANDRA-18153 commit ac3f1cc3ce0edc4d31487a9c2817004909a9f613 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Fri Jul 8 05:49:25 2022 +0200 STAR-1265: Support for ULID based identifiers (additionally to UUID) (#482) (cherry picked from commit 9cf43b15ac8150f24f7dd423635586b5908ca262) (cherry picked from commit b89caeab62efec7b999a74898d1b9cc5e8e6d614) (cherry picked from commit b5c110ccc8535b56ac9ab97e4626ac0c94d5fdd4) commit 0675754e1ab15da48884b49661480900aa5e6f45 Author: Ruslan Fomkin Date: Thu Jul 7 13:37:21 2022 +0200 STAR-1467 Add Directories constructor for mocking in CNDB tests (#483) It also adds test to cover CustomStorageProvider, which includes testing the added new Directories constructor. (cherry picked from commit 3cc2ae983cd191eb1e1ac5c5a05965ac33991ca5) (cherry picked from commit 859197d21ff6fe6fc05c9dbe5e651b0d2bc71f5e) (cherry picked from commit abde7fe2a8a77ab9949d6eeab18fe55d19bd6f6a) commit 88f1561c1b646615dfd911274ef4e3d510f02b6e Author: Mike Adamson Date: Mon Jun 27 11:11:02 2022 +0100 STAR-1453 Compute internalLatency timers for custom verbs (#477) (cherry picked from commit d211a70bf9fcf14d57851064cce46fa7bdc6b1d3) (cherry picked from commit bce3a1eeeaee282d5daa2663c3bf421975e461f6) (cherry picked from commit eb90b04a6395d593fb512899bd645e188bb272f8) commit f7a1e0e225fec4f4adefe94c24f58797916ec6e6 Author: Piotr Kołaczkowski Date: Tue Jun 21 11:43:43 2022 +0200 STAR-261: Reduce the running time of testRandomRangeAndBatches This test required 1+ hour to finish and hit timeout on CI. (cherry picked from commit bd41794af4cb29204e19ec360b7eea4830a0dac6) (cherry picked from commit c33c80311a854f9d08c31533e938be89f2eadbec) (cherry picked from commit bbdb810e5197b5751ed548a5cf23f76aba5332d7) commit 80e54f15865d7ff587650a27574fc39e3922f39a Author: Mike Adamson Date: Mon Jun 20 14:41:06 2022 +0100 STAR-1431 Make sure SSTableWatcher is called in Trie SSTable open (#475) The TrieIndexSSTableReader was not calling discoverComponents causing issues in CNDB with the SSTable components not being loaded correctly. (cherry picked from commit 4e3034f17bbe6fd47ec129f2aa4b8b659e4924dd) (cherry picked from commit 6b9fe97eaef8da3f436e99b9835925e26af86325) (cherry picked from commit 7b02c7b45d63c6d27208c065bd052e8ff4957ce1) commit 5a769fada5654ee738214e301f8d45cd90f1ab05 Author: dan jatnieks Date: Tue Jun 14 12:15:55 2022 -0700 STAR-1347 Port CNDB-4547 Add snitch method to sort by keyspace affinity used for reads and counters. (#471) Co-authored-by: Sergio Bossa (cherry picked from commit 05c68a9e313b39c3e26c7b5637ab9067ccbdb943) (cherry picked from commit e176352859e2a71196d251e1b360457537f61b1d) (cherry picked from commit 5092c87aeefd4852a5503884a8cf1c0e7a7ebfad) commit 5849192f32192c5c87310f5cdf510b380878f7a4 Author: dan jatnieks Date: Mon Jun 13 09:14:13 2022 -0700 STAR-1348 Port cndb-3889: do not drop data if cfs is invalidated with dropData=false (#473) Co-authored-by: Zhao Yang (cherry picked from commit d2146ad90322865657b7c7e962ed1658f9c8ebb8) (cherry picked from commit fa4f09ddaaecdd93ad83c5825a76e535731e066f) (cherry picked from commit 5e65f0e7cfbb66ef76ed51279578ec6138deb3f9) commit 630f6d5310c2580f82ad76b591c787faf179e2c8 Author: dan jatnieks Date: Sat Jun 11 12:35:01 2022 -0700 STAR-1352 Port remainder of CNDB-4337 - expose all compaction picks in an aggregate, and their status (#472) Co-authored-by: Stefania Alborghetti (cherry picked from commit e2d0af54554de06f86a19d13af31be8eeaf7d462) (cherry picked from commit 0f4277c6ee8fce9ae3e0359fe416e75cf779214a) (cherry picked from commit 18a0412e09659c9ed073efe6fed9e4d3d02d4f1e) commit 635c2d43ad9e61ae0d21c6c42745cc5a4b7eb8a8 Author: Piotr Kołaczkowski Date: Fri Jun 10 10:40:28 2022 +0200 STAR-1422: Increase the schema agreement timeout in SAI tests to 60s Addresses: org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in org.apache.cassandra.distributed.test.sai.SAIUtil that uses org.apache.cassandra.distributed.Cluster was not fulfilled within 10 seconds. at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:165) at org.awaitility.core.CallableCondition.await(CallableCondition.java:78) at org.awaitility.core.CallableCondition.await(CallableCondition.java:26) at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:895) at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:864) at org.apache.cassandra.distributed.test.sai.SAIUtil.waitForSchemaAgreement(SAIUtil.java:128) at org.apache.cassandra.distributed.test.sai.SAIUtil.getIndexes(SAIUtil.java:117) (cherry picked from commit 86ff718e2b7125afbe053a95f941e335aec1a748) (cherry picked from commit ddfb15e33eba5a5b01dd739d2ecbe3e316ce9bd7) (cherry picked from commit f6fa5194e5b2a5d584e0ab5a0fd18decd1fd1546) commit 554adf4f1e1f6b9d35414414e23ccd147c10a7c5 Author: Ruslan Fomkin Date: Thu Jun 9 13:00:49 2022 +0200 STAR-1340 Add more metrics for counters and batches (#453) The patch adds 2 more metrics for counters: - the lock wait time, - the number of locks in a counter mutation. No tests for them as OSS doesn't test existing metrics either. It also adds the number of batches executed in the coordinator and tests them. (cherry picked from commit 28e3d323bfc37971822bb6d605e0f3f219cc09ca) (cherry picked from commit f5ae60392394fbb28dfab990e8fad55d040972a2) (cherry picked from commit 30815f8c4a6249fc42f74d62ae9fd5f2e9bdbfc8) commit 20f995238555fc8a1d0d397fcf8d8fa0f087125e Author: Ruslan Fomkin Date: Tue Jun 7 20:18:52 2022 +0200 STAR-1353 Change Verifier to allow no column family store (#469) Change Verifier to not require a column family store if the option mutateRepairStatus is false. Also SSTableReader can be created without column family store by passing metadata directly. (cherry picked from commit c006bfc148e4515dcea8a541c8a3a5bf1ca5b42f) (cherry picked from commit e5121d0427a7bb4592ff135bf43a6e94b66b9c7d) (cherry picked from commit ba30f639a6305977cc1758eda2ac2e68a9fbf9d1) commit c673dfeb28e4a1256b3220f5f4a77f37fbf98446 Author: Mike Adamson Date: Mon Jun 6 17:06:08 2022 +0100 STAR-1375 Handle NO_METADATA flag in ResultMetadata.withOverriddenKeyspace correctly (#466) (cherry picked from commit 4499e5e65a30c39c9db122ae603ed5f9644dc5f6) (cherry picked from commit 1d99e6ab4871f0427b1f327ee8294514ceb63e90) (cherry picked from commit 0b38aad25101a579a48d7afebe7cc71c38759516) commit 62e3cbcb4d93b0fd50ccfd53e078cd3c6f9c4711 Author: Mike Adamson Date: Mon Jun 6 12:13:54 2022 +0100 STAR-1374 Handle empty ring in AbstractReplicationStrategy.getNaturalReplicas (#465) (cherry picked from commit b3b22783bb897150817f4dd7bf28213653f5de4a) (cherry picked from commit 05b01737094634c2987a04998ce86fb918f2c1a0) (cherry picked from commit 5ca4dc6a9b28b29d9bc8f8138f077b59a6d84d24) commit 4a777192b2dc0697de07e238f71a16c2d673adaa Author: dan jatnieks Date: Fri Jun 3 11:58:17 2022 -0700 STAR-1345 Port CNDB-4290: make GlobalTidy public and improve repair logging (#470) Co-authored-by: Zhao Yang (cherry picked from commit 207a5e0f1e96c4f0827adc669b01ff1c28362ebf) (cherry picked from commit 9aa7ffa68026bb1c562fc625bba92bfeb0fbd184) (cherry picked from commit 1976a2de72a61c579f5e55e738099ea06265d229) commit 407305906907c4b22fe71eba112bc0f7f9dc03de Author: dan jatnieks Date: Thu Jun 2 08:59:16 2022 -0700 STAR-1343 Port CNDB-4117 Adds snitch method to customize if multiple nodes from the same rack should be accepted when calculating natural endpoints, and plugged it into NetworkTopologyStrategy. (#457) Co-authored-by: Sergio Bossa (cherry picked from commit 9fb939216948df8d12ff6b5b7071d7b18086cd5e) (cherry picked from commit e8892d6f8baa877d529cbecd95b9e97b0e45f3cf) (cherry picked from commit b85f9e204b72d636dae48aad24242535ce63af3f) commit d01cceb0d7c8ae6f50d485b88b209d16934398af Author: dan jatnieks Date: Thu Jun 2 07:11:29 2022 -0700 STAR-1344 Port CNDB-3626 - add operation id to sstable notifications (#463) Co-authored-by: Stefania Alborghetti (cherry picked from commit c7e7105305290af9640a4cd3e51d94bf36ff859a) (cherry picked from commit a8eebb7f85f5fa761a5781c8c1e7780ce02c78ce) (cherry picked from commit 21e77e1172ccc91a087149c532887862d1d1a1e1) commit 6cc32b2e549c7518c4807955ef93993b4badcbce Author: Ruslan Fomkin Date: Thu Jun 2 08:40:57 2022 +0200 STAR-1349 Expose scrubbed SSTables in the result (#461) Port of CNDB-3616 (cherry picked from commit d9aa20c0ebd81fb9663738f3cbe0af2e7e1cf249) (cherry picked from commit f47426c18dc42d2d93cfa1bc4c61ab21f66403b4) (cherry picked from commit 1ad907a7c0c3601f714b58b4d98c4f4c86ebad5b) commit 3e335e22c4bd078a8ab4a7e242a81e46dcf991f6 Author: Mike Adamson Date: Wed Jun 1 12:25:00 2022 +0100 STAR-1358: Relax NAME_LENGTH restriction to 222 (#460) (cherry picked from commit 0f7683e165dd3f825a5197e3e9b3aee60e1828e7) (cherry picked from commit 3b53d5191bdf6d6a177b51b87f4b30ba663f380b) (cherry picked from commit 44c531c9cff778148c0a1136304a9ef865c57ee0) commit d5b43ca329165ca914795098bb565aef98395cd5 Author: Jakub Zytka Date: Tue May 17 17:24:40 2022 +0200 STAR-1691: Port more bits of CNDB-4090 (#548) In STAR-1335 we ported most of CNDB-4090 but we missed out a call to StorageProvider.invalidateFileSystemCache() which is required to invalidate the remote storage cache in CNDB whenever we encounter corruption. This was discovered because the RemoteFileCacheCorruptedPageTest unit test is failing. (cherry picked from commit 116f5d55d5a35ecb66613712da29d2840dd8f974) (cherry picked from commit 3feca122aa72298f0fcbf6fb1b83985b4a1255e2) (+1 squashed commit) Squashed commits: [c4c6df9debf] STAR-1335: port CNDB-4508 - add abstraction to StorageProvider for clearing file system cache (cherry picked from commit e1525b268673eeabbd113d59a1c428a502f0d740) (cherry picked from commit 2c271f06c1b3c23cf52438f9c6e09db74b2121a9) (+2 squashed commits) Squashed commits: [627e49a5b25] STAR-1335: make Component::repr public (cherry picked from commit a3494f008a97b3a9a716c428a6578b0d1f4b7841) (cherry picked from commit 5a39a3eeba9f29f9c6023c127cda5be25748eb6b) [4723529bc74] STAR-1335: add legacy DSE API to FileUtils, Descriptor, CorruptSSTableException so that rebasing CNDB is easier (cherry picked from commit 448204a647105b928b633adc5bcde6c140264fcb) (cherry picked from commit 7ecb47d58dd3d41d6ea925fce0894f3439a153af) (cherry picked from commit 85710897cc346b1a655f8a560d1f26200ff581b3) commit 80ccb6563c258174a12db915fa1d0ce579dd9556 Author: Mike Adamson Date: Mon May 30 15:30:06 2022 +0100 STAR-1354 - Handle empty ring in NetworkTopologyStrategy.calculateNaturalEndpoints (#459) (cherry picked from commit d0c4650b223a9ab82d25fce45788a723a8e3f58c) (cherry picked from commit bb04b97b0c9d568131308f613da2deb7480f773d) (cherry picked from commit 4dd4224339f56af98cf6d0c264d188e7b3b9520a) commit c44266a6d890816ee3b7af4ed109c62c17da3c33 Author: Piotr Kołaczkowski Date: Thu May 26 10:40:59 2022 +0200 STAR-1346: Ignore BrokenBarrierExceptions past the end of the test Fixup of STAR-1262. Fixes: java.util.concurrent.BrokenBarrierException at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) at org.apache.cassandra.utils.memory.LongBufferPoolTest$1.freeAllAndSuspend(LongBufferPoolTest.java:448) at org.apache.cassandra.utils.memory.LongBufferPoolTest$1.testOne(LongBufferPoolTest.java:355) at org.apache.cassandra.utils.memory.LongBufferPoolTest$TestUntil.call(LongBufferPoolTest.java:638) at org.apache.cassandra.utils.memory.LongBufferPoolTest$TestUntil.call(LongBufferPoolTest.java:616) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) (cherry picked from commit 5b24215640bac363ea1ec60e816cd1a51eef3a86) (cherry picked from commit 39e166e0f1a7a78768138ef7baa38feaf3285d21) (cherry picked from commit d95dacdc2cd2c32948954236cd484d9bc010646c) commit 43ec0eadcdc745ac3059abf927d24692704432d7 Author: Piotr Kołaczkowski Date: Mon May 23 12:37:20 2022 +0200 STAR-1339: Fix the race causing assertion failure in MemtableTrie Nodes must be created in bottom-up order, so that readers cannot observe incomplete nodes. Fixes: java.lang.AssertionError at org.apache.cassandra.db.tries.MemtableReadTrie$MemtableCursor.descendInSplitSublevel(MemtableReadTrie.java:676) at org.apache.cassandra.db.tries.MemtableReadTrie$MemtableCursor.advanceToFirstChild(MemtableReadTrie.java:626) at org.apache.cassandra.db.tries.MemtableReadTrie$MemtableCursor.advance(MemtableReadTrie.java:559) at org.apache.cassandra.db.tries.MemtableReadTrie$MemtableCursor.advanceMultiple(MemtableReadTrie.java:567) at org.apache.cassandra.db.tries.Trie$Cursor.advanceToContent(Trie.java:206) at org.apache.cassandra.db.tries.TrieEntriesIterator.hasNext(TrieEntriesIterator.java:49) at org.apache.cassandra.db.tries.MemtableTrieThreadedTest$1.run(MemtableTrieThreadedTest.java:75) java.lang.AssertionError at org.apache.cassandra.db.tries.MemtableReadTrie$MemtableCursor.descendInSplitSublevel(MemtableReadTrie.java:676) at org.apache.cassandra.db.tries.MemtableReadTrie$MemtableCursor.advanceToFirstChild(MemtableReadTrie.java:626) at org.apache.cassandra.db.tries.MemtableReadTrie$MemtableCursor.advance(MemtableReadTrie.java:559) at org.apache.cassandra.db.tries.MemtableReadTrie$MemtableCursor.advanceMultiple(MemtableReadTrie.java:567) at org.apache.cassandra.db.tries.Trie$Cursor.advanceToContent(Trie.java:206) at org.apache.cassandra.db.tries.TrieEntriesIterator.hasNext(TrieEntriesIterator.java:49) at org.apache.cassandra.db.tries.MemtableTrieThreadedTest$1.run(MemtableTrieThreadedTest.java:75) Patch by Branimir Lambov, reviewed by Piotr Kołaczkowski (cherry picked from commit 7e300ccf326e17161394989958f3f6ec53310cd0) (cherry picked from commit f79a0c79caee2cced1ed805788325d8485f350fe) (cherry picked from commit a2733f31e6b4ad7f4fa26acd44e209cf43af8ec9) (cherry picked from commit e0ada10f05d6c3770b2bd60bc5b0c1791e474c10) commit a07f4a49f74f7bd526ce7ccb6e2111a061aaaa3b Author: Piotr Kołaczkowski Date: Wed May 18 13:38:20 2022 +0200 STAR-1321: Fix flaky tests in DescribeStatementTest Wait until schema cleanup is completed before launching the tests that rely on clean state. Some tests retrieve the whole schema or the list of functions / aggregates, and they don't expect objects left by other tests. (cherry picked from commit 0ac5727ec3bb78619ab82b0b51a243652ef06311) (cherry picked from commit 198fc029507a4fb2599a279e982080496c0cd0f8) (cherry picked from commit 1f6bc1812690680e21084b86be921460c1a5fe94) (cherry picked from commit a605b0043534adadd0139c159c2f77b519af23fe) commit 0ac6bf47bbd280bb210edece4e4f0b5bc431e11e Author: Ruslan Fomkin Date: Thu May 19 17:36:19 2022 +0200 STAR-1314 Use more CFS getters for CNDB mocked test (#449) Introduces an additional getter in ColumnFamilyStore and increases usage of its getters, so ColumnFamilyStore can be mocked in a CNDB test. (cherry picked from commit 04dbab025c11d782d5d335c2a96ca95fcbf65191) (cherry picked from commit d566637a2ff639b162a0d2653a21bc90aa690eed) (cherry picked from commit 0c4c84239a31799bf6983118655564fcb166115b) (cherry picked from commit de6933c5f9c264e35f6cf5b69e125f729027bed5) commit 42ce4605511742d403c56d4e2759d53f79c9a670 Author: dan jatnieks Date: Fri May 13 07:46:21 2022 -0700 STAR-1255 Port CNDB-3376 to fix problem with compaction statistics (#440) Co-authored-by: Stefania Alborghetti (cherry picked from commit 346400fc858466eac3d781a7481ef129a353d1f8) (cherry picked from commit 8d12f3b1eb4ade363ceed86c5ad1394097569c9e) (cherry picked from commit cb63c4f0912eb86a8ab28fd050fbbd2e0d6cd8c7) (cherry picked from commit d7cfcfc2d05f903e454a233e8e8f6aa14e3532d7) commit 8bd1eab0b17206c6dcfe5bf104c4a663946e0c75 Author: dan jatnieks Date: Thu May 12 07:27:30 2022 -0700 STAR-1297 Add TableMetrics.flushSizeOnDisk method so it can be mocked for testing (#441) (cherry picked from commit 543d814693f7803d52b277e711f638da5de79ba7) (cherry picked from commit 3ecc8bf2c074f345081ebcb09ca6d7581849948d) (cherry picked from commit 8be5e39abd06771ff4537e722d91841b95071e4f) (cherry picked from commit 0d232b254c3249cab8a71b9877803505b3394d2e) commit 1d19093f72639a9dfb7e7ee82dd818c6f346d214 Author: Mike Adamson Date: Thu May 12 13:09:32 2022 +0100 STAR-1328 Decouple StorageService from PathUtils (#443) (cherry picked from commit e9d62099c23c59f775390e26e27dd452896de63f) (cherry picked from commit d3b6c28190833945f5b297655db4ee14328de814) (cherry picked from commit 8d08a7fcaa4341b8a4ecb255f2ea40fe97a4b89c) (cherry picked from commit 3ead15c116100c6ac4272ac4c831175fe88f6e4f) commit b507ce9d1336f94f774bad1d26abb60e750c4cbb Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Wed May 11 17:29:30 2022 +0200 STAR-1320: Upgrade dtest-api to 0.13 (#442) (cherry picked from commit f8a389e4792837b66e8d73e01677eb2f4e8bbd38) (cherry picked from commit f215584298cbeb82ce7213a93f096377272d8fce) (cherry picked from commit 13733a12f38d80da8c8ec52e6e36f954ecc7236b) (cherry picked from commit 48fe81b217d9e8dfd5985350d91159618f62f37a) commit 348e3553ea122a2e7d69db62139596c5aecbf61a Author: Jakub Żytka Date: Tue May 10 17:03:10 2022 +0200 STAR-1317: fix ConcurrentModificationException in ActiveOperations (#438) (cherry picked from commit 40db93abc0b5e12161625601324db3d15560b33e) (cherry picked from commit b4640ba918862987ad5a03b5c5829f7194b1f7d3) (cherry picked from commit 43b32bce98e9a4a3b59d03fd232c4a21d83bac78) (cherry picked from commit 2900c1742b1a4ea9455a2269f0493171922d3b83) commit f522030152c3559bd5bb3d8d41a336c0f25e371f Author: Piotr Kołaczkowski Date: Thu Apr 28 14:15:34 2022 +0200 STAR-1262: Multiple fixes in BufferPool and LongBufferPoolTest This commit contains fixes for several thread-safety and state management issues found in BufferPool and LongBufferPoolTest. BufferPool fixes ================ 1. LocalPool is meant to be managed by a single thread. Unfortunately there was a code path that allowed a thread enter the context of the LocalPool owned by another thread by attempting to recycle a chunk using its recycler reference. This led to a situation where unsynchronized data structures were modified from multiple threads. The patch ensures LocalPool state is modified by the owning thread only. The following sequence triggered the bug: a) The app asks the BufferPool for a tiny chunk on Thread A. We get the chunk from the thread local LocalPool A, we set its recycler to the parent LocalPool and owner to its tinyPool. b) Eventually the tiny chunk gets evicted. Now its owner is set to null, but the chunk is not free yet, so nothing more happens. c) In the meantime buffer(s) of the tiny chunk get(s) transferred to Thread B on the client side. d) Eventually the client releases the last buffer of the tiny chunk by calling BufferPool#put, but it does that on Thread B. e) BufferPool grabs a reference to the thread local LocalPool B and calls put on it. So far all correct. We are returning the buffer to the LocalPool B, owned by Thread B. f) First we call free to mark appropriate freeSlots of the chunk. It is the last buffer, so all bits turn to ones (freeSlots == -1). The chunk has been evicted, so its owner == null. We're not the owner, so we don't need to remove the chunk from the micro queue (no need because it is not there) and we can give it back to the pool we got the chunk from. So we call chunk.tryRecycle. g) chunk.tryRecycle realizes freeSlots == -1, CAS succeeds, now freeSlots == 0 and the chunk needs to be recycled. h) chunk.recycle gets called and calls chunk.recycler.recycle() underneath. But the chunk was allocated by thread A, so chunk.recycler points to LocalPool A owned by Thread A. Here we enter the context of the wrong thread! i) chunk.recycler.recycle() gets the parent chunk and its slab and calls put again. Notice we're calling put the second time now. We're calling it on a LocalPool A, but we are still on Thread B. Danger! j) LocalPool A is the owner of chunk's slab we got the buffer from in step b). So owner == this. k) Now, we're additionally unlucky, and it turns out this was the last buffer of the normal chunk as well. So we have free = 0 and owner == this, the first condition after free is true, so we remove the chunk from the micro queue. The micro queue is managed by Thread A, and we're doing this on Thread B. Here we crash. 2. Tiny chunks (children) are released before normal chunks (parents) in LocalPool#release. There is a chance some of those tiny chunks are free and releasing them would trigger recycling them, allowing also the parent chunk to be freed at the end. The tiny chunks could be free but not recycled yet, if they were returned by a different thread than the one that allocated them (which is thecase in LongBufferPoolTest). It is very important for the test to be able to guarantee recycling of all free chunks. BTW: After all the other fixes, the original order also works properly, but is less obvious and harder to debug. 3. MicroQueueOfChunks#clearForEach could present an unconsistent state to its user, by calling to the externally passed lambda between nulling the items and updating the count of items. This led to an NPE after changing the release order described in point 2. 4. LocalPool#release now properly marks released chunks as EVICTED, so they can be partially recycled. 5. When a chunk was evicted, it was first released and later marked as EVICTED. The actual order of operations with partial recycling enabled was as follows: a) release: clear owner -> tryRecycle -> try change status EVICTED to IN_USE (fail) b) change status to EVICTED This not only didn't really recycle the chunk, but could also race with an other thread trying to recycle the very same chunk and accidentally mark an already recycled chunk as EVICTED. The order has been fixed and now chunk is marked as evicted before attempting to recycle it. 6. The code for recycling of chunks has been simplified and made more obviously thread-safe by making the public Chunk API smaller. The chunk status management has become the only responsibility of the Chunk class. Assertions on chunk status have been added. Recycling is managed by one thread-safe method `tryRecycle` and partial recycling has been made an internal responsibility of Chunk. The original Chunk code had multiple public entry points for recycling the chunk, and it was very hard to prove the absence of race conditions. 7. Fully free chunks are always recycled fully now, even if the recycler can recycle partially. The original code always recycled evicted chunks partially to the global pool, even if they were fully free. LongBufferPoolTest fixes ======================== 1. LongBufferPoolTest forces all local buffers to be released and recycled to the global pool after all buffers had been freed. This ensures all allocated and freed buffers will get their lastRecycled counter updated. Earlier it was theoretically possible (although unlikely) a chunk got stuck in the local pool and never got recycled. The test returns many buffers on a different thread than the one that allocated them, so the bufferPool cannot recycle such buffers until they are evicted from the local pool. 2. LongBufferTest burner thread announces freeing memory not earlier than all buffers are really returned to the bufferPool. Originally it announced freeing memory immediately after all buffers had been taken from the queue, which was not equivalent to freeing them. There was a tiny race window where the main check of the test (in Debug.check) thought all buffers were freed, where in fact they were not. 3. DebugChunk now registers the number of the round when the chunk was acquired from the global pool. We have no guarantee all chunks from the global pool would be ever acquired by the test - e.g. the global pool creates more chunks than needed internally, and they might never be given to the test code during a single test round (again: unlikely, but possible). Now we only verify "if a chunk was acquired, then it is released". 4. The test suspends all threads after they freed the memory completely, do the recycling check and then resume all threads. This change has many advantages: * The bufferPool is brought to a consistent, predictable state at the moment of the check. It makes it very easy to verify the correctness of that state with a debugger. * We can add a new assertion on memoryInUse == 0, to check if all buffers were truly returned to the pool. This makes it more explicit that the test is doing the right thing. BTW: adding this check uncovered a bug in memoryInUse tracking (fixed by a separate commit). * We don't need the code checking for stalled threads. CyclicBarriers do that for us. * The worker thread code for freeing memory completely was separated from the other test code, and it is way easier to reason about. I confirmed that the test works fine even if the delay between rounds is as low as 10 ms. The above fixes should make LongBufferPoolTest insensitive to the duration of a single round. ------------- This commit fixes the following intermittent errors thrown by LongBufferPoolTest: ERROR [main] 2022-04-13 13:05:49,184 LongBufferPoolTest.java:618 - Test failed java.lang.AssertionError: null at org.apache.cassandra.utils.memory.LongBufferPoolTest$Debug.check(LongBufferPoolTest.java:108) at org.apache.cassandra.utils.memory.LongBufferPoolTest.testAllocate(LongBufferPoolTest.java:292) at org.apache.cassandra.utils.memory.LongBufferPoolTest.main(LongBufferPoolTest.java:611) java.lang.AssertionError at org.apache.cassandra.utils.memory.BufferPool$Chunk.get(BufferPool.java:1315) at org.apache.cassandra.utils.memory.BufferPool$MicroQueueOfChunks.get(BufferPool.java:576) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.tryGetInternal(BufferPool.java:900) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.lambda$new$0(BufferPool.java:739) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.addChunkFromParent(BufferPool.java:952) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.tryGetInternal(BufferPool.java:907) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.tryGet(BufferPool.java:893) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.access$000(BufferPool.java:710) at org.apache.cassandra.utils.memory.BufferPool.tryGet(BufferPool.java:205) at org.apache.cassandra.utils.memory.LongBufferPoolTest$2.testOne(LongBufferPoolTest.java:517) at org.apache.cassandra.utils.memory.LongBufferPoolTest$TestUntil.call(LongBufferPoolTest.java:579) at org.apache.cassandra.utils.memory.LongBufferPoolTest$TestUntil.call(LongBufferPoolTest.java:557) at java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.NullPointerException at org.apache.cassandra.utils.memory.BufferPool$MicroQueueOfChunks.add(BufferPool.java:513) at org.apache.cassandra.utils.memory.BufferPool$MicroQueueOfChunks.access$2200(BufferPool.java:480) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.addChunk(BufferPool.java:963) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.addChunkFromParent(BufferPool.java:956) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.tryGetInternal(BufferPool.java:907) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.lambda$new$0(BufferPool.java:739) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.addChunkFromParent(BufferPool.java:952) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.tryGetInternal(BufferPool.java:907) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.tryGet(BufferPool.java:893) at org.apache.cassandra.utils.memory.BufferPool$LocalPool.access$000(BufferPool.java:710) at org.apache.cassandra.utils.memory.BufferPool.tryGet(BufferPool.java:205) at org.apache.cassandra.utils.memory.LongBufferPoolTest$2.testOne(LongBufferPoolTest.java:517) at org.apache.cassandra.utils.memory.LongBufferPoolTest$TestUntil.call(LongBufferPoolTest.java:579) at org.apache.cassandra.utils.memory.LongBufferPoolTest$TestUntil.call(LongBufferPoolTest.java:557) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) (cherry picked from commit 6b1d96d3b5747e683778fdaa53e2bf319c484357) (cherry picked from commit 9590a6d91399ddc71f27ae4de7ea3e34e9567aa0) (cherry picked from commit 332a125c51098db8487a1400ec57782c8d662b6e) (cherry picked from commit 6928ad13b687d7e60e4ad7b58024ae9510703ba0) commit 89b103294b3ed2026adf5ea024ceb982c81033dd Author: Jakub Żytka Date: Fri Apr 29 13:35:55 2022 +0200 STAR-1294: fix race condition in CompactionOutOfSpaceTest (#434) CompactionOutOfSpaceTest suffered from a race condition where compaction would remove an sstable that was being checked by StartupChecks inside of CassandraDaemon::activate. This was fixed by running activate first, before initializing CqlTester. Explicit registration of CassandraDaemon in StorageService was removed because CassandraDaemon does that itself. (cherry picked from commit 62e012f71535f1c7fd26309c9bb5b412f1ad1c1f) (cherry picked from commit 8b53f4057969f494ac8200312f4859ec4408083d) (cherry picked from commit de134e5722d0e727079bdb92340c1ed3503cfe21) (cherry picked from commit 64147e926900b574e4139635b4d8e1a114d42320) commit d44c43f2a86aa51d4cc76d722780bdf888da28f2 Author: Ruslan Fomkin Date: Wed Apr 27 14:51:45 2022 +0200 STAR-1278 Wait for SAI index creation (#430) (cherry picked from commit a852ab37d171ccfc0597456ef00a79d39e34c926) (cherry picked from commit b50260869ef29f185db0b7d099ea9a1030c91c4c) (cherry picked from commit adf8886ca2be4ff0ec21d8dd30975902b10b30b1) (cherry picked from commit 8917d15815fd120f088ea551eba36ae8493c46fd) commit ca7b475cc45df763e4fcf5f953fc4ec49feccfde Author: Jakub Żytka Date: Wed Apr 27 12:47:31 2022 +0200 STAR-1291: bump server side request timeouts in test cassandra.yaml to avoid spurious failures in CI (#431) (cherry picked from commit 2f0c21e9366b89565c89570a86ffbdc006512eac) (cherry picked from commit 8c2255a62a6a21cdc1c6e2dded44131bcad5b1dd) (cherry picked from commit 25416020729b78d1e8c4e84e4fb7c849cf34c124) (cherry picked from commit 5a2d16d4ed97c1b9c4388db8c57abd1e6381fc0c) commit 828af4f11b05acb9e5f2ad97a649536e65d9c519 Author: dan jatnieks Date: Mon Apr 25 11:15:42 2022 -0700 STAR-1258 Expose setOpObserver and cleanup methods in AbstractCompactionTask for external callers (#426) (cherry picked from commit 3b7e97735c49b175c214e3c60b1f3929d78f41e3) (cherry picked from commit dbab8c0bb42cc9070aac1123f669d87963674e7d) (cherry picked from commit 8dd460125e020c5819a474e8831b904c3b9aae72) (cherry picked from commit f78cfe4e7d7b91a1fb32be7d94ba5c7a208371cf) commit 1834a5fb31bd85297a6d572eee0b35009373afcd Author: Ruslan Fomkin Date: Mon Apr 25 12:45:01 2022 +0200 STAR-1272 STAR-1273 Make Tracker and CompactionTask visible to tests (#428) * STAR-1272 Add visible for testing API in AbstractCompactionTask * STAR-1273 Make Tracker.notifyAdded visible to tests (cherry picked from commit dd35a9bf12ffd0b4f3f933547581a2c940010bd2) (cherry picked from commit 518cd8f52c4a6edc014ea7b40750731f14400e08) (cherry picked from commit ff9e2753df4fa9b87394e1cf5891f43250924e7a) (cherry picked from commit 70a9ed667369d8514dfa0505e4f62d16e3e5c166) commit 7c749565235d46c3581f32702eff530e85aa9a14 Author: Jakub Żytka Date: Fri Apr 22 13:05:57 2022 +0200 STAR-1185: fix race conditions in SSTableReader.GlobalTidy (#416) (cherry picked from commit 9e0869a9735d3ad9aaa9b06f80d09c6a274ac90d) (cherry picked from commit 2995937a7771dcdd2e8a502e83a0c59b4507fbc4) (cherry picked from commit 97f3a43b7373e6f708dbf17ffffe20f9381ac36d) (cherry picked from commit 4383628fa18aac0b60e231aeb3de940ef5b61cdd) commit 81cf3b49ce7108dc33411ea3d7a83abab550454d Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Fri Apr 22 07:44:19 2022 +0200 STAR-1260: Improve testing performance (#424) Improves unit testing performance by: - disabling immediate flushing of local schema changes - this is not needed as unit tests never restart the node - shutdown messaging service immediately - disabling bytecode verification - skipping all build steps when testing (testclasslist goes will just run tests, assuming the project is already built) The improvements were applied to default IntelliJ test configuration. They are not used by default when building with Ant - they can be enabled by adding properties to ant command line Some tests needed minor fixes due to those changes (cherry picked from commit 3cb86e78c9c804981c753b28d4edf7e442dfbe97) (cherry picked from commit 8e272210edb5ebf1af2b8cefbeb746ec2709117e) (cherry picked from commit 2b9a184d129df985ae731437ca03562583728e2e) (cherry picked from commit b007c540b04f0ddfc63e5553919b47fd9bdfd676) STAR-1260 Fix rebase conflict with CASSANDRA-18000 (cherry picked from commit 3febcde1d250dfef60a5006e83bac19187eeed0e) commit 605e4bd19e5d03881112691a39d0141f3b8265cf Author: dan jatnieks Date: Thu Apr 21 15:14:55 2022 -0700 STAR-1257 Port CNDB-2292 - fix memory leak due to lack of call to BackgroundCompactions.setPending (#420) Adds UnifiedCompactionStrategy.setPendingCompactionAggregates for use by external callers. Co-authored-by: Stefania Alborghetti (cherry picked from commit 76c73d68d774af9d52ae82430fe972d15751545b) (cherry picked from commit 741ee334351c629bf0cb04f89efdcc7f9bdc084b) (cherry picked from commit 8cb3ff05a2fa305ed983273995dd9732bb00e2c0) (cherry picked from commit 20378b277557a645bc1560e92d2db1060adce022) commit 5780d3aecc3239d510fa48383b70e9f481d232a9 Author: Ruslan Fomkin Date: Thu Apr 21 16:45:49 2022 +0200 STAR-1246 Add unregister of compaction progress listener (#425) (cherry picked from commit 7bacbfcab3b571d77515635a1b12d735f98a2f39) (cherry picked from commit 4f4a59eb051ed5566c0a38a4025a60731b264ab9) (cherry picked from commit a8074966190b972e60d87b7643df033cdb599e9d) (cherry picked from commit f4216b27eb54bf0bac8585ca08c54aeabbd4ab40) commit 4b8209787be04f0066f829e36b116c90dd0efe22 Author: dan jatnieks Date: Mon Apr 18 15:09:42 2022 -0700 STAR-1214 Modify UnifiedAggregate constructor argument type to "? extends CompactionSSTable" (#423) (cherry picked from commit eb9919c469cf7e98b2cab4cbe219895a2b6f8c0e) (cherry picked from commit 068ce08cd72876ce1b7ed1d7c9b375b83b4870e5) (cherry picked from commit 88697692b95bcb0ae273db9e5f9d2d010f3cc7ab) (cherry picked from commit 9d13eeddf70037295f54f438f5049556045752a1) commit 65cc5ce92b9bc288dec4ecd42951c5f1e67e886b Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Fri Apr 15 06:28:27 2022 +0200 STAR-1248: Report leaks as test failures (#418) * STAR-1248: Report leaks as test failures Additionally: - report before/after class times (cherry picked from commit b9217c7d9287d2ab666f1906c31beb82853edf82) (cherry picked from commit 0f56603485ec25f05238e7a586fc7ed9fe8e2cae) (cherry picked from commit 112862e08236f82a4f4ae38f36f94725605e144c) (cherry picked from commit 18755c8ac8c40a8e9ebdb487a3841e3b492cca69) commit a7398395cd3ee6300dfcc90a9aaef9dca1e26718 Author: Piotr Kołaczkowski Date: Thu Apr 14 21:44:00 2022 +0200 STAR-1263: Disable mock-maker-inline mockito extension (#422) STAR-1263: Disable mock-maker-inline mockito extension Due to a limitation of distributed test API InstanceClassloader that cannot load .raw classes, mock-maker-inline extension fails to load in distributed tests. Fixes distributed unit test failures caused by STAR-1253. Partially reverts STAR-1253. (cherry picked from commit 4cde4d31fde7b40d4a78bd2b92462d4731cf98ca) (cherry picked from commit bd1b91992ef6a17d640292806b5ae65cce42cd58) (cherry picked from commit 102c36158fb2482abcd3d07711b52f515581a48b) (cherry picked from commit 221573dfe3c5a7704693a5ff8747abf27f12b7be) commit 8a817c2f3e32dcb82a6738fdea25875c27b66c22 Author: Piotr Kołaczkowski Date: Wed Apr 13 10:40:28 2022 +0200 STAR-1261: Don't try to establish TLSv1.1 connections in test TLSv1.1 has been disabled globally in OpenJDK and various other JVMs in April 2021. Although it is possible to unlock it by editing the java.security file, the test cannot unlock it by itself. Settings in the java.security file are stronger than the settings passed at runtime when building the SslContext. Fortunately, support for TLSv1.3 landed in OpenJDK 8 some time ago (but is not selected by default), so we can use that protocol to test the protocol options. Fixes InternodeEncryptionOptionsTest.negotiatedProtocolMustBeAcceptedProtocolTest. (cherry picked from commit 2a621379566831c96e0968704887b20b386870f0) (cherry picked from commit ba31552c03fb0ad8347fe077fb7e3f4742228829) (cherry picked from commit 80963149d0e1d010cff955e548b73ed535f0897f) (cherry picked from commit 817e0a16a028a5997dfd8d2297a33177f88ed4c7) commit 6b687f791832409767f26f4cc0a765b773732dbc Author: Mike Adamson Date: Wed Apr 13 10:31:57 2022 +0100 STAR-1259 Relax access to Schema.load for persistence-dse-db-all (#419) (cherry picked from commit a83bd8270a11641e5c9d600bdd50faaa2adfa77d) (cherry picked from commit 94619cdeb0c30946b7e7ce8c845a8688447b4db8) (cherry picked from commit d9e305c56611b63861c5e93c59141dde282cd243) (cherry picked from commit a172b5f70743efda4713e4cf88e9d7e4590ec6e0) commit 1434dfa71b465361785b80ab6aa3a19aff3bd453 Author: Ruslan Fomkin Date: Tue Apr 12 22:35:25 2022 +0200 STAR-1242 STAR-1243 Extend compaction pick for CNBD need (#413) This patch allows to set the transaction UUID directly when a compaction pick is created, which means that in CNDB we can use it as a key for compaction tasks, making debugging of compactions much easier (since we can grep the same UUID). Currently, we're using the name of one of the sstables as the unique key of compaction tasks, but this is problematic when analyzing the logs, especially after a restart. Also, the transaction UUID was already a property of the compaction pick, but it was nullable and volatile. It is now final, which is an improvement, and a boolean now signals whether the pick has been submitted or not. Port of CNDB-2291 and of minor part from CNDB-3376 Co-authored-by: Stefania Alborghetti (cherry picked from commit d0265d09e0df0815e764e3fa01d2ebb056439349) (cherry picked from commit e0e6964f52c052a0872b7fccf21e546e5f516e40) (cherry picked from commit f7e356dd3b2011a1f62da3e7d247788e8cef7dea) (cherry picked from commit b3fcb8d322091d571a5f494b3a725dc5b7f69d64) commit 9f99d88949480a0c5f00ea272946fb6e4bc4ba9c Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Tue Apr 12 19:54:33 2022 +0200 STAR-1235: Fix deadlock (STAR-996) (#408) The deadlock happens when we want to flush a change as part of Nodes update, when we update some Nodes value which requires immediate flushing. Flushing triggers compaction, and compaction tries to update some endpoint state in Gossiper (sstable versions). Endpoint state in Gossiper is automatically synced with Nodes, so any update on endpoint state makes a synchronized request to update the relevant Nodes value. This way we attempt to update Nodes inside updating Nodes which causes a deadlock. In general we should avoid such situations, but in this particular case, sstable versions is not maintained by Nodes and therefore we do not have sync such change with Nodes (in fact, it would be empty update). The fix consists of determining if updates on Nodes are really needed before actually calling synchronous update on Nodes. I've made updates in endpoint state synchronous because otherwise we could have a situation when gossiper reports different values than those represented by Nodes (or by system tables before Nodes were introduced). (cherry picked from commit 981cf736af1774d06409249a9d1fba61dc079a4c) (cherry picked from commit 3e4d9b56325ab3ba69eea5b098c0cb5c12aca644) (cherry picked from commit 4686a56f980b91f6074cdf76d19526fca1855fa1) (cherry picked from commit 3ef4a0d8b1c058c99f1ec9a8803b63a8110c05a8) commit eaa8ec0a5e1a6913f678a7de8566f5ad130ff615 Author: Piotr Kołaczkowski Date: Tue Apr 12 15:40:57 2022 +0200 STAR-1253: Fix resource leak in SAI MergeOneDimPointValues (#415) STAR-1253: Fix resource leaks in SAI MergeOneDimPointValues and BKDReader All iterators added to MergeOneDimPointValues should be eventually closed after use or immediately if unused. Fixes code introduced in STAR-121. Fixes flaky KDTreeSegmentMergerTest. (cherry picked from commit ca8b8d9918b5ee0e1f7f464b04fb62af8d76dd2c) (cherry picked from commit aecd3a01e580ed16d3d281f53687689132615b38) (cherry picked from commit 61dd6269c76bee5082eda4c0b502ba0f907bc924) (cherry picked from commit 1d95d735deb5f3fa150d21861476a26fee7adad3) commit 282c66b0ef475734d411bd52d6c8f011082cd4d8 Author: Mike Adamson Date: Mon Apr 11 13:39:35 2022 +0100 STAR-1249 Relax access to setTracingId in Message (#417) (cherry picked from commit bbd4a1b07fe731e98c12a00c6c4d06cc34bc82ee) (cherry picked from commit 2362c83b7f4dddbbc8000af0982bbaeeb0dc1175) (cherry picked from commit 6227c5bbbad4d88fc9c2c71a8d7138967ac3e1e3) (cherry picked from commit 3c8b7c8883212bd6a8b0083867692a4a8c1d54ab) commit bf56e846f1aa180fc6410b0d7a20891e1ab69cbf Author: dan jatnieks Date: Sun Apr 10 18:05:21 2022 -0700 STAR-1252 Add totalCompactionsFailed metric (#414) (cherry picked from commit 323855bec0a5d4095fb404aacc3b085bcb5c22d8) (cherry picked from commit a724da481cc208af1f2d9513b825d17f75a7a26e) (cherry picked from commit af8a6566cc536e4ad6212d094c8b8937d481b6a0) (cherry picked from commit 026c9395cee09863ff40e1243164f3b3319c8278) commit a36159e8525eeec6a3aa1eed3133cca0f6a21437 Author: Piotr Kołaczkowski Date: Fri Apr 8 21:03:26 2022 +0200 STAR-540: Fix issues with SAI queryability and status propagation (#410) STAR-540: Fix multiple issues with SAI queryability and SAI status propagation * STAR-540: Don't query index in unknown state Allow index to be queried only if its status is known to be BUILD_SUCCEEDED. For all the other states, including the UNKNOWN state, it is inherently dangerous to query such index. A query to an index that's in the middle of building process may return incomplete results. Fixes flakyness of QueryWriteLifecycleTest manifested as: junit.framework.AssertionFailedError: expected:<[MS, AL, TN, UT, KY, LA]> but was:<[MS, AL, TN, KY, LA]> at org.apache.cassandra.index.sai.cql.IndexQuerySupport$BaseQuerySet.validate(IndexQuerySupport.java:568) at org.apache.cassandra.index.sai.cql.IndexQuerySupport$BaseQuerySet.rangeQuery(IndexQuerySupport.java:551) at org.apache.cassandra.index.sai.cql.IndexQuerySupport$BaseQuerySet.execute(IndexQuerySupport.java:354) at org.apache.cassandra.index.sai.cql.IndexQuerySupport.executeQueries(IndexQuerySupport.java:215) at org.apache.cassandra.index.sai.cql.IndexQuerySupport.writeLifecycle(IndexQuerySupport.java:97) at org.apache.cassandra.distributed.test.sai.QueryWriteLifecycleTest.testWriteLifecycle(QueryWriteLifecycleTest.java:30) Additionally, this commit fixes the code for waiting for index readiness in SAIUtil. The original code ignored the fact that if the schema with the new index hasn't been propagated to the other node yet, that node does not know anything about the index and doesn't return its status. Unfortunately a lack of status was considered as "index queryable" and the waiting didn't happen. SAIUtil.waitForIndexQueryable has been modified to rely on index status from gossip. This way it can see the exact same state that a coordinator uses to determine index availability. The downside is that now some tests need to run with gossip enabled. * STAR-540: Fix SAI index status propagation issues This commit fixes sevaral issues related to how index status is announced in the cluster through Gossip. 1. When a new node gets bootstrapped, the status of indexes is now properly announced through gossip in all cases, even if indexes didn't need building because they had been already streamed. 2. Index status is now received from a peer even before the peer goes to NORMAL state, while the peer is bootstrapping. This way, when the new node transistions to the NORMAL state, we already have up-to-date information on its indexes and the newly joined node can become immediately queryable. 3. Additional DEBUG logging has been added to display index statuses received from peers. 4. TestBaseImpl#bootstrapAndJoinNode now additionally waits for the other nodes to notice the newly joined node switch to NORMAL state. This prevents some problems caused by tokens suddenly moving later while the test code is running. Fixes flakyness in NativeIndexDDLTest. That test sometimes didn't do keyspace cleanup because the first node had outdated token ring information and incorrectly believed to be an owner of the whole ring. (cherry picked from commit c164af08887be8e180499a901c65c95dc71ed94d) (cherry picked from commit 68112ba43b1842e43cc9dd50674953f0009ec4bb) (cherry picked from commit 8dd8b34758ce306f7bae96bc7724ec2b1ba0e5df) (cherry picked from commit 8f0eb437d6a445ed7eb5c5f368e5ca7b47df99d7) commit 9f5e8219830d516e0179654f2eecc3954f8bed15 Author: dan jatnieks Date: Fri Apr 8 09:48:21 2022 -0700 STAR-1245 Refactor and expose startCompactionTasks in CompactionManager. (#411) Make handleCompactionError public in BackgroundCompactionRunner (cherry picked from commit 7d0b6647a65f88efb71790af80f4376a1c0aa789) (cherry picked from commit bf9236a6e7860c7df570591453a527540a845827) (cherry picked from commit f265dad738999b5ef07e4ce0108c31b5004e22d2) (cherry picked from commit 49de3c51cb15d298fe473e91b5d4cf18dac1e3ae) commit 40eff8abd0babea1fd666fa27177890ae91baae6 Author: Mike Adamson Date: Fri Apr 8 10:25:02 2022 +0100 STAR-1251 Relax constructor access on AlreadyExistsException for TenantAwareQueryHandler (#412) (cherry picked from commit 44d64bc23321c770b69bbab2610996e334a8fe21) (cherry picked from commit 7c2297ed56183fa2b8e3e0dc98a04704795bb5d9) (cherry picked from commit 7057d6ec662b4915d9692fdf7f19cb73286ffb1b) (cherry picked from commit 016a7a9804cbec1b9ad223d61ea398d6ac6eaac7) commit f251c31a83f34ba0bba4076f7faa0ed6075ea59f Author: Mike Adamson Date: Thu Apr 7 18:17:43 2022 +0100 STAR-1249 - Relax access to tracing methods in Message (#409) (cherry picked from commit 8ff3b81c6a1bee5c011fa4e684a71477de6790e3) (cherry picked from commit d4850a710f2bbd3ecf46e103e17fc4a50afa0c2f) (cherry picked from commit 2c9aa44ac9bfad0834b27e639902d65c35885699) (cherry picked from commit dcf87040e3da9ba7cb0a78b7451f5cf9334461e6) commit 8c9f28b3f3bbb77e78c02ac72e66d910f487fc67 Author: Piotr Kołaczkowski Date: Tue Apr 5 21:24:35 2022 +0200 STAR-1233: Use Guava concurrent hash set instead of Apache Mina Apache Mina is not really in the direct dependencies of the project. It compiled accidentally because it got onto the classpath through one of the transitive dependencies of Netty. (cherry picked from commit 3d3e2ee1cc1a057d1d69ba07815d015ef8f377e2) (cherry picked from commit 8cfc38220ac01719159cca801097e8f3dcd097c6) (cherry picked from commit 8ea374bd58af1d9bfa156b44560f05af8b79ed32) (cherry picked from commit dc47329721b9da0e8dd8eb75f0472fb8a4891766) commit 9d2ba61a37edb49421c6586e97966536047e38a6 Author: Mike Adamson Date: Thu Apr 7 15:02:52 2022 +0100 STAR-1244 - Make ResponseVerbHandler public (#407) (cherry picked from commit 8d050e930cc63a7cf79726fb7bc70248c2869f06) (cherry picked from commit f43a8fb78c72c0535afda6ee6efbff8ffb55ca76) (cherry picked from commit 64c7a12070138b643ea15ffe0c4fa18e65b9d9a6) (cherry picked from commit afe1ac419182c927ce68ce252ae2a1ff9b5d844c) commit 1b87b699e68ca94d9b6298603c91e3462a2da88b Author: Piotr Kołaczkowski Date: Tue Apr 5 10:22:41 2022 +0200 STAR-1239: Fix NPE in ReadCommand when index unavailable Fixes the following null pointer exception that may be raised during a read query using index if the schema with that index hasn't been fully propagated yet: java.lang.RuntimeException: Can not deserialize message MessageImpl{verb=7, bytes=c22adae7cf7b0ca710070100817502041d84a560b22111ec9c92efe6fdbada366247a1a80300100c616262726576696174696f6e066163746976650d617265615f73715f6d696c65730c6176675f646d765f77616974036764700269640269700b6d75726465725f72617465106d7572646572735f7065725f79656172046e616d650b6e6f6e5f696e64657865640a706f70756c6174696f6e0b74656d706f72616c5f69641574696e795f6d7572646572735f7065725f79656172077669736974656411766973697465645f74696d657374616d7000020c616262726576696174696f6e0b74656d706f72616c5f6964000100000b74656d706f72616c5f6964000000000010ee6136d2d17c11e8a8d5f2801f1b9fd100000005f07ffffffff07fffffff00d80b847dd5ec3bf69f98e41d9bf0ea9c00000002040200000008d55555555555555502000000082aaaaaaaaaaaaaa9000001010000060000000402000000082aaaaaaaaaaaaaa902000000087ffffffffffffffd00000101000006000000, id=142042, version=100, from=/127.0.0.1:7012} at org.apache.cassandra.distributed.impl.Instance.deserializeMessage(Instance.java:398) at org.apache.cassandra.distributed.impl.Instance.receiveMessageWithInvokingThread(Instance.java:419) at org.apache.cassandra.distributed.impl.Instance.lambda$receiveMessage$6(Instance.java:405) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at org.apache.cassandra.index.SecondaryIndexManager.getIndex(SecondaryIndexManager.java:1264) at org.apache.cassandra.index.SecondaryIndexManager.getIndexGroup(SecondaryIndexManager.java:1293) at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:982) at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:880) at org.apache.cassandra.net.Message$Serializer.deserializePost40(Message.java:797) at org.apache.cassandra.net.Message$Serializer.deserialize(Message.java:655) at org.apache.cassandra.distributed.impl.Instance.deserializeMessage(Instance.java:394) ... 8 more (cherry picked from commit 99469531228cd816ada3bf341fcc91eaf6ec589f) (cherry picked from commit f04de2af522f097b6e76b752a0e3861c269eb3b5) (cherry picked from commit 24e5f7723849c7b0c2717fc6a5d58ba595a0843d) (cherry picked from commit bf3d479e98239a12fc54b2a3daa63e5ef72c55fb) commit c56b818fa619c5b0b8e3ce7282709cadde9ba4d1 Author: Ruslan Fomkin Date: Tue Apr 5 19:20:29 2022 +0200 STAR-1230 Provide CL API to update to remote storage (#404) * STAR-1230 Provide CL API to update to remote storage Extends CommitLog's API to allow update its directory, so a commit log from remote storage can be replayed. * Add test of CL's forPath * Fix test method name (cherry picked from commit 17a05588b7ae9c16d494ca8e0c3d77c8d8b09884) (cherry picked from commit 241c97160985fda6b5adb1d3a59e29ade710386b) (cherry picked from commit 33cb1c6caed816bc7ff0bdd35e957431d814520e) (cherry picked from commit c47530b3cf84e9d2cb3113103a4a013effc8cd79) commit 09306bb6ca62a79ecd32c6aaaeb450e683692a95 Author: dan jatnieks Date: Mon Apr 4 17:22:05 2022 -0700 STAR-1221 Port CNDB-293 Adapt compaction-related code for scheduling tasks coming from etcd (#394) Co-authored-by: Aleksandr Sorokoumov (cherry picked from commit 85a339748ddaea0e9e9e668c668ebdc75bb88ac8) (cherry picked from commit b73913e70d45da33dc2407859c06785f09b76f39) (cherry picked from commit f728a92ce9d8edd0723c5291c99995f7aa426688) (cherry picked from commit 28156c3d9676801e6f14e4b0723b9c3d1e35f0df) commit 32851cc0a88a60e8da16e965649220a1a1e17255 Author: Jacek Lewandowski Date: Thu Mar 31 08:53:48 2022 +0200 STAR-1219: Fix race conditions during initialization (CASSANDRA-17505) (cherry picked from commit 6f828abaa295877712ed6b25f5cd4c60fd0958fc) (cherry picked from commit d20afb01812b0b78fbd77b68a6d7ac1e1221f8e0) (cherry picked from commit 37fdae75806c2c5bfeb523629565d467e3fb06eb) (cherry picked from commit 148be1293947b2d7fa5c7d98851870cb2fbcd579) commit 17d04cbda87e830f8f1fe260c8fa0b4d8887bb5c Author: Mike Adamson Date: Thu Mar 31 18:30:44 2022 +0100 STAR-1112 Improve debug logging for index writes (#402) (cherry picked from commit f2cdc9b126d255f4c28437a7ad0b62b8c824d01e) (cherry picked from commit 96ad36d84a28d541d5ac83a346291b41d05e3267) (cherry picked from commit 86efe5419d920a252003d975d7436f42e562cd84) (cherry picked from commit 01ef55f02649b88559fc335ad5c189d131517d6b) commit 42e4379ec4738963a536d91ec1426c8ba4180135 Author: Piotr Kołaczkowski Date: Thu Mar 31 12:51:08 2022 +0200 STAR-1233: Fix CME in StorageAttachedIndexGroup StorageAttachedIndexGroup was not thread safe and accessed from multiple threads both for reads and writes, which resulted in the following exception: ERROR [MutationStage-1] node2 2022-03-31 11:11:31,938 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[MutationStage-1,5,node2] java.lang.RuntimeException: null for ks: sai_query_keyspace, table: table_2 at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1430) at org.apache.cassandra.db.CassandraTableWriteHandler.write(CassandraTableWriteHandler.java:40) at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:649) at org.apache.cassandra.db.Keyspace.applyFuture(Keyspace.java:478) at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:218) at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:62) at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78) at org.apache.cassandra.net.InboundSink$Filtered.accept(InboundSink.java:64) at org.apache.cassandra.net.InboundSink$Filtered.accept(InboundSink.java:50) at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97) at org.apache.cassandra.distributed.impl.Instance.lambda$receiveMessageWithInvokingThread$7(Instance.java:423) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165) at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137) at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.ConcurrentModificationException: null at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1585) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at org.apache.cassandra.index.sai.StorageAttachedIndexGroup.indexerFor(StorageAttachedIndexGroup.java:185) at org.apache.cassandra.index.SecondaryIndexManager.lambda$newUpdateTransaction$33(SecondaryIndexManager.java:1323) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.Iterator.forEachRemaining(Iterator.java:116) at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546) at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:505) at org.apache.cassandra.index.SecondaryIndexManager.newUpdateTransaction(SecondaryIndexManager.java:1331) at org.apache.cassandra.db.ColumnFamilyStore.newUpdateTransaction(ColumnFamilyStore.java:1437) at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1408) ... 16 common frames omitted (cherry picked from commit 66ab6853da261b2bcbaee1fea18214b3f30f7cd0) (cherry picked from commit 0acee1a08265416bfaef74b6b1c72aa8d504ef1f) (cherry picked from commit 394d486c9bef9ffcb050ab14941b493248b28e3c) (cherry picked from commit 6f9d1b58449b766cdc0fa94f8ad8114446437328) commit 312a09e5c0b29f39655694d71b019b5662bf57d1 Author: Piotr Kołaczkowski Date: Wed Mar 30 17:52:40 2022 +0200 STAR-1229: Fix resource leaks on unhappy path Generally try not to throw in constructors and if an exception in a constructor can be thrown, make sure it happens before allocating resources. This fixes resource leaks reported by SingleNodeFailureTest.testFailedBkdReaderOnMultiIndexesQuery. (cherry picked from commit bda679cd0afd49799b2769450e3178a93058b116) (cherry picked from commit 7fdf59e0d8472ab1d4a1de50df2a8b5f5150aae2) (cherry picked from commit 2a99a3a39fd3922f142f33f8a3d98c52cb511e6e) (cherry picked from commit 29078cbbf5df6322a99be6d8d7689f91ab642bd7) commit 45639340488c2d8c1d9f1f2c2fb3217e3009a53f Author: Ruslan Fomkin Date: Wed Mar 30 19:43:34 2022 +0200 STAR-1231 Add BATCHLOG_REPLAY to ColumnFamilyStore.FlushReason (#398) Needed for CNDB (cherry picked from commit 4e81f0dc8356a39af637076ec39ae1dabbccf21a) (cherry picked from commit 9a765b1a9b26fba37fba9a24263d94e8bdd7fd73) (cherry picked from commit f8d2854a66f4f94198fd38f123159a6351ee277e) (cherry picked from commit 7a18abab3d55f49cb27b53103ab28000cfa5a808) commit 52f26ee20b682abe0c37d266bfedbdf8aa6d0a2b Author: Mike Adamson Date: Wed Mar 30 18:05:28 2022 +0100 STAR-1226 Allow mutation of where clause relations (#397) (cherry picked from commit a303a08b59702ecaf0c68d1bef5184a23a48b04c) (cherry picked from commit 83881ad8fe36601cafeca7d7e879a683163eaaf1) (cherry picked from commit 3da8b623eece6ecc0e9723eb5a8f9a5e648f1ee2) (cherry picked from commit a34ff14cb957c812dca0bd51ef844696b37ad502) commit 04c7b2bc79b93408803a884aa2c57abe0b5fe5ec Author: Piotr Kołaczkowski Date: Wed Mar 30 09:06:38 2022 +0200 STAR-1223: Fix iterator in StorageAttachedIndexSearcher (#395) * STAR-1223: Fix iterator in StorageAttachedIndexSearcher Make the RowIteraror implementation conform to the iterator contract. (cherry picked from commit c4d379233b1f73edac84e88e2c72de6d3421bf5f) (cherry picked from commit ed94c9b034a9aa83ff25e84be41e53dabfebcab4) (cherry picked from commit 8b95b32719ff9f429c5568d074690cd0f0d4a9ef) (cherry picked from commit a509cd053ab74404efbac3c97ac84ad2d1f68f59) commit 475bbfc451148ef22354f732121ad011b40f0ab5 Author: dan jatnieks Date: Tue Mar 29 15:11:53 2022 -0700 STAR-1213 Make ColumnFamily.concatWithIndexes and CompactionStrategyFactory.getRealm public (#386) * Refactor of ColumnFamily.concatWithIndexes * Make CompactionStrategyFactory.getRealm public for use in cndb (cherry picked from commit d0c1af83e6c48a5c0203756b41f8e078b09efe00) (cherry picked from commit acdea04cad343bf400cc2a4e06ed45ba7628f1d0) (cherry picked from commit 5cc12f068693cf854d83839eeb3041af5dd0de2b) (cherry picked from commit e837ba63ec6c5461b11a9b52618410112a776341) commit f9d5132245d77015f32460270e92fb6a8b7337ed Author: dan jatnieks Date: Tue Mar 29 11:30:38 2022 -0700 STAR-1210 Port CNDB-1338 Pass to CompactionObserver#onCompleted if the task finished without exceptions (#388) * Port CNDB-1338 Pass to CompactionObserver#onCompleted if the task finished without exceptions Co-authored-by: Aleksandr Sorokoumov Co-authored-by: Ruslan Fomkin (cherry picked from commit 24e65ac318c31235ace38017410f24aed4219bae) (cherry picked from commit 8e1f7634e1b125301ab74e7bde34db19f828c383) (cherry picked from commit 61c89cc033642fdb8b3761f25c619cd19b6be553) (cherry picked from commit 5fb6587f9bd8f65bdc66f64ecfe65e809854c7db) commit 12f1b1e4593f95af9952d219be53d41db5abdd2c Author: Mike Adamson Date: Tue Mar 29 12:57:18 2022 +0100 STAR-1218 - Implement ExternalRestriction (#389) * STAR-1218 - Implement ExternalRestriction Allows mutation of SelectStatement restrictions by external QueryInterceptor implementations * Added ExternalRestrictionTest (cherry picked from commit e7d18d8cc95734e7406d2c3d49fb97c39bdc90df) (cherry picked from commit 69af96e07e540cb9c2e726e9a14dd51d16e4cc5e) (cherry picked from commit 2a82204ff666e1fc2e59af2b7c871ebc02f37db5) (cherry picked from commit 02db65f003815a54681360e6afbf3666c88ee56b) commit 566375a96f9557e19fda430351f2b48babfca09f Author: Ruslan Fomkin Date: Tue Mar 29 10:48:38 2022 +0200 STAR-1217 Add CompactionProgressListener and totalBytesScanned (#390) Allows to register listeners to observe table operations called in compaction manager. Adds totalBytesScanned to compaction progress. This is port of CNDB-3258 Co-authored-by: Zhao Yang jasonstack.zhao@gmail.com (cherry picked from commit 50008b1e50c3f72bcef5dc8081b0a651def1ef8e) (cherry picked from commit bf123c2ba41083d8e15263082731a7443601f04d) (cherry picked from commit 5a3ef2092e9abe7daadfcddd64c3b6496d19cc5e) (cherry picked from commit 74b7a3a82b122100d8ed05f7b862a1eb632ae892) commit de3061081fcbb37d903e8768c01e1eea18b4297b Author: Mike Adamson Date: Tue Mar 29 09:47:03 2022 +0100 STAR-1112 Add QueryInterceptor to QueryProcessor (#393) (cherry picked from commit c7d3220580c919c4f91a9641265237bbffbb8003) (cherry picked from commit af9298fc46f35a83bd24bd3eb6a45e2f271e6b6f) (cherry picked from commit a161b726585b135f74c8b55310ae759b498e14e9) (cherry picked from commit b655559a4039dbf7cf3dcaffbdc16d2acf0bc787) commit 5ba49465b273444b9f64abaf78745ee8dc1d0503 Author: Piotr Kołaczkowski Date: Mon Mar 28 16:31:17 2022 +0200 STAR-1180: Fix premature skip to the next keyRange If queryController.select(key) returned false, it was possible that the loop in nextKeyInRange advanced the currentKeyRange prematurely, potentially skipping a bunch of keys which led to missing some rows in the result of a query. The code has been simplified by taking the check for queryController to a separate method. (cherry picked from commit f13d8142f7fd217d99d32cf7744695af2d60b616) (cherry picked from commit 858e3f9dde099f71a93d830d1777eb70431c95d3) (cherry picked from commit 69a307464fedd61c82e629d589e94dc85e372c51) (cherry picked from commit 70df51bfa10415cd6685d46765e27a08048c01ed) commit deb3a9b0944ea83468d04c1c189265b0f390605d Author: Piotr Kołaczkowski Date: Fri Mar 25 16:28:26 2022 +0100 STAR-540: Improve test diagnostics (cherry picked from commit 44af3473eba55125580c1bf45541e2c337e35722) (cherry picked from commit 2e964f323c38b6013488b181e0262c2e2ba5d08f) (cherry picked from commit 040bd2d1914f78d55ce551098f9c64d3df00ef18) (cherry picked from commit ad0c6e124722395595b35c996cd3d41ed8160907) commit c6b0b288269d0b512b668edb31df68e8b9e12f5d Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Mon Mar 28 09:46:14 2022 +0200 STAR-977: Add info whether data was dropped to SchemaChangeListener (#387) * STAR-977: Add info whether data was dropped to SchemaChangeListener * STAR-977: Add a test for remove without dropping (cherry picked from commit 0d81a43d7d4194d60231a7cbc796b3ce88a9514e) (cherry picked from commit 841994268b8c1ddc5f58ed30d89e9e5b9102a99a) (cherry picked from commit 1aa95d67f4f12af9efe50a2f9167f6e5c9686deb) (cherry picked from commit bb08df41b11f44b023c525e56b19df5a026a1cc0) commit 7720f68b384004346ddab00ae10f636cc227c6e6 Author: Piotr Kołaczkowski Date: Fri Mar 25 12:25:34 2022 +0100 STAR-1207: Open iterators lazily in StorageAttachedIndexSearcher (#385) STAR-1207: Open iterators lazily in StorageAttachedIndexSearcher In STAR-1180, the StorageAttachedIndexSearcher was modified so that each computeNext() returns an iterator for a different partition. This fixed the correctness issue, but the implementation was far suboptimal - it required fetching all data of a partition eagerly before the final merged partition iterator could be returned. In case of extremely wide partitions that behavior could destabilize the server. This commit gets rid of eager creation of the row iterators. It keeps only one iterator open at a time, and opens new iterators only when the previous one runs out of data. The data retrieval commands are issued lazily, while the returned iterator gets iterated. In addition, the code has been tidied up, organized and documented in order to improve readability and future maintenance. (cherry picked from commit c04af690562446231df509c69a69aadba68de2f8) (cherry picked from commit 6ee5317f16aa5df9d90c50fa8d72535fcfdc7ddf) (cherry picked from commit 4200696b6c55e065057d2dad9c0248b81e4e2105) (cherry picked from commit 5bc7900205b55131c82a86fa46c1d5ec53ada84a) commit 2ad417369f5194e0cd59ab46fc0205c9414fe425 Author: dan jatnieks Date: Fri Mar 25 00:37:33 2022 -0700 STAR-1198 Porting CNDB-3254 to support shared multi-tenant compaction (#384) CNDB-3254 - changes for better compaction fairness in CNDB (shared compactors) This patch performs small changes to CompactionManager and the UCS environment so that CNDB can override some limits, and use the compaction manager to just submit tasks, without any automated rescheduling. Further, the aggregate prioritizer has been removed and combine with the UCS environment. Co-authored-by: Stefania Alborghetti (cherry picked from commit 153ffb82b4a85d089046be84107106a9f40fa305) (cherry picked from commit b2f0cc280a766b068feb7f881e85c0090ba09340) (cherry picked from commit 75a7548394697c7e441a424020acf63acb9970b9) (cherry picked from commit 6efcd47892ef7b2ddcf92f89523038357df77771) commit f842e4b90368fcdf24d171b848ebea6363e7dfe9 Author: Jakub Żytka Date: Thu Mar 24 17:34:47 2022 +0100 STAR-1183: clean handling of no active segment in commitlog (#382) (cherry picked from commit 40e0941b1ae8e2e1f13958fc03d9daa1cff3598f) (cherry picked from commit 2815146d06cc3f763385a3e4df258e577a1cc444) (cherry picked from commit af7af2c95cb65fe1c16409bcc6de5eb857f3dfe2) (cherry picked from commit 0b9dad45ce13a301d70dd4b121b4c67d9f8a8617) commit 73c70b33a1d239dd55a07d7a09a8e607605923b2 Author: Ruslan Fomkin Date: Wed Mar 23 09:43:01 2022 +0100 STAR-1205 Add everywhere() to KeyspaceParams API (#383) To allow creating Keyspace with Everywhere strategy the correct approach is to extend its API with everywhere() method. This is then tested in a very simple test. Needed for tests in CNDB. (cherry picked from commit 8d2210c375f5e62da9e6e443ba8b908dc14881b8) (cherry picked from commit d390b4e6026b386ef8068f4a3c9cbeedb930920f) (cherry picked from commit 27eb798a96053cd13845d428e7a68cfe8308d581) (cherry picked from commit dbe37a61d057426d9a7a8342b24695e3a99a7ba5) commit 4e02a4263da8286c760dbbca98710f1e488a38a1 Author: Branimir Lambov Date: Mon Mar 21 15:33:01 2022 +0200 STAR-1193: Fix flake in TrieMemtableMetricsTest (cherry picked from commit aac83b555425e8bb61d30dfdd041f47f0cb0dbd8) (cherry picked from commit bf1d3b3034aa92ff184341857e7edba8ac278cc3) (cherry picked from commit ac2a1b382e89b4b72301571313348d63a045969e) (cherry picked from commit 52d54700eb1aa50192f4116f54ea71fae23ce354) commit fd64dfa4b6cb30d96aa5febca06a359cfbd1e31f Author: Piotr Kołaczkowski Date: Mon Mar 21 12:37:54 2022 +0100 STAR-1180: Merge same-partition row iterators in SAIS Multiple tests in IndexConsistencyTest started failing when we committed SAI row-awareness (STAR-406). The StorageAttachedIndexSearcher was returning a separate UnfilteredRowIterator for each row retrieved, even if multiple rows belonged to the same partition. Unfortunately C* coordinator code does not merge such partitions properly in some cases. This commit adds merging of the same-partition iterators to the StorageAttachedIndexSearcher and fixes IndexConsistencyTest. (cherry picked from commit 97d706ba754d89f9128f8d5982a4394d56c2d29e) (cherry picked from commit aeae8e892ddc11999e8e914ba6f66def28e6f414) (cherry picked from commit 41dd9270ebd00253040e21130376008e7b3a1c55) (cherry picked from commit 77d697bb02a4d66e7fd0d53a17ede6dcd7fcaf71) commit 3db5bccd13dcc71a90750ed400435bd17308ae87 Author: Ruslan Fomkin Date: Tue Mar 22 09:22:07 2022 +0100 STAR-1199 Test replayer on truncate (#379) Extends the wrapper around initiating mutations read from the log to allow custom methods to be called on replay or skip. Use it to test commit log replayer with TRUNCATE. This port of DB-2721 and needed for a test in CNDB. (cherry picked from commit 14b343a0496c6ca8c42f17853842d96a7d9112ee) (cherry picked from commit 9ff2b6da3c8a382fab27598afc6a77a0277da887) (cherry picked from commit d8d281888b0d4d76fff708fc8f2ea368f50a4c41) (cherry picked from commit b5201ea4daaecc301f083ed044e814312dc6dc87) commit ab7d68e8fdf6adde7a5c688fa3de07bbca0cf8d7 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Mon Mar 21 06:55:41 2022 +0100 STAR-1187: Fix PropertyFileSnitch (#378) * STAR-1187: Fix PropertyFileSnitch * apply review comments * don't error out unknown host when we rather do not expect that (cherry picked from commit 57af35126d2a53a33ebe20cda0eb83b862bbe9ec) (cherry picked from commit 2807e69ec556660964441eb4e66a8ba03e731eb2) (cherry picked from commit b4749423ab68e23f05da48c2da8b53853eb7d5d6) (cherry picked from commit 24fcdf0d797c71a95989a5794c07a7131ccd54bb) commit 5d53a5eadb19d39af1afeaf9ba7752a3d3c2202a Author: Ruslan Fomkin Date: Fri Mar 18 21:26:03 2022 +0100 STAR-1176 Add custom replayer filter (#374) Extends the commit log replayer to provide custom replayer filter implementation. A test is added to see that a custom filter is used. It is port of CNDB-1285 and needed for CNBD. (cherry picked from commit 66ba75b553347553c6543989ddd381285b3b8fc0) (cherry picked from commit f6813d739a236411acf281c9ffd641bc00975962) (cherry picked from commit 82735e7d2b5575ab0f045e8bcfc507ab32f3db60) (cherry picked from commit 216ed5e94a89c7969d2984577492b6c306821a12) commit 88f12da2d2e44ae321612d9bafcc8229b1bcd468 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Fri Mar 18 20:24:26 2022 +0100 STAR-871: Fix TopologyChangeTest (#377) (cherry picked from commit 8807e26b20e86e8087739dafc473f0e64cbbc975) (cherry picked from commit 7c958c066c36ddcae4c1b83131b13d8ff7159d03) (cherry picked from commit 0575f709ab18d7b067a556ad0b67c2c2b6b939a9) (cherry picked from commit 5959c4d1a03cb34c497eee17d755a3158a889202) commit 5a4e121dac05f2671aaa94dabdfc6d52ebaf364b Author: Jakub Żytka Date: Fri Mar 18 10:42:53 2022 +0100 STAR 1174 (#375) * STAR-1174: use PartitionPosition interface instead of concrete DecoratedKey where appropriate * STAR-1174: sstable resolution added to Directories (cherry picked from commit 2c18c004ee82a808469ad22e85b32c0819d0682c) (cherry picked from commit c4aca4fb5e722b66863bd7b03f8a75088e5f3d4e) (cherry picked from commit 3a0228226fe65310a70b67a2a5be3331a1f86647) (cherry picked from commit 98d61fb7e54f1634729470d2830adaaf26bf2304) commit f1ab5b62e61f79200c33efb288b1640c01e8b615 Author: Ruslan Fomkin Date: Fri Mar 18 10:00:58 2022 +0100 Enable testUnwriteableFlushRecovery of CommitLogTest (#373) (cherry picked from commit f2838ce897e7f0f66053986d96aaa10664ea94cf) (cherry picked from commit 49c7355487036fa5820938249444d18583ce608a) (cherry picked from commit 78d51234ccebf260b3f18dbe9e04d3e2eb4e0ee2) (cherry picked from commit 437e8eea4dd8213bbbcc3ddba0ee464fd1ff1891) commit 70f7db5ae826d1f4f3bb40750f31a588a0ea8bac Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Fri Mar 18 06:24:30 2022 +0100 STAR-1181: Fix ScrubToolTest (#376) (cherry picked from commit ae72e761924d380a345037ecc1926cb6f8df11bd) (cherry picked from commit 8ee09df3410dc1c0130a75333d3f7cfb58fee0bf) (cherry picked from commit de9c731e3a3a3d049a2cbcd1f5256e5543e2a166) (cherry picked from commit fdbd1cf6fa25fc504771fbb24876a105a5823237) commit dde2ae96071b1638ccfa00a4b0ca81ee00b8029b Author: Piotr Kołaczkowski Date: Wed Mar 2 18:01:19 2022 +0100 STAR-1028: Add segmentRowIdOffset in PostingListRangeIterator The PrimaryKeyMap operates on sstable rowIds, not segment row ids. Sstable row ids are different from segment row ids, for all except the first segment. Therefore segment row ids must be translated to sstable rows ids before passing to the PrimaryKeyMap. (cherry picked from commit 750b10bfe60fd824ba2f66fe912da375459ff13e) (cherry picked from commit 85275903ba53ead4630482cbc4670cafb7f7afec) (cherry picked from commit 625d6a3bdda434a628fc27ea7e610b71517e6445) (+2 squashed commits) Squashed commits: [682e430d84e] STAR-1028: Fix (and simplify) test assertions Expected and actual were swapped, which resulted in misleading assertion message. (cherry picked from commit 5a019ae9b901e4fc674f3fb9af29417fb949b724) (cherry picked from commit 7cb7c8b21ef91c492999273cb8a44cbb09f59b58) (cherry picked from commit 40199b57de47284d9dc475e961a1926407655ab8) [88f2b60b96d] STAR-1028: Allow to disable index segment compaction Add new index option: enable_segment_compaction. (cherry picked from commit 1281cfc92de77b65ed93efee05b743114d1ba04e) (cherry picked from commit 7937bd13b5d589d207a2ea9ce7ccf95722e2c71d) (cherry picked from commit c57d15c56916331626d442116dd57819ad206dfe) (cherry picked from commit 60580c930c4f1b5e68eebb7df48db8f4c8b72eae) commit ddb8b68925a07f0e54aeaf16ef8f83e64e509db6 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Wed Mar 16 20:48:29 2022 +0100 STAR-1177: Fix accidentally removed keycache invalidation after compaction (#371) (cherry picked from commit 933d4313c19df5af607e44247e90e33fcbd7ab4d) (cherry picked from commit 00af9c5ec6bcc6e094879db02b2d9e90e26e84bd) (cherry picked from commit 88c60e96df347855c155c0abdfc24566916d8ec7) (cherry picked from commit ad2d31da4261561932d9e9e1fe6f646ee5f47342) commit b9e53831170589a67a759170200312d31f899b6a Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Wed Mar 16 12:32:51 2022 +0100 STAR-1170: Fix missed marking corrupted sstable as suspected (#372) ...during compaction when compression is enabled (cherry picked from commit ad57882ede925d4d426df5edc871532f106887c8) (cherry picked from commit 24dde5c6bc9ef05b4b4efc265c803b0e8c547d43) (cherry picked from commit 630b984d48622a7c7d1ffed709298f9d6e83c088) (cherry picked from commit 9d417dca74d078baa2867409fdc8b984c82d6023) commit 540a2f6805bae36a38db12f2afacad21b747dca6 Author: dan jatnieks Date: Tue Mar 15 11:27:35 2022 -0700 STAR-1175 Read path distributed tests for ReadCoordinationMetrics (#370) (cherry picked from commit 05b2fdf2c076e7e3ebffd435b1ffb39b0fd7a711) (cherry picked from commit 9942d1a463c9f33a1990cab72936215a62f334ed) (cherry picked from commit 9626472397cb96e236cef096906a81c28790c6fd) (cherry picked from commit d09e6c44c9c9bf47de25eaf361cbf7f43dce42e1) commit 7cf7a4e0b673d8803675580943f0b7e6d85acf6f Author: dan jatnieks Date: Mon Mar 14 14:17:24 2022 -0700 STAR-1151 Port ReadCoordinationMetrics (#364) Port ReadCoordinationMetrics to add metrics that count when the coordinator is either not a replica or a preferred replica for a read request. (cherry picked from commit 3967db9d0df6973a9520df18c4a04960346461ad) (cherry picked from commit 953297976a35914591ab547919ec7ba43fca0a41) (cherry picked from commit cd56ad2f3f797b8167d9891587e836352347700e) (cherry picked from commit 926c40391757ebef8cce65f17d8167bbaac64c62) commit 3efe6d7a86a9f0c15cb89f27316185bbe901d4de Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Mon Mar 14 18:48:29 2022 +0100 STAR-1171: Fix wrong overwrite file open options in PathUtils (#369) (cherry picked from commit 86180e68f0eba8629e8760cf905417d9ebbd87b9) (cherry picked from commit 5504876a6a7b7453e0f5d8fa4c2b072abbd4e6ab) (cherry picked from commit 056f59b803464b3082d77a23b38ac087657ed767) (cherry picked from commit 859e26d94077e2ee47fb0b486445e934390a0584) commit 761ea0c1cf87b45495cdc61594b5031eb2bdc13a Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Mon Mar 14 13:50:27 2022 +0100 STAR-1169: Fix KeyCacheCqlTest for big table format (#368) (cherry picked from commit 1454206f73b31f819621361f6e374a04986d6981) (cherry picked from commit 9af75e9a55e3996874bc541afbec19a8ae71cc68) (cherry picked from commit f8680140974104d1e6d9fbc24c67a7fe30c17429) (cherry picked from commit 9e0b5aef7ab5f18080fe7944edc47232878cb1a1) commit 7bd6c3c2833398a7752d4efc7685c44e161bb450 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Mon Mar 14 11:53:14 2022 +0100 STAR-1166: Fix LongCompactionsTest (#367) (cherry picked from commit e1f6c580733279089fc9115c6f20f121a672cf5d) (cherry picked from commit f038e107507f2bd580d802416c7f413aece53d92) (cherry picked from commit c667fab4d0db22792c3c1f07ee08eb35542aad94) (cherry picked from commit fd4e73973404bc602aa4aa6dd64db5d44bdd7008) commit 41d703b8f76bd32f1d8b6ad8c23a857882272371 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Fri Mar 11 17:08:35 2022 +0100 STAR-1159: Add system properties for defining initial RF of system distributed kss (#366) (cherry picked from commit 6ec8567b57326554948db7fcbaf928a73f827f01) (cherry picked from commit f80a21bbef3658c65d4c2002bb1c623aa3946c6e) (cherry picked from commit 74b992e263e8555769344e70d8412ae8aa1b2b4b) (cherry picked from commit 37fa90c9f718c3419219deef1df3a746bfec51fc) commit 91ed416f0331528635fea98cd501f79a677cb5ae Author: Ruslan Fomkin Date: Fri Mar 11 11:16:41 2022 +0100 STAR-1096 make TruncateStatement extendable (#338) Separates prepared TruncateStatement from parsed raw statement. Raw statement implementation is customised to use a provider for creating a TruncateStatement instance. The provider can be replaced through Cassandra properties, thus allowing to provide custom implementation of TruncateStatement. The customisation is needed and ised in CNDB. (cherry picked from commit 60f9a447e468add7e39c12be800854a7c455335b) (cherry picked from commit dcc63b978ed76135fc7a35a292a14752e0c5fb03) (cherry picked from commit 3da2210efcababaf7334217e293190322f161692) (cherry picked from commit fcfb159cab234a704dd9c8ac250cf9a5e8922906) commit af0b6233ed4febc3189f2ff7782e0b1e0a4fa259 Author: dan jatnieks Date: Wed Mar 9 17:04:06 2022 -0800 STAR-1150 (#362) Add clearCache to Roles and AuthenticatedUser; Return affected roles from IAuthorizer.revokeAllOn; Add IRoleManager.getRoleMemberOf (cherry picked from commit bc83cd92b6836e862e5aa98c18f785efcbc2558d) (cherry picked from commit d13899e33219e1b4f7c0fa9b35a59c1818f34afb) (cherry picked from commit 49646a9e27869b94e0ee8fbe3609439f0b55e3b4) (cherry picked from commit a0f37668478922846bc47840df0187903fc2b520) commit 25b40f0d9a45b635a401693c5b70db96ba581159 Author: Ruslan Fomkin Date: Thu Mar 3 10:26:53 2022 +0100 STAR-1160 allow truncateBlocking to get endpoints set (#363) Refactors truncateBlocking in StorageProxy to provide set of endpoints. The method is not static any more, so it can be tested in Cassandra extensions like CNDB. Needed for CNDB and is port of fixes for CNDB-1462 and CNDB-909. (cherry picked from commit 8c830e3c4ae282b638166acaad623d66a2f083a3) (cherry picked from commit 4a8048bb905fbc6d8d37061c1f5e7d344df6000d) (cherry picked from commit 7a025ef6ecdcc82a0c298aabbf350d9b81b80d7b) (cherry picked from commit 7335072e5ef0d32043fa805cdf2c6964c4055930) commit 9f999fb11fd04eb04cbd9728e11a389b180c14a1 Author: Jacek Lewandowski Date: Mon Mar 7 10:19:27 2022 +0100 STAR-782: Improves compaction performance This patch uses a "cursor" abstraction to remove a lot of the overhead in compaction. For key-value workloads this typically halves the CPU requirements of compaction. See the included "cursors.md" file for details. port of DB-4159 by Branimir Lambov; reviewed by Stefania Alborghetti and Aleksandr Sorokoumov with added: - Support for compacting to BIG format. - Refactors writers to remove repetition and extract the common core. - Fixes a problem where a separate static clustering was created for the byte[] accessor but code was using identity equality with Clustering.STATIC_CLUSTERING. - Cursor compaction is enabled for all strategies. - Column subsets are decoded more efficiently. - Disables key cache migration during compaction as it is a performance bottleneck. patch by Branimir Lambov; reviewed by Matt Fleming and Jacek Lewandowski (cherry picked from commit f61394c09c483d79fd5bc2fdcec16a378d6e31fd) (cherry picked from commit 41c35b5bdf5da96472d5b6d761d58d9190f8dc19) (cherry picked from commit 702dc7b6ea0786b525a133fb9e7c731fbfd47bdd) (cherry picked from commit 41ee610f8d18e55c0055a27cac147a29dd292f4f) STAR-782 Fix CompactionOverlappingSSTableTest from CASSANDRA-18507 STAR-782 Update comments in CompactionTask to explain changes made to the live set of sstables are seen by the iterators in the compaction controller and tracker. commit befd56e0b7c7b499c37062aa1e86a547296cb2e0 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Fri Feb 18 10:38:11 2022 +0100 STAR-1128: Let provide explicit mutation serializer (#355) (cherry picked from commit b4ccc1d4a270d41afc6f7129f374eb1261bbae4b) (cherry picked from commit 7b7b0779a32b295ba5b419393c0ed5c4462f088d) (cherry picked from commit c2ea26b06e011ecdbedaa2b3d34699455b64ed84) (cherry picked from commit a900a0d8fe0e085327d0273a1417a6f1cd874464) (cherry picked from commit 7e25f181af34ae63fa24381b005cb77ecbe09722) commit 9bd6555eb2afc4a54b62c4bbeb03039dffe234cf Author: Jacek Lewandowski Date: Thu Feb 17 14:16:47 2022 +0100 STAR-1134: Available space validation and properly resolving paths in CommitLog (cherry picked from commit 2b898fb6a6f55741f03b849ea5b8e038d7d29550) (cherry picked from commit 6bacc46895b6c611a81fa850f56ecd995d59590e) (cherry picked from commit 17903a8b8fb712f98a76438edfd54bb6664c13f4) (cherry picked from commit fcf80fb14053680896543607c5c0729888fc06aa) (cherry picked from commit 05902ea11680141db36f165f86e0185219fe1dbb) commit 7ab05cb4a59eaf743fd45e13c93046131643e416 Author: Jakub Żytka Date: Thu Feb 17 09:28:34 2022 +0100 STAR-1137: generate index files during prepareToCommit phase, not commit phase (#351) (cherry picked from commit dc5b54753f82cc670c539374e31f60f084f34ef6) (cherry picked from commit 092ae6b5de4df7a0fcf3d8b1cc5046c6a4926752) (cherry picked from commit f536582d8d5a48e769a5428e3bbac8312e6968e2) (cherry picked from commit 367a66055f63733fe560927e67091dfeb85546ef) (cherry picked from commit 2cae96676908b611ac75c58897952c8bbca46a9a) commit 896389a60fe4b5fa3d0802dddd4047247b79eb06 Author: dan jatnieks Date: Tue Feb 15 12:16:27 2022 -0800 STAR-1126 Get replication strategy using KeyspaceMetadata.createReplicationStrategy (#345) (cherry picked from commit c8ce8e5acc55b5b41ec8d0ea0645e04268caca72) (cherry picked from commit 7c288ce1daec8dba55f5f4bd4b4636a89b10e45d) (cherry picked from commit 25061b7abc074fbc22711466682d4c42daa9dbf6) (cherry picked from commit dbba1f131d009a26873deeb8a9209ec971b6212e) (cherry picked from commit 8defbd1de7cf2f8e7b76a2b5ea6515afbadd2df3) commit f2a54614ba6e1d12d916ef1ab70a9b726b4f2763 Author: Jakub Żytka Date: Mon Feb 14 20:52:54 2022 +0100 STAR-1132: dont fail listing on OBSOLETE files (#342) (cherry picked from commit dcb0231b72c1c5150ccf1d87f73dfa8dc53fb7fc) (cherry picked from commit a4668d9afd000268f9cf35bee442d5d0a12a713d) (cherry picked from commit 77c6cb1993c427c7f13fa0247a3f8535b61410bc) (cherry picked from commit 3ebc009db6412b8c421b59e09b1d07b3b73c262c) (cherry picked from commit bd19b10b551f37cede4c90cd733b077798afb8f5) commit 62b58b0a9f7a90dd24d1c4c0bebc1c9499b98b55 Author: Jakub Żytka Date: Mon Feb 14 08:53:27 2022 +0100 STAR-1120: make FileType assertion in Directories more useful (#340) (cherry picked from commit 79abd74276d54593c90f1ec1e2de0a2945e258b2) (cherry picked from commit 77acde06a8003d6dc5f0277043d1af0b3d5a5ab9) (cherry picked from commit 6d3a919b4004d94a4bbb94e14ab22317cc5a0811) (cherry picked from commit cd006abb9302b9f33ce8536e3890cdbb1fe1ec89) (cherry picked from commit e4f702ef39035301da544d605d4a682265f38ed0) commit 00e3a5073d657ce7c4bd4ec0fb4b734a60c63f8e Author: dan jatnieks Date: Sat Feb 12 15:39:38 2022 -0800 STAR-1103 Port DSE code for uncompressed, non-memory mapped commit log segments needed to support remote storage (#337) * STAR-1103 Port DSE code for uncompressed, non-memory mapped commit log segments needed to support remote storage (cherry picked from commit 746e2395afd775da2e580e70a053dc982f438373) (cherry picked from commit 1a4d918388a78ddcc8e86413c66d5add1b9ef859) (cherry picked from commit 105dfb30f5d4696adc00e29708cf543a5d5af8b0) (cherry picked from commit 9b6e80bb4f602b11a51736b9120e8a27ccb8f21d) (cherry picked from commit 21b873d80c57ad0ec2c36194b4ef41de34204b92) commit e2e9b684040f58864f5104131c5c5aa8c9241b78 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Fri Feb 11 21:34:14 2022 +0100 STAR-1061: Make it possible to override keyspace name in statements and results (#323) (cherry picked from commit 9374397caad20c32a800dd7937c21a05bb3c321b) (cherry picked from commit fab241af8cac2e2f5e0fab4cc4dce119170d12c0) (cherry picked from commit caba2262d42a96bd5108b21b66c9374b50ce9cf4) (cherry picked from commit 1164f2baf8b2d1d84980f90f50061586321fee30) (cherry picked from commit d1c489d05203cc814402fa371c91b31752a7a8e6) commit 906ff1d7779ce8f9148c15d0268f51d3621ebf1c Author: Ruslan Fomkin Date: Fri Feb 11 08:53:51 2022 +0100 STAR-1097 Store Raw CQL query inside statement (#333) Original CQL string is needed in the implementation of Query intercepter (STAR-1112). Converged Cassandra stored it in QueryHandler.Prepared, which is difficult to access. It seems that it is more straightforward to move storing CQL string into statements. This commit changes to store query strings in statements and adds an access method to the string to CQLStatement. (cherry picked from commit ab7fbd22bb4525a9df5e51ae1ff46a68e545c132) (cherry picked from commit af67f2bcb4c55a149b454ecac53cc1f60d66f5b2) (cherry picked from commit 3a52a9a174be3adae6c38a4792ad9eb759bed8a9) (cherry picked from commit 85c6a78c975aa97ebca9193cfac88cdf079fc393) (cherry picked from commit d21078359e6cb4ce786ced04456888c0f4d09ffd) commit 521bc160d694efc5b9b4f2a7646b8d64b9818905 Author: Zhao Yang Date: Mon Feb 7 13:45:34 2022 +0800 STAR-1048: skip remote sync task for push-repair which should use only local sync task (#330) (cherry picked from commit 326d847fb40812900ae5e83bf3059c3d0ed78ef5) (cherry picked from commit 2b3318204699280c619c82c2e140d0180ce71911) (cherry picked from commit 889eb68cee58a0aa28e23d1bc90f71d3f38045b0) (cherry picked from commit fc74b0d6555388d8d58ba48a5674adf212be96f0) (cherry picked from commit 641f86df51453e1bcba44deb0ff1e8a24f5d95b4) commit 5167183a8c579b90138de9d2fda686a6be69ce91 Author: dan jatnieks Date: Fri Feb 4 09:51:47 2022 -0800 STAR-1076 Add isIndexQueryable(String) (#328) (cherry picked from commit ad6aa9018544b34ac44e218b2e16425630e89e04) (cherry picked from commit b5c5ed04aa3ccba78cc207a2064cc1170500f593) (cherry picked from commit cda2ec6646632cda3bdc5f6b3bf4165d2c054b31) (cherry picked from commit e06b0f3eaa13e20e5a644456290e90fac7bb56ed) (cherry picked from commit 8cd5a448fe9de1b9d446dd533e052feb88661a3a) commit dbbf545feb20ff48223b9de3ae1d859862cb802c Author: Jakub Żytka Date: Fri Feb 4 10:56:07 2022 +0100 STAR-1074: make naming of DD API to set column_index*_size_in_kb consistent with getters (and common sense) (#325) * STAR-1074: make naming of DD API to set column_index_cache_size_in_kb consistent with getters (and common sense) * STAR-1074: make naming of DD API to set column_index_size_in_kb consistent with getters (and common sense) * STAR-1074: allow decoration of ChunkCache behavior * STAR-1074: ChunkCache position invalidation is executed via RebuffererFactory::invalidateIfCached (cherry picked from commit c4b491824f6ead42430feb901b77b54ed92abfab) (cherry picked from commit 83410a52a9c00b20ba31a4411a3b706cf98cc929) (cherry picked from commit 535336ac70f10f7844295798f37a58aed26e7f26) (cherry picked from commit 5fc99c763b6186d039f858be801f682bf702eebf) (cherry picked from commit 1226d243ad07daeb47d00c52f99198eb2ebd2de6) commit 0acf0e98b794cdc7de906e4f85609b46ab2973de Author: Ruslan Fomkin Date: Fri Feb 4 10:08:18 2022 +0100 STAR-1078 extend StorageHandler.ReloadReason for CNDB (#326) This patch updates StorageHandler.ReloadReason to reflect the fact that PERIODIC reloads will be replaced by reloads only when there is a change in etcd. (cherry picked from commit 954d71078a9f36843cc445fdc5a07a08a8b04cc0) (cherry picked from commit 389252b027ce25c1109bbcd103212b99e225ee84) (cherry picked from commit 8de6c71edde54fbfa06e9c044c036f2c841deafd) (cherry picked from commit feae6131cb9fef0f3ffd2f86d97cc38c4d133fdc) (cherry picked from commit e457019e8103ed3e9f99747b77eb5c4e50c6413c) commit c22ac21211b14d2bb43b171ba1efd89445fe60a9 Author: dan jatnieks Date: Wed Feb 2 12:13:32 2022 -0800 STAR-1069 Use JVMKiller interface in JVMStabilityInspector to allow other implementations. (#324) (cherry picked from commit e9a789a972a2510038a86f3e9f012e06fdcfeaa0) (cherry picked from commit 1546d87fa35e780bb0601abd047594d730e619bc) (cherry picked from commit 19fb3bc589cfa6d7c9acdaa791225206a8b80c82) (cherry picked from commit 1721286d2643dfb7e76b713ea7389cf9bb44d4a4) (cherry picked from commit 82051e45d0208cce2a4f5c8679de22ec3139764d) commit 36821d330857be4be56c9e217b0cec134a1ec0ac Author: Jakub Żytka Date: Tue Feb 1 20:07:08 2022 +0100 STAR-1065: add commit log position to truncation notification (#322) (cherry picked from commit 5f2c6efbf6b3e747ef9fa74ff05e95d36968ff89) (cherry picked from commit d1d1199e90c8ee26d8cc94348f5c1db0c3e2f112) (cherry picked from commit 34c0e2c5d24752c846d7e24509defaee01b8d1ef) (cherry picked from commit aab4bdd3faa59aa7df629673029d9ce72de2efbb) (cherry picked from commit 813702c446dc4bf073ad715a119a2683bbb7912c) commit 3a8ab2fde2f4d68819ec2da137695dc0da2fab73 Author: dan jatnieks Date: Mon Jan 31 10:34:59 2022 -0800 STAR-1056 Update FSError to store File instead of String value (#320) (cherry picked from commit c390f5f9694bac00d29814abb54be4cf940f347b) (cherry picked from commit ff24c10a09500674c04f8e823e3dd89b63a66258) (cherry picked from commit bf506178ab2df565df85c6a038e87d0090b5aad6) (cherry picked from commit ea192158a7dd215ae20352c6f96f2c8de463eb80) (cherry picked from commit 02a92fdff0cf9aadb5707be8561ff253b9dc1ce3) commit a732ef745e325bc6938b5ddc46e114914d1b3715 Author: dan jatnieks Date: Thu Jan 27 10:15:48 2022 -0800 STAR-1042 Adds methods Descriptor.filenamePart and TableId.tableNameAndIdFromFilename needed by cndb (#317) * STAR-1042 Adds methods Descriptor.filenamePart and TableId.tableNameAndIdFromFilename needed by cndb * Test coverage for Descriptor.filenamePart method (cherry picked from commit 2dd6075547f746569959a91afd53cddf7fc1e106) (cherry picked from commit f4d208ea75e9fdd00d6986725b30cb2174a76090) (cherry picked from commit b2ff0a0268827b8242db1dcce5a7e6ce283e0ddd) (cherry picked from commit ec596ab7457364f1ce97a0a86a7e25f17b162cd6) (cherry picked from commit 042efc316e3f4e42b3059bbdade3a83b40b930c9) commit c21c71ccc7af18d33ddaf9164e2882abaf4b03ed Author: Jakub Żytka Date: Wed Jan 26 13:49:04 2022 +0100 STAR-984: WriteOptions (#313) * STAR-984: WriteOptions - enum for describing required properties of a given write (cherry picked from commit fe135e076eea213275e25b079d02180ccedb14f1) (cherry picked from commit e69e0a0d0c247c9063ef6a83d80f31ac04decc8f) (cherry picked from commit 5a1b60ed2ce799ba029fcb425a652bc44d06abca) (cherry picked from commit 680a27433ce0f0da4e40d3450a483b4660e16a95) (cherry picked from commit 318321f93f1a4c83897207309132aa9b88cd5adb) commit 69bd292dfa900b90725b389527f408d8ba2d6b9d Author: Ruslan Fomkin Date: Mon Jan 24 10:03:19 2022 +0100 STAR-1036 Filter keyspaces to update in PendingRangeCalculatorService (#315) This commit allows to avoid updating all keyspaces and only update the specified in the filter. This can be usefull when keyspaces are used for different tenants. (cherry picked from commit c20ec17f1338bf58e85d5ff90e2c4c2e6e2e1692) (cherry picked from commit 321f729f41e007c5d44fc70fe3beb705402918e6) (cherry picked from commit 20398103f898105ad261c44b43f735b1362897d1) (cherry picked from commit 29059bc3e40361a6a324f0705762573f72523dc2) (cherry picked from commit 3562c76671e9d54b2fd7949697fe80293e9c7e67) commit 3bbd00a40eb60b3f9c1e27f49bc8fd11ebc63ceb Author: Ruslan Fomkin Date: Thu Jan 20 10:42:38 2022 +0100 STAR-1032 Add hook to allow token metadata per keyspace (#312) This commit extends TokenMetadataProvider interface to allow storing TokenMetadata objects per keyspace. This is needed for and used in CNDB. (cherry picked from commit 7a3ad018aa2b9d4ae855f1b31d2eae5b67ad0598) (cherry picked from commit 139c2ff22b3006067287cdff4e19b621ab7041d6) (cherry picked from commit 350b71fe85b1e33a6b52ecfb0a8a099857eb43fc) (cherry picked from commit c50c1b34c26bff775bd115fea7e3becab8bea0de) (cherry picked from commit 7bd24a92b40ab0b8d8f14763b321355a9e28b776) commit 487fb581368cb112d61724032a35cf0a31f07aff Author: Zhao Yang Date: Thu Jan 20 16:36:33 2022 +0800 STAR 993: Add abstractions to run repair across regions (#311) STAR-993: add result to RepairRunnable STAR-993: Add push repair to Repair STAR-993: make repair prepare message timeout configurable STAR-993: add repair progress reporter to hide SystemDistributedKeyspace STAR-993: add parent repair session listener STAR-993: add isOfflineService to repair option STAR-993: Add API to check sstable global tidy reference (cherry picked from commit 8bd82db908d3b3fe64d0439607e30f7109d79ce5) (cherry picked from commit e857671e58b21b81630c4c77b5ed0682ab98ab31) (cherry picked from commit d82c8ee4c9e75338a1f7fc3b718f5ef8b2a637fc) (cherry picked from commit 79a7bbad45deab6434bebb648182c611b2b8e9e8) (cherry picked from commit fa7971b5dd4af2525fc7cc8d3fcd0d92491efa06) commit 85efd058a8b06ac87f24dbb1b8f96f4a8931c20e Author: Ruslan Fomkin Date: Mon Jan 17 13:30:36 2022 +0100 STAR-1015 add new CL error policy to reject writes on sync errors (#307) Adds fail_writes policy to commit log failure policies, which rejects writes on sync errors. The policy is tested using byteman to inject sync errors. (cherry picked from commit 6b785ca61d3afac69d45a54579d7a3310ec621b2) (cherry picked from commit 92be95841c5be95c0e203b10472b7ce64b25d98f) (cherry picked from commit 7de1a72ec45d8e5986395e6c745c53c4437db842) (cherry picked from commit f6925cb75f76d3bfb936ff16deb46c7e8ff73689) (cherry picked from commit 13258ff69e8afcf9e036997cfdf5ade776162336) commit 3fbef30aa296249a5d3daadf02ed6c42cc5fcb18 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Fri Jan 14 12:40:07 2022 +0100 STAR-893: apply some performance improvements to Nodes impl (#292) * STAR-893: Use read-only interface instead of preventive duplication * STAR-893: Simplify access to commonly used fields (cherry picked from commit be31ce7d8050c140b48c4d573f48d4e742a9c40a) (cherry picked from commit 2b352645b61f74194bd1e226cd963e3d66ac905f) (cherry picked from commit 81c5ee87f4bdd878774d29f8225f15d11d56664d) (cherry picked from commit 101f4158d2548aced6e4130762fd956625ba5538) (cherry picked from commit 61fe029e69f687d6f1c39a9c5e5d6c77bfe9793e) commit 1b299a491d82d54d0918be55307f0f3d051314bb Author: Zhao Yang Date: Thu Jan 13 15:51:10 2022 +0800 STAR-992: add APIs for replayer to discard flushed remote sstables (#310) (cherry picked from commit 343a71733bcf39df209e4008c6ffb8b614567542) (cherry picked from commit 4c558b8cb0e5d45841838c8e8181733da5cd47f5) (cherry picked from commit e9927b0b7b3d3b7e6503de05fa8326ebd75a8b7d) (cherry picked from commit b8bca6799d62317d64c0cac694f213c7e4d78b7f) (cherry picked from commit a822bcb26ad1853bcba100789a739b893ba79d2d) commit 3c346ffeefcb3b7cb8a5a8b581fcc947c3403386 Author: Jacek Lewandowski Date: Wed Dec 1 10:31:09 2021 +0100 STAR-996: Implement node information component The stores all the information in memory guaranteeing proper behaviour in case of concurrent access. The changes are serialized and stored in system tables (as it was so far). (cherry picked from commit db0d3ccd22c72c534ccba8c8d244e2422aa3595a) (cherry picked from commit b1dc27ded8b644d461756ee53e53f384f72a7a9d) (cherry picked from commit 8fb840bf0f03b316837452884e35f3938aec82e9) (cherry picked from commit 6f87736a3e262d67d1c633fcd2197e1f3867ab11) (cherry picked from commit a7a829641771967dea6e8040159f0777b732ce78) commit 497cf1504e872727f96cde0bd96530bd26c593a3 Author: Jakub Żytka Date: Tue Jan 11 13:32:57 2022 +0100 STAR-983: add ReloadReason.SSTABLES_CHANGED (#302) (cherry picked from commit 53f6e5080f9758331cf8e0937faa181a65cf5aeb) (cherry picked from commit 46adecb618d41135cef8c15bf5466a5149d1307c) (cherry picked from commit 97cdd8abb6704fd940a8ce007ea1e0e343c28462) (cherry picked from commit 451232227de23884fd396ee6afce75d7c9b759a7) (cherry picked from commit ff4d83a975ab710a11303d37dfa4081f8381ec35) commit f4b605fea88111dd3df60ae3e57751288cab2835 Author: Jakub Żytka Date: Tue Jan 11 10:27:58 2022 +0100 STAR-1010: port CNDB-3312 - add micrometer metrics for key cache and counter cache; add error log when failling to acquire counter lock (#305) * STAR-1010: port CNDB-3312 - add micrometer metrics for key cache and counter cache; add error log when failling to acquire counter lock (cherry picked from commit 8eac69ea5387726af6e4566bb7041c2a56cafd79) (cherry picked from commit d7765b2082c93fbb20a6e0686e98e2b747b31084) (cherry picked from commit b0a6cd8b5610046912c71db25e25044686d56d18) (cherry picked from commit 3554e47f6bac177ac6af3b9d0b97208e6f267e07) (cherry picked from commit b57b9cc21685f2c256ff768ed59015c273e2b260) commit 0860cebdf8b2f5454add98392ff5cc9033e416c3 Author: Zhao Yang Date: Tue Jan 11 13:39:38 2022 +0800 STAR-1016: build index for remote reload sstables (#309) (cherry picked from commit 01e645d2795a2dcf296d37d240135a2817cf320f) (cherry picked from commit aca61131e3cca399133fca6f7513598222f21b44) (cherry picked from commit d89a44fdfe1de363224df7360fe48c6e1816f154) (cherry picked from commit 3eb9bf6dbc7d59467d4aaef1a81aeab557ecbe6f) (cherry picked from commit e23cb56019386b00129f78bae75d6535d09c8a30) commit 9ce6ee987105cf0b27e9427cf390cf1070228c21 Author: Zhao Yang Date: Tue Jan 11 11:22:41 2022 +0800 STAR-987: make DD#setConfig public (#297) STAR-987: add OBSOLETE FileType STAR-987: add Descriptor.validFilename STAR-987: remove AbstractLogTransaction#type (cherry picked from commit afd7aa18d1afa594c9649b2c318371782f18e50a) (cherry picked from commit 2e313c95ee7a714148d38fb1be62fcc4101bccfb) (cherry picked from commit 875ee83338507a890113fa83650de0a72a8bf8da) (cherry picked from commit 7237e9d04b1c1ebb56ed98593cc82c2f73fc3ac4) (cherry picked from commit cec28dc8484396d23901a2da335e2f92edb5f01f) commit e8fbe2666e6d3b9e6bfc893a054d14f737ea09a7 Author: Zhao Yang Date: Mon Jan 10 21:17:32 2022 +0800 STAR-1017: add operation type to SSTableAddedNotification (#308) (cherry picked from commit 300e9f0b61b8aef5b27262d7cec118d478159b5c) (cherry picked from commit 2bae0cea4a402f02683641702542031b890d6427) (cherry picked from commit 163d16459ca5f683743e75e6ff218d9520ba2543) (cherry picked from commit 2036425e4f7728c5881359521bd92596a374efee) (cherry picked from commit 54494c13276b7a180669ef0ed73c15daeb439c2d) commit 60bcebf16b904548ba0d24a855bf3c8428dcf210 Author: Jakub Zytka Date: Tue Jan 4 10:45:28 2022 +0100 STAR-1007: add recursive delete which ignores IOExceptions (cherry picked from commit 9e6032c7337bc69e4303b2e823a68e8f42b8c595) (cherry picked from commit 65a28442412572a550d705b3d5a787e90cd69bb8) (cherry picked from commit 36ebe3fc8354b7547e11b00abce708008c886ee0) (cherry picked from commit 6114c261715d16c4b236134752defc1211b439a0) (cherry picked from commit 4c11f488d3698209a529d9530d63f88d4ae0e601) commit 67b51138b6892670399e11ce7b8c672095b666ce Author: Jakub Zytka Date: Tue Dec 14 16:39:40 2021 +0100 STAR-979: add cndd-related OperationTypes removed unused OperationType.DOWNGRADE_SSTABLES; changed OperationType.UNKNOWN to relevant ones reference cndb-only code in a test (cherry picked from commit 5b3b106257f2884ce457a1a3b7641af4a35270b7) (cherry picked from commit 343e1da916288130225929f869b1f399cfedad37) (cherry picked from commit 855ad55ae7c658536e5a7eee793ff9e90abd55ae) (cherry picked from commit 79b85fc634cdf92ee8d79dc19b3364dfe5f5f90c) (cherry picked from commit 55de513de86984540dc7acb5046e4ce6a34806f3) (cherry picked from commit 681abd9b164fefde84fe1a87b534594ab751ee2d) commit 302aa4002dcc45b68301ef33eb781e37fc3b4999 Author: dan jatnieks Date: Tue Dec 21 22:15:53 2021 -0800 STAR-946 Make methods to configure and reset DatabaseDescriptor public (#294) * Make setConfig public * Port CNDB-2562 - add method to reset DatabaseDescriptor from CNDB tests. This is an unsafe method that allows CNDB tests to share the same JVM. Co-authored-by: Stefania Alborghetti (cherry picked from commit a058a79070abfc5cf4d1afb69c8fdd3671e85161) (cherry picked from commit ee578f9b2a5a28ba7b0b29e3249bd801ce4f0e7a) (cherry picked from commit 723b266b8f6f7e4a236e3bef99ec907856c06884) (cherry picked from commit fa2266295cf08e534b159fc76cc05e4f45e68d5d) (cherry picked from commit a8fb7bd0109ec0ce2560ce7e63c742c642e38b57) commit fc8c44beb7156e6e47e6b8327cb3e46302fa2bfd Author: Mike Adamson Date: Tue Dec 21 16:15:43 2021 +0000 STAR-406: End-to-end Row Awareness for SAI (#97) * STAR-158: Add support for multiple SAI on-disk formats (#238) * STAR-901 Trie and Lucene style terms point id lookup * STAR-906: Row-awareness for SAI (#278) Co-authored-by: Zhao Yang Co-authored-by: Piotr Kołaczkowski Co-authored-by: Jason Rutherglen (cherry picked from commit 8d9935f8df74c7be3a09d87ddd8fd8111bd36212) (cherry picked from commit 854b299cf9d9676d32776151f66f44684c99d817) (cherry picked from commit d895ae6b4a6584a807faf041d0e1a8d9dc5e259c) (cherry picked from commit 339548767982a97df276c3971a264f6f8d472ce9) (cherry picked from commit c407c2eec2b49a82cf8993ff9ec900cf2b3fbbd8) commit 74363fc623b0fb87486980f56054873072580a28 Author: dan jatnieks Date: Mon Dec 20 17:31:21 2021 -0800 STAR-955 Bump jackson to version 2.11.4 (#299) * STAR-955 Bump jackson to version 2.11.4 (cherry picked from commit 178f11a0fbeb0e1d436658621170f375aa758a75) (cherry picked from commit d5fdac0d72f98cc246391e0bf1eb5e6cf0411e3f) (cherry picked from commit d68fc1b466e16077d316677233f0554723081341) (cherry picked from commit ba5b21b125ab52f58e8696f42c3c96b120a726eb) (cherry picked from commit daba3cdc177bc63ee973c45b7beea91a50fe9652) commit 8e27652aa1c40dd3dfb0386d1e8a2c06c65bb312 Author: dan jatnieks Date: Fri Dec 17 14:45:07 2021 -0800 STAR-986: Fix compilation issues in DataDaemon class (#296) * STAR-986 Add ColumnFamilyStore.getStorageHandler method for access to the storage handler * STAR-986 Port DatabaseDescriptor.enableMemtableAndCommitLog flag. * STAR-986 Ports ability for StorageService drain to stop mutation stages after compactions and flushing due to mutations still being generated and install a barrier in each keyspace preventing any more mutations after that. (cherry picked from commit fdece72ae69c72bd35e11f2fe9c5e4ceda0fe6ca) (cherry picked from commit 9fe96bf5acdaf902c91d5974e517543009e6103b) (cherry picked from commit 93828e5136b9a330885031194e74c61ef331a904) (cherry picked from commit f60906948be1340d8151229e79f9e8d1b67eda98) (cherry picked from commit f7824d83132f128e2c36fe2a24f683e667e1dffa) commit 237b00a8762870bd6b9e03d1eef51ddd8cc204dc Author: Jacek Lewandowski Date: Thu Dec 9 10:33:38 2021 +0100 STAR-959: Add preferred local address to the snitch (cherry picked from commit 37223625c273db347315872bea6a3e2d266db3ed) (cherry picked from commit 935e3dbf42b92bb380a677a0a9a066aa27415c0c) (cherry picked from commit daefc9197bff49c751d8101488fc9cbbbf4f4ef5) (cherry picked from commit a2685c709c2853d59b2130901ea2f679ac8e136b) (cherry picked from commit 3df7f16ec183a7de778a510cc1fc44947b20976b) commit 2458b4bfee31de1110450c82293817e0ed955655 Author: Ruslan Fomkin Date: Tue Nov 30 09:02:18 2021 +0100 STAR-934 extend native library API for serverless (#290) Extends the API of NativeLibrary to create a directory by providing the path as string, so specialized implementations of a file system don't to do additional conversion into Cassandra File, which is then converted into string representation. (cherry picked from commit 3ba1a169e1250459ebd429f802ef8d58429d7aba) (cherry picked from commit aff175bd3bf72b32623c72aeb518ef09bb616b66) (cherry picked from commit f93bb4810535087a6170b4a2659d470d1ccc4c4c) (cherry picked from commit 83a4da21385464d150e2029a3edfbcc3f2bacfd4) (cherry picked from commit 58c3e22db02d9f3f2eeb8fd90c634d8a9b73753f) commit 5eb4fc8d48cf7d99581514be42e5f5c219ad5342 Author: dan jatnieks Date: Thu Nov 25 13:12:28 2021 -0800 STAR-933 Improve support for remote file systems (#288) Add methods to PathUtils: * deleteContent method that recursively deletes the contents of a directory, leaving the directory empty; * listPaths methods to list all the paths in a directory, optionally using a provided filter. Add method to Descriptor: * validFilenameWithComponent to return the Component from an sstable file name (cherry picked from commit 4f1c86b26b6a1b409c986de305b261e9b49c26c6) (cherry picked from commit ef840e5c148b5eac45ec17e62e46e55fdb044cb5) (cherry picked from commit dc75ab607f5a4f7689415ebf187f1aeb6c244c24) (cherry picked from commit a5a4794e98fb18e10a4a7f7eb1afc1318a6f8f24) (cherry picked from commit 1570f199c5362ecc1fbae62e2d622fa9e10cb4a5) commit 9bcec888abdb120fdc71f297e231dc822085a188 Author: dan jatnieks Date: Thu Nov 18 14:09:10 2021 -0800 STAR-912 Add InternalRequestExecutionException (#285) (cherry picked from commit 41cb66e534a0c3a2f135e9566b60a9f9173f0cee) (cherry picked from commit 8b71940f8b11b0ec3305b15e0b25f132eb49028e) (cherry picked from commit 19d2ecbaa0e3eafc7b827d2a6c9b0a074d632c67) (cherry picked from commit b3cc478962702ea6e705b0f5ebab0e3a93add9fc) (cherry picked from commit 1568f867d0d242e3d2afabd92672a536061cc6c7) commit 69d7818ccc3a6deaad8083f9549b68eb13f82990 Author: dan jatnieks Date: Fri Nov 12 16:43:12 2021 -0800 STAR-913 Add methods need by cndb to MemoryUtil (#283) (cherry picked from commit 7d5184da5cc464c078a1c6ff285b8056f4d1f08f) (cherry picked from commit 54627bd522d0d6551c4304f8d55278441bfb1c42) (cherry picked from commit 5322b772caa85f40e00fad47a6f4cdde4084a06d) (cherry picked from commit b060159ca905843de1d87b36f0f15f3dfe35f828) (cherry picked from commit 4aa1f1c550f946b6bac491aa5d131977d138a7e3) commit ad3ee02e41cab1089f6e4ad088fff814ca95556c Author: Zhao Yang Date: Tue Nov 9 17:17:03 2021 +0800 CNDB-1707: Port remote file system abstractions to Stargazer (#273) * LogTransaction: add ILogTransactionsFactory to provide custom log transaction * UCS: Port CNDB-2134 to disable shards on UCS L0 * UCS: add CompactionAggregatePrioritizer to prioritize sstables based on remote file cache * NativeLibrary: Add INativeLibrary interface to provide custom implementation * SSTableWatcher: to discover custom component before opening sstables * StorageProvider: support custom file system and change Descriptor to use URI * StorageFeatureFlags: disable features that are not supported by custom file system * StorageHandler: to reload sstable from custom file system (cherry picked from commit e27ee69bb404ba4c21b65d2aef157ac54425f2ef) (cherry picked from commit e98d05a294ad2d62d1b46dad081b5f60a16b7064) (cherry picked from commit 493bf4cb1236f6cdf819e8acbdbcf1f04d0573c5) (cherry picked from commit 2932216864ade578d487420efa88ee17ac01a0ec) (cherry picked from commit da98fddb2eccf91b389dd4a21976e9f81ed7b764) commit eb82c55d4d3fa76185fe6bf7b3073744e95767fb Author: dan jatnieks Date: Mon Nov 8 16:55:40 2021 -0800 STAR-909 Port DynamicSnitchSeverityProvider from bdp/cndb (#276) * STAR-909 Port DynamicSnitchSeverityProvider from bdp/cndb Co-authored-by: Zhao Yang (cherry picked from commit 150e4329d52291d2fb4d9e87a2e86e41667b5469) (cherry picked from commit 21a4dd38fa9bc667954b4522cc1e56b231ad2567) (cherry picked from commit 70b10013160034e93ecaaa21ad270e53cebf5be8) (cherry picked from commit e2f01d3ac3c43b5a464c5a8024b536f0a944fed7) (cherry picked from commit 5765292fd4e3405ffbeca5c63c9db9ba7d958fc3) commit 07cd5c9dfdcb44141d7b70a475864b01fad02fc7 Author: dan jatnieks Date: Fri Nov 5 13:50:07 2021 -0700 STAR-891 Add pluggable disk/commit log error handlers and shutdown hooks (#271) (cherry picked from commit c6cd4ee6673f9b43f7de7630e7669a873f4bf01c) (cherry picked from commit fb5c241b5bf22ae4683152b82a04605f925ef108) (cherry picked from commit f7f8e0a3adfcc09dcf365a6d452d9bbdbe6c76a9) (cherry picked from commit 0ff7339d9da4a64994c9d92ff89832463ffbcf55) (cherry picked from commit e7ca94c40ddd904ca50bb1f734a815b6a7ec620e) STAR-891 Fix conflict in DiskFailurePolicyTest with CASSANDRA-18294 commit 609cb2d1bf9cb6558a13ecd2c7ab80fa45ae622a Author: Ruslan Fomkin Date: Thu Nov 4 11:59:36 2021 +0100 STAR-888 extend decaying histogram's snapshot API (#279) Makes Snapshot class of DecayingEstimatedHistogramReservoir public, so its API is accessible by external components such as CNDB. Adds public getter to retrieve the array of bucket offsets. (cherry picked from commit e57753c4059c23a9082c17ca4e890d6228f0a121) (cherry picked from commit 572fa866c4c928a671856a9b83b7098c7fe84929) (cherry picked from commit e91f1f6443b8a98d84915055c936a62d3b2415a0) (cherry picked from commit 3442e7b5d2f4118aee83e8c51c20086de29b4106) (cherry picked from commit 279b209767630faa69760918ade0a4a8bf69eae7) commit 5d20b490c3a9ad42af861fe1563659c6c2826ec6 Author: dan jatnieks Date: Fri Oct 29 11:51:12 2021 -0700 STAR-894 Port Mockable FS from OSS (#269) * STAR-894 Port [CASSANDRA-16926] CEP-10 Phase 1: Mockable Filesystem Co-authored-by: Benedict Elliott Smith Co-authored-by: Aleksey Yeschenko (cherry picked from commit 477fda837f0c1aa79344a31546377f0f3b1a47ba) (cherry picked from commit c459c65e91f218fc43cb1a0720ff39ed9757c7f7) (cherry picked from commit 545c778afde34bd96806affc37157c224e0ec1a3) (cherry picked from commit 7ae967f2f02e831f03c9bcfbe8b705b6c6261f50) (cherry picked from commit 3528cd9b6a2dc5bc28a6ff8140ff132273b3a6b8) STAR-894 Fix MockSchema missing import for DeletionTime (cherry picked from commit 3f353a1563916929013524885c44e97116e5c1bb) STAR-894 Fix BinLog conflicts with CASSANDRA-17933 (cherry picked from commit e9aa46de6632f2320b54a38c176bd5d6cdb6cfa3) commit 3ae6763a0acbc9d109820cd624706fa347680825 Author: Jakub Zytka Date: Fri Oct 8 15:31:08 2021 +0200 STAR-886: port pluggable guardrails implementations (cherry picked from commit aff4ab00de2295fa07253d95d0e133e700879c1e) (cherry picked from commit d3f271c032655aca345b1816fd23dc57642211c3) (cherry picked from commit 6a60617ce67575addd920e72e1173e32b91c6a79) (cherry picked from commit 9bde42042699295bc7df1eda24c2c264a0d6a688) (cherry picked from commit f565905584f85baa9a64439dfafc39a7c186bb14) commit 662f1812e72bebbc08252a30b1276b549459c134 Author: Jacek Lewandowski Date: Wed Mar 30 07:48:23 2022 +0200 STAR-790: Adjust schema updates (cherry picked from commit e0982c6a84c3cc07b70f06e78d2cb0e77b777ae3) (cherry picked from commit 6171c0da0040d2325f365df5b248b03af773b2a0) (cherry picked from commit c8299f2cdd4c1dedadc5f24dfa6c7ab353926f83) (cherry picked from commit 2149776827f04b35f54080a99a76057f874c6c72) commit 1605a5a8c2da238f02424e1314195ef1a6f8cf99 Author: Branimir Lambov Date: Fri Apr 2 12:57:19 2021 +0300 STAR-508: Switch Trie to Cursor representation This replaces the Node-based walks and transformations. The result is drastically less intermediate object creation, improved performance and somewhat simpler code at the expense of the concept being a little harder to understand initially. Adds further documentation and expands tests for sliced tries. (cherry picked from commit 2b3c4c50d65a7ae93f0ee9bc9848064ff03733d2) (cherry picked from commit 43c520617fe46b92d4cba58eae8365777082b5bb) (cherry picked from commit 40dcfd51aec524dfbfad23c0cf60472a09bae4ce) (cherry picked from commit d3436c4d95860a60ff34e5936fd6eb2060946e7e) (cherry picked from commit 96d876b312fad9081ff4bce1a84db5f47d5b6806) commit 2b0aaca4e3fdb592ef5b25964c6a3693fa1911e2 Author: Branimir Lambov Date: Thu Sep 9 10:40:46 2021 +0300 STAR-791: Introduce SSTable and CFS abstractions for compaction The main objective of this refactoring is to enable compaction strategies to operate on a lean abstraction of an sstable and the compaction space instead of the full-blown open SSTableReader and ColumnFamilyStore. The compaction process itself must still operate on SSTableReaders which provide the mechanisms for reading the data; switching between the two representations is done when compaction signals it is ready to start compaction on a set of sstables via the realm's tryModify method. Most files in the compaction package have been changed to rely solely on CompactionSSTable and CompactionRealm, with the exception of CompactionManager and BackgroundCompactionsRunner, which are part of the CFS implementation. Also does some small fixes and simplifications identified during the refactoring: - Fixes bloom filter size in Upgrader calculated for splitting to compaction strategy's sstable size limit while files weren't actually split. - Stops checking an sstable's bloom filter if its minTimestamp is already above the current min for purge functions. - Some collection construction/processing simplifications. - Breaks up compaction -> CFS -> compaction reference cycles. - Refactors some method to lower their complexity as requested by sonarcloud. - Changes some remaining ...LatencyPerKb names to ...TimePerKb. (cherry picked from commit 943ae99a14950006406b2f77f314e23027f200fe) (cherry picked from commit e0fd645ab0edf2d35adf88cfb6fd15beaf4432e9) (cherry picked from commit 65797dbaf642bcddc8c7250d6ee52fe03c0a401d) (cherry picked from commit 48e1cec32ed8d69b23ad63dae83c52ae6fcea531) (cherry picked from commit 6e4b5bd638aa2d8257a8ecfea620901cfe9be234) STAR-791 Fix conflict in TimeWindowCompactionStrategy with CASSANDRA-17970 (cherry picked from commit f84e8040ccdccaa88758e8f489d827e4131c5761) STAR-791 Fix conflict in CompactionController with CASSANDRA-18118 commit cec9dbd9ddcc6e464d9f1729d6998d93bd7c2e93 Author: Matt Fleming Date: Fri Oct 8 20:13:35 2021 +0100 STAR-887: Port Request-Based Native Transport Rate-Limiting (CASSANDRA-16663) (#262) Ported from OSS commit d220d24994400d4342f5281f1a51514a6ae8c2fd. (cherry picked from commit b762e3cf353cdbad5e77838a4e5922e328d7b10e) (cherry picked from commit 34374c13d620e352da25e35147d5c1fcf15c997e) (cherry picked from commit b9046be5364fef3fbdaa47006ba311d9fd223e29) (cherry picked from commit f4bba1307f45f937e2ad32e450214e98ec05ab95) (cherry picked from commit dbe509498b6902671a7e70646ece02fd9a6deee7) commit 72712f04daab1e61568ba9ee5c223b72d9efb9e7 Author: Jaroslaw Grabowski Date: Mon Sep 27 15:49:01 2021 +0200 STAR-882 port BTree reduce function (#256) (cherry picked from commit 7373d2f73c9974f0aac1f7c46878d01df9f98db3) (cherry picked from commit 41342106f381f0b26d3ff71a5210934b5047b0e8) (cherry picked from commit 2c9e4a3d0465097e07810270a6b140aae699773f) (cherry picked from commit 2088ad4a88224c904baee4bfa3c1f8a8aee5f6f6) (cherry picked from commit 314ab1ffa10e32d6074d2500df3dc74abf0c5b3d) commit 67b9707847c38e9f696a4d6daae87b49993addc5 Author: Matt Fleming Date: Fri Sep 24 13:13:15 2021 +0100 STAR-881: Fix FailingRepairTest after making Verb an object (#257) FailingRepairTest uses serialization to pass Verbs back and forth between the nodes during the test. Unforuntately, Verbs aren't serializable anymore because they're no longer enums and this broke the test. Instead of passing a verb around, pass the verb Id and lookup the verb inside of the test method. (cherry picked from commit 47d071970b13b42c8f9f4a5051bd51043000bfe6) (cherry picked from commit 6ed9a947ef46297ce1318f98fb10a6f341bdb12c) (cherry picked from commit eaed731c7673c689244a34b2c9e2894670655fb7) (cherry picked from commit 4432e3cee0ee2cb4d38444e8242166574174e8ac) (+1 squashed commit) Squashed commits: [5d65744ea5b] STAR-845: Port MessagingService implementation from DSE (#254) Add support for: - Registering new verbs at runtime - Decorating existing verb handlers with another method - Running a callback after the MessagingService sends a message (cherry picked from commit c7d0ba0cc1d457786b971a0b901c90c773af476d) (cherry picked from commit e8e13eda7a3ca731053a4125814fb3cbef9f45eb) (cherry picked from commit 964892a31c749e00446632cc2f5225d2aa5dc6ba) (cherry picked from commit 1968d3cadfac40083c0a1381f770e0ef0888a8bf) (cherry picked from commit 012cd010741d5e3ebfdf93ca40ab743543e51571) commit 5ce23cf15e9a00dfc69d61de055fc905d2cc4ca5 Author: Jaroslaw Grabowski Date: Thu Sep 23 15:25:17 2021 +0200 STAR-844 Read/Write hooks (#250) This is a port of - https://github.com/riptano/bdp/commit/b6f0a18cb832c62f05cdcbd9cdcc2923f2fa727f - https://github.com/riptano/bdp/pull/19468 The first change set introduces QueryInfoTracker (QIT) interface and hooks it to StorageProxy. The second adds ClientState to the interface. The original QIT utilizes ReadReconciliationObserver in the ReadTracker paths. Only onRow, onPartition and queried callbacks are utilized by CNDB and thus only these methods are ported to Converged Cassandra (CC). The callbacks are a bit different tho: - The callback methods are added directly to ReadTracker as CC doesn't have ReadReconciliationObserver. The class was added as a part of NodeSync effort and it is rather superfluous. Porting the whole class would add unnecessary complexity. Adding the required methods directly to the ReadTracker makes the interface cleaner and easier to understand. - CC operates on ReplicaPlans instead of plain host lists, that is why queried was changed to onReplicaPlan. (cherry picked from commit c32e91ff0fcb06bc4b7637670627c65dfa27e21c) (cherry picked from commit 093bc63c61e98ce68095f0eda368bbe21eca506c) (cherry picked from commit efdfa1f49cfe9610211964409c2771b30f9059d9) (cherry picked from commit 2ab888d2ad41267710cc35d0154d2b5d4cf1b51c) (cherry picked from commit 546bc4a7b5603e8de6c621f047055f0713e61d82) STAR-844 Fix rebase conflict in ParamType with CASSANDRA-17981 (cherry picked from commit 12cd994fbecc2066352cd95f59f56a515f3a7a8e) commit 022df3a05c2159e5d159559abda4908a75f47cd5 Author: dan jatnieks Date: Thu Sep 16 09:30:37 2021 -0700 STAR-865 Refactor and port metrics code from cndb (#247) * STAR-865: Porting metrics from cndb-884, riptano/bdp@03b23db6a5697baaf71d46d661c0ac1c908bc33e and riptano/bdp/#19515 Co-authored-by: Zhao Yang Co-authored-by: Jake Luciani * STAR-865: Porting MicrometerChunkCacheMetrics from: CNDB-161 Add MicrometerMetrics class CNDB-780 Add Micrometer metrics for the chunk cache Co-authored-by: Stefania Alborghetti (cherry picked from commit 5e0d8897b5d71a5d74a6adebe07d8060733ed0f0) fix ConcurrencyFactorTest the metrics require reset if we want to measure, especially the max values (cherry picked from commit 3f8951413a95c8bdad58be50629a70a1d7871f57) (cherry picked from commit c51b4c4e0c955218df1bf7cd2f0ab507e3133bf4) (cherry picked from commit bc8b4ed0830670e00766808df89ae06208337946) (cherry picked from commit d45d8e2a979cf693b8e7bbcf2e4710a756a1c616) commit aa804f73783cca3d0f25c13664fb3a92964d13ba Author: Branimir Lambov Date: Wed Aug 18 18:49:57 2021 +0300 STAR-859: Further improves precision of memtable heap tracking Partition key ByteBuffer and columns btree were not taken into account and some ByteBuffers were not measured correctly. Also fixes flakes in MemtableSizeTest caused by including allocator pool in measurements and updates it to test all memtable allocation types. (cherry picked from commit d8d3e8bee57f968beb202ea08b86482a55d04a4a) (cherry picked from commit f8963ca07e76f6651454432604cdc764a7583dc4) (cherry picked from commit da97b31fc941822562635fdc143e8fa45e2f591b) (cherry picked from commit 221ac396f051d32b3e5dbb2aafc891a29fe9e5b6) (cherry picked from commit b468218c0849ef5eecec0802a05da2b5e3dbbdde) STAR-859 Fix test-name-check complaint about MemtableSizeTestBase by making it abstract (OSS trunk does the same) (cherry picked from commit 239d9876df1126565b1f908e7e35a98cccb51076) commit a790c81bdb56b9a6d333fb4ee961e6bcc8ecf53a Author: Matt Fleming Date: Wed Sep 8 10:02:14 2021 +0100 STAR-789: Refactor and port pluggable cluster membership from cndb (token metadata) (#242) Introduced TokenMetadataProvider to abstract access to TokenMetadata and make it pluggable. (cherry picked from commit 8c0a970a2408ea52e24cdf2b0a3abf244fd54f76) (cherry picked from commit 58b15c25ebf09664ad1bcf0d95d341ce14877eeb) (cherry picked from commit 1b2034e4cccbad3b19e75930d0b0f797c39e71c6) (cherry picked from commit b24d83f41201372ad6b9755fbe54d847f747fbf3) (cherry picked from commit 19e533726fedce069b11646abc4b5eb5f4a282f2) commit 5fb25593068866801c3755f8a88abb975816673e Author: Matt Fleming Date: Fri Aug 27 15:17:27 2021 +0100 STAR-868: Fix test failure caused by typo during CNDB FailureDetector port (#245) (cherry picked from commit adf34e3a2d264c128e192cd84a7844f5b748d4a0) (cherry picked from commit f2a9b43f088ea65e0a46ad3f5db6b8cbeba67b6c) (cherry picked from commit 68ea68831c78284c2237bccf2bcfd0bda7549bd2) (cherry picked from commit 4c04a8157e332d76d26d4f8b2e7bbbd650fa6ed7) (+1 squashed commit) Squashed commits: [47787240498] STAR-842: Port pluggable Failure Detector (#237) The failure detector is now configurable via cassandra.custom_failure_detector_class (cherry picked from commit 8127d4379d9dd351cc64992ea23b7cab3e522933) (cherry picked from commit d9625b6b713f80bd37fe0780e6dd2533875602fc) (cherry picked from commit 31974390b9751738933a294e40fe0657e57f1559) (cherry picked from commit dc27959f02ab79a7f3ef158e1fe15a5af2309bfc) (cherry picked from commit 294407b0278f3cab10eab6fc1883cc6c8d24e67f) commit 4b7b55702ba90c9f7b0b29870e84c6dc89a6670f Author: jasonrutherglen Date: Tue Aug 24 13:45:37 2021 -0500 STAR-123 custom lucene analyzer and some language unit tests (#192) added more language tests added brazilian cql test passes added support for setting a lucene analyzer cql json test passes fixed up some things cleanup added query analyzer cleanup; added constants added exception handling in unit test added bad options unit tests added char filter removed comments and extra code added illegal arg ex to LuceneAnalyzer#hasNext added stop word support; prior to removal reworked, no more stop words added lowercase filter test added ngram filter test added simplepattern test; snowball off added czech and porter fixed alloc removed commented out code removed extra code fixed minor issues maybe fixex setMinMax cleanup reverted reverted to a new byte[] per tokenized term cleanup cleanup fixed sasi test fixed unit test bug cleanup refactored for npe addressed review comments fixed npe bug fixed a couple of bugs removed json_ from options names; applied sonar comments fixed sonar comments fixed unit test bug changed exception thrown get -> create fixed minor issue (cherry picked from commit 3227a57ab49a94dd8bb80bf48c429667c969eb52) (cherry picked from commit add6b8d8e626d5ab4a31bdda6fcb75eed4890438) (cherry picked from commit 1c60b2d462190ca65ef88fcfc5b0c4e56315befe) (cherry picked from commit 4d3912f4e5150eb0d917b689f6aefe59099339b7) (cherry picked from commit 5afe63d6b0dc2ac37ab4f1ee8278c2ccff5ccb95) (cherry picked from commit d6c1172cd8fb2c6ddceb1dd915f364ebd69874e4) commit e8df7965cff6919857a357e6d1cd13f815b977db Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Tue Aug 10 14:01:25 2021 +0200 STAR-823: Fix deadlock in CompactionManager (#230) STAR-823: Refactor background compactions CompactionManager.BackgroundCompactionCandidate task is scheduled on compaction executor and when it detects there are compaction tasks to run, it starts each compaction task as a separate job on the same compaction executor and blocks until all tasks are finished. When the pool size for the executor is n, and there are n background tasks submitted in parallel, and all of them find that there are some compactions to run, they will schedule them and block until the tasks are finished. Though, the tasks cannot start because the pool is full - we have n background tasks there waiting for the compaction tasks that cannot start. Another thing (perhaps minor) is that we use getActiveCount() on the executor to check how many tasks it is currently running, and based on that information either schedule new tasks or not. The problem with this method is that it returns an approximate result and should not be used for making such decisions. To address those problems, running of background compactions was refactored. The whole logic for background compactions was extracted into a distinct class BackgroundCompactionsRunner. It allows flagging CFSs for compaction and schedules scans through the flagged CFSs on a dedicated executor so that scans and compaction tasks are no longer sharing the same executor. Co-authored-by: Branimir Lambov (cherry picked from commit 96bf61c4cd80ee837ae34fe9c3076e5df48ed7fb) (cherry picked from commit 804b885f893c6e91727694c6d353c0ba8365e2cf) (cherry picked from commit 2c0dcd76b397446a5d0994c6e39749a0a257d89d) (cherry picked from commit 0d72d4691ac23a6d6fab3d6ac0a1c4d364247edb) (cherry picked from commit 4cea2b1fccb0bd6521ec09d84452f95cc0740ec2) (cherry picked from commit 840d8042e29048fd21af61e1bc6021d635446a96) STAR-823 Fix CompactionManager after compactingCF was removed commit f1decbe13777e45e4d5d27d176757add46432188 Author: Branimir Lambov Date: Tue Jul 13 18:08:20 2021 +0300 STAR-767: Do not copy to heap for flushing This was failing because off-heap native clustering keys were used in stats metadata without being copied, referencing memory that could be overwritten. Also fixes a problem creating retainable/minimized versions of clustering bounds and boundaries. (cherry picked from commit f00e340447f0be54cbb6ccb72a5b31baf94a14d7) (cherry picked from commit f0904a323f7370d7cded6654b16e12273cdf923f) (cherry picked from commit 80a0383d1d26b5676194fda63daddf285e44a631) (cherry picked from commit a7354ef5fb529541e9a16859b2ee62544af79733) (cherry picked from commit 38bc45579cc87b252ecb279196e11bcc549a83a7) (cherry picked from commit 7e02ff7b8db3aa403ea5e35e3d8bb0d1f482f2a5) commit 223d849b87d06e0242b72d691c0a39f397081e40 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Fri Jul 30 16:04:24 2021 +0200 STAR-762: Implement paging in bytes (#223) Add page size in bytes flag to protocol Introduce PageSize object Protocol version changes No support for describe statement yet Simplify SecondaryIndexManager page calculation Add page size in bytes to DataLimits Refactor pagers Add / pull some tests Add some toString implementations Add PageSize to expected classes in DatabaseDescriptorRefTest Fix AggregationPartitionIterator So far we were passing the main page size to the AggregationPartitionIterator, which: - was pointless because there is no paging when we aggregate everything - it was actually harmful because AggregationPartitionIterator is a subclass of GroupByPartitionIterator and the later updates the subPager's limits with the minimum count of main page size and the number of remaining. It is correct if we use grouping aware limits, where count applies to the whole groups. But when we do aggregate everything, simply CQL limits are used and count limit applies to rows. Concluding, without fixing that we would limit the number of aggregated rows to the main page size which is not what we want (cherry picked from commit e11d71626e5a8db0a957e708855ffc8f18e1c0f5) (cherry picked from commit 13d45692b66304c2249e3f8241e80169b75ecb96) (cherry picked from commit 4f65564eb53c0f676040208dd76ba79e30a1fb10) (cherry picked from commit 9d084ea9ae4e83654c372f26b0855a20a8ffeb9d) (cherry picked from commit b0f05f2afcccfe9c734c2b18f77afe688d176a20) (cherry picked from commit 8dd49b0b9d110f2db14e1a9b425e4ba670b047f8) STAR-762 Fix conflict in DunpTest with CASSANDRA-18215 commit 217089a8172929638bc4db468fa9c2e7cd69e54b Author: dan jatnieks Date: Wed Jul 28 19:57:57 2021 -0700 STAR-847 Close client to prevent driver queries after dropping user (#227) (cherry picked from commit 09693322222669f4e1975e8066331441789da8b5) (cherry picked from commit 91a753da702a1fd4d16bd047ec3db6c6ae107b70) (cherry picked from commit 56a9b56db25cb4680b1e6ead0f913953f2784f96) (cherry picked from commit a2c6bb40eb314e9f1c1e596a8afdffa4809bb996) (cherry picked from commit 1bd3f630e796dec5baa04bea1ac5816bc10c7f4b) (cherry picked from commit 0449b6c301129829b97f634f063beeefa0a07f18) commit 6e9c8b753ed550b0284779bf6c924ff621c8add9 Author: Mike Adamson Date: Fri Jul 23 13:05:19 2021 +0100 STAR-815: MultiRangeReadCommand always used by SAI and remove info log in query path (#222) (cherry picked from commit d40f9e30625e5c0a74bf3f519ff37fc3e97648f3) (cherry picked from commit 41b5b8f6d9bd87bead7deff1d31c2f691b4912d2) (cherry picked from commit d83a0ae3a13d782fa20c04a2702e11bd6a566ca9) (cherry picked from commit 9338023fae77910260d6579b5cb5f22ecbb9f621) (cherry picked from commit 9f8ad3687abd7348ba6e0fe794bbf2c638f5ccb1) (cherry picked from commit 6fe00460393379059cfc1dca4463834eac027471) commit b278605c4851074e62f7dada816e91bbc287052a Author: Piotr Kołaczkowski Date: Thu Jul 22 15:30:01 2021 +0200 STAR-821 Cleanup minor code-style issue reported by Sonar (cherry picked from commit d8345192d63b91873d0bdbe0ecbd8bb39264bdda) (cherry picked from commit 2e3828b1b849f0b2f8c1ca2ca91b8971977c61a2) (cherry picked from commit ad9a9e615a4c42b6f38ee483b0424f8dd671cf83) (cherry picked from commit 7a6e63cc1bf4fd0a3da3dd0af31331cfb01808e7) (cherry picked from commit 5c053f6c94e3938d8f19d516f4d8fd4d37be2378) (cherry picked from commit 1339dad862a550af10e0ab42dd586fab70f90080) commit 52b923048c5a9fa4e98de7101999a5067e135faa Author: Piotr Kołaczkowski Date: Thu Jul 22 15:26:58 2021 +0200 STAR-821 Replace EmptyElement with an empty AndElement No need to special-case an empty where clause, when we can already model it with an AndElement. And it has a nice property that it is already in a conjunctive form. (cherry picked from commit 9cc2050f4608057b41ee78f715faafc827faf526) (cherry picked from commit df049f04810732d62a1081a9926a0c3b35d04254) (cherry picked from commit 34f3243c7a2ea88a7a0d27d2ba942d4935374648) (cherry picked from commit 0ceea80baa98338c22a23e000e6c76bb855f3fed) (cherry picked from commit 55f8e50147b521f52a8a3ee1763b19d807fe1def) (cherry picked from commit 37a64307a541b626ba30381e11c7eef81014353d) commit 4a14ca791b34693ba30d095bba11cfd7e5ae6451 Author: Piotr Kołaczkowski Date: Fri Jul 16 18:04:08 2021 +0200 STAR-821 Allow empty container elements Looks like the CqlParser can construct empty container elements when given invalid queries, before returning the error to the user. (cherry picked from commit 6f0f9a8b427b9363e385906b038a814a986e46f4) (cherry picked from commit a5e539c6a19ab50f4b06395fb7d1d90b4466c493) (cherry picked from commit 262e67398def5ecbe2b3d0730b4e9feb2e55e5e6) (cherry picked from commit 0bcd97f3977ff39f3927da38d985c073f3091aba) (cherry picked from commit d0bfb179309c3f92e9049fc9fa0553f1892a32f9) (cherry picked from commit e72fc12800796f6b4eafb96e6de1977b751ddd20) commit dddc2af207eb79f7274e7248f082f45e34910d31 Author: Piotr Kołaczkowski Date: Fri Jul 16 10:58:02 2021 +0200 STAR-821 Reject queries with partition key restriction nested under OR Examples of queries that are rejected now: (pk - partition key; ck - clustering column; a, b, c: regular columns): SELECT ... WHERE pk = 1 OR a = 1 AND b = 2 AND ... ALLOW FILTERING SELECT ... WHERE a = 1 OR pk = 2 AND b = 2 AND ... ALLOW FILTERING This commit additionally fixes ORed clustering column restrictions with ALLOW FILTERING and no indexes. The following query was missing the rows with ck = 1: SELECT ... WHERE ck = 1 OR a = 2 ALLOW FILTERING (cherry picked from commit b28f8cda06b3382667fbf6d26a43655f74c77bf2) (cherry picked from commit 830f60b811c88d7489e4408b2edbdc966fb60ac4) (cherry picked from commit 49f90bbb7bd8cc149f9c9f0196b766ca55715495) (cherry picked from commit 22a83faccb02dd65f09a063bfb490b8c26b909a8) (cherry picked from commit 1beb9a8ac4e72f82f59b95d99f6082eff60410e4) (cherry picked from commit 5cf9628365025d4b9a57a13f4e4754303d6d274b) commit 09e3682faf4c24ea3e253164c4ed32ef6d736121 Author: Piotr Kołaczkowski Date: Wed Jul 14 17:08:41 2021 +0200 STAR-821 Add ExpressionElement#conjunctiveForm() This new method should allow to simplify transforming an expression tree to a flat statement restrictions set. Conjunctive, flat expression form makes it possible to look only at the top level of the expression tree just to grab all restrictions that should be "ANDed", and there should be no need for any further recursion in StatementRestrictions builder. Also way we can enforce that partition key restrictions must be top-level by just inspecting the root element of the where clause expression. (cherry picked from commit d4ae7e8698f0cfb8cfaa23c48d359f232cc51a81) (cherry picked from commit 9eb01edef2dd9936598603eae690fd7d4996709e) (cherry picked from commit 0d0075ed625b78e1fd720e7d00a91d891b3500a6) (cherry picked from commit 0fe909e3e0df280c04d941e81cedc8e2e72e6557) (cherry picked from commit 98164a3d4bace52e502f7b04f007f584b162e53d) (cherry picked from commit 6895a872f12e0ff443819b8df782280c01a092f2) commit 346840f1eddf24be78f3ee6416c2bffaa46c5e67 Author: Piotr Kołaczkowski Date: Wed Jul 14 13:35:10 2021 +0200 STAR-821 Add a way to flatten WHERE expression tree Collapsing levels with the same operator makes reasoning about expression properties much simpler. AND and OR are commutative and associative, so nesting is just a distraction. Additionally refactored the expression tree container nodes to be truly immutable and fixed a subtle error in rename() that caused an OR node to be replaced by an AND node. (cherry picked from commit f2ace24ba6f797049752e1e750ec2f548327ee0f) (cherry picked from commit f3034a94b8540cac380c5ad3fe5acfb5e10f3b00) (cherry picked from commit 6106aa06e353019b594699f0d98c5414b5ec4b9e) (cherry picked from commit f439c225ffc5fed637822ff29127d0e5f87fbcbb) (cherry picked from commit 2ed8423b0e3368148ac22cb4772e57e4a80e16a6) (cherry picked from commit 136caa3a1091d3dd6c559e8786f9ee196ec6db1e) commit 8d813f693c274be475ca0cf173fbe550fda95cca Author: Piotr Kołaczkowski Date: Wed Jul 14 13:31:08 2021 +0200 STAR-821 Add failing test to expose problem with OR and PK restrictions Handling OR and primary key restriction in WHERE does not work properly. To be fixed by subsequent commits. (cherry picked from commit e41091328a9281184ba22bd0bd48b78f72f165b4) (cherry picked from commit 9739fd830f5223930d4968b84c822c78e5441a08) (cherry picked from commit e56b1e8f75c3633603e791021725503133128d33) (cherry picked from commit 2030c8435972c6404cfe8e59a48c289f2014b64a) (cherry picked from commit dd3fd8720a5f490e3d5c2e8a128d9e8971195a55) (cherry picked from commit 2f50201da789570b835d0d4f5db64af2412522c9) commit ea8d17f88e768d36639c3e7a06e1765e7951994d Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Wed Jul 21 23:20:15 2021 +0200 STAR-839: Add the ability to choose the log file configuration for testclasslist-xxx targets (#225) Also added logback-test-jenkins.xml which has disabled logging to console, default logs on trace level and filter out trace messages. (cherry picked from commit e3e170f91846e2a076884f03bb070007e578d128) (cherry picked from commit 3a395418fe1d2b03221f1cdbad9d71549d4fef11) (cherry picked from commit c34871606ad40f207fcc0e523aaec60a3855def6) (cherry picked from commit 8bd69c6ea43d9ebf691405f5ca484c1b18d67c57) (cherry picked from commit 8fc7cc24e1c6c19125e9721dba2e129d1cda9968) (cherry picked from commit 3e19245243035ff5f9b98306ef7443049665e245) commit 55cfeb02b485726f3af604114182e616129595be Author: Aleksandr Sorokoumov <918393+Gerrrr@users.noreply.github.com> Date: Thu Jul 15 15:31:43 2021 +0200 STAR-13 Add the Unified Compaction Strategy. (#132) This is the implementation of UnifiedCompactionStrategy, whish is intended to not only replace all other compaction strategies and CSM, but also to optimally choose the configuration that will result in the minimum read and write costs, given a specific workload and dataset size. The strategy will choose either leveled or tiered merge policies depending on the workload and the costs associated with user queries and inserts. This strategy should be considered experimental at this stage. -- This patch also introduces a compaction simulation that can be run with: ant compactionSim With cmd line arguments: ant compactionSim--args="-wl R50_W50 -t adaptive" Co-authored-by: Stefania Alborghetti Co-authored-by: Branimir Lambov STAR-13 Introduce compaction interfaces. This patch introduces 3 interfaces: - CompactionStrategy : to encapsulate the behaviour of a compaction strategy - CompactionStrategyContainer: to enacapsulate the additional behavior of CSM - CompactionStrategyFactory: to create the right compaction container (CSM or the unified CS) CompactionStrategyManager and AbstractCompactionStrategy now implement these interfaces and have been encapsulated in the compaction package. In a future patch, UnifiedCompactionStrategy will also implement these interfaces, therefore standing on its own, without the need of CSM, which should eventually be removed along with the legacy strategies. The factory will take care of instantiating either the new strategy or CSM. This patch also introduces CompactionStrategyOptions, for option validation. CompactionParams now uses this class. Co-authored-by: Stefania Alborghetti Co-authored-by: Branimir Lambov STAR-13 Implement compaction shards The unified compaction strategy is now decoupled from CompactionStrategyManager by implementing its own thin compaction container. When a compaction candidate needs to be produced, the strategy takes a snapshot of the eligible sstables and disk boundaries. It then applies a set of equivalence classes, which implement the same partitioning of sstables currently performed by CSM. One of the equivalence classes splits sstables across token range boundaries, or shards. The number of shards is specified in the compaction properties, and the shards are created by splitting the local ranges by this number. sstables are assigned to a shard by looking at their first partition key. When flushing or compacting, a specialized writer splits sstables at the shard boundaries. If the current sstable is larger than a minimum sstable size that can be specified in the compaction properties, then it is split when a boundary is reached. getNextBackgroundTask() now returns a list of tasks, which are processed by the compaction executor asynchronously. The following minor updates have been performed: - the adaptive algorithm now searches for better choices of W every 5 minutes rather than 2 minutes; - the cost calculator now uses a read multiplier of 0.1 rather than 0.25; - all sstables in an bucket are compacted if their number is >= T. Compactions no longer stop at T or F. This may skip levels but has proven very effective in tests when switching from tiered to levelled. The documentation for the unified strategy has been added as a mark down document. Reviewed by Branimir Lambov, Dimitar Dimitrov, Justin Chu, Aleksandr Sorokoumov Co-authored-by: Stefania Alborghetti Co-authored-by: Branimir Lambov STAR-13 Add a section in the UCS markdown about differences with STCS and LCS. Co-authored-by: Dimitar Dimitrov STAR-13 Update disk boundaries if current boundaries are null or out of date, even if the corresponding table is just being reloaded due to a metadata change. Fixes the failing TestCompaction_with_UnifiedCompactionStrategy.compaction_strategy_switching_test Co-authored-by: Dimitar Dimitrov STAR-13 Simplify container creation and reloading and allow inheriting the state of the previous container when switching strategies. Fixes the failing TestCompaction_with_UnifiedCompactionStrategy.disable_autocompaction_alter_and_nodetool_test Co-authored-by: Dimitar Dimitrov STAR-13 Make the enable/disable and isEnabled/isActive behavior of UnifiedCompactionContainer similar to that of CompactionStrategyManager. This includes always starting up the backing compaction strategy. The previous behavior resulted in every compaction task being interrupted while autocompaction is disabled. Fixes the failing TestDiskBalanceAfterJoiningRing.disk_balance_after_joining_ring_ucs_test Co-authored-by: Dimitar Dimitrov STAR-13 Make compaction shards split inside disks and apply disk boundaries Co-authored-by: Branimir Lambov STAR-13 Permit early open in UnifiedCompactionStrategy Co-authored-by: Branimir Lambov STAR-13 Limit the number of concurrently running "oversized" compactions, in order to limit size amplification. "Oversized" here is defined as close to the maximum shard size. - Calculate the limit for the number of concurrent "oversized" compactions based on a configurable option for max allowed (tolerable) SA as a fraction of the expected uncompacted dataset size. - Use reservoir sampling + keeping a non-oversized alternative to ensure that the limited number of oversized compactions that will be submitted are uniformally chosen from the available shards. Co-authored-by: Dimitar Dimitrov STAR-13 Remove CompactionStatistics STAR-13 Port isTrulyWrapAround Co-authored-by: Sylvain Lebresne STAR-13 Fix UCS tests STAR-13 Refactor repair out from compaction strategies Some major refactorings: Move mutateRepaired from CompactionStrategyManager to ColumnFamilyStore Move repair related codes in CompactionStrategyContainer, PendingRepairManager to LocalSessions CompactionStrategyContainer: Added a method to acquire the ReentrantReadWriteLock.WriteLock such that it can be passed and used for mutateRepaired Co-authored-by: Justin Chu STAR-13 Refactor compaction statistics in order to simplify the code and reduce duplication and add some of the statistics (already available in CompactionAggregateStatistics) to TableMetrics. Co-authored-by: Dimitar Dimitrov STAR-13 Improves the handling of compactions at the lowest levels of the compaction hierarchy: - shard-spanning compactions's size is divided by the number of shards spanned for the purposes of deciding in which level to put them; - the levels hierarchy starts at the average flush size for the table; - selects the tasks to run randomly to give each level and shard equal chance to run its compaction; - shard spanning compactions are given more chances to be selected. Additionally, this applies a couple of fixes: - getNextCompaction was run too many times because the decision whether or not to run it was made at scheduling, which caused many to be scheduled, and they never checked if one didn't run before; - early opened sstables could be selected for compaction, causing multiple compaction passes over the same data. - Do not run multiple getNextAggregates Co-authored-by: Branimir Lambov STAR-13 Change the newly introduced compaction metrics to be aggregate metrics instead of per-table metrics. This will make them easier to record/monitor in Fallout/Grafana, and will also enable computing them more efficiently from a cached value of the AggregateCompactions metric. Co-authored-by: Dimitar Dimitrov STAR-13 Update UCS defaults Co-authored-by: Justin Chu STAR-13 Track compaction rate in backgroundCompactions Also: - switch to sample-based exponential moving average, which is much simpler to implement correctly at the expense of expressing averaging periods in terms of updates count instead of time; - add debug logging of the compaction task count decisions Co-authored-by: Branimir Lambov STAR-13 Spread compaction threads equally among the levels Fixes the problem of long-running higher-level tasks starving level 0 or any level from continuing by reserving compaction threads for each of the levels of the hierarchy. More specifically, the whole part of the ratio between compaction threads and number of levels is reserved for each level, and any remainder is distributed randomly as before. Replaces the oversized compaction mechanism with a simple limit for the aggregate size of running compactions, which is now also applied when single compactions are above that limit. This should prevent running out of space at the expense of several highest-level tables extra, i.e. slightly higher read amplification, until someone reacts to the warning, which I think is a sensible tradeoff. Also removes unsupported options from the documentation markdown and adds max_space_overhead description. Co-authored-by: Branimir Lambov STAR-13 Fix Bloom Filter key estimation for ShardedCompactionWriter STAR-13 Unable to set max_space_overhead in UCS Co-authored-by: Justin Chu STAR-13 Log the number of compacted SSTables, and the shard and bucket identifiers when rejecting a compaction bigger than the max space overhead. Co-authored-by: Dimitar Dimitrov STAR-13 Fix Bloom Filter tracking BloomFilterTracker uses meters to avoid the situation when subsequent retrievals of recent metrics return 0. Tracking is done at CFS instance instead of per-SSTableReader to reduce overhead. SSTableReaders set correct BloomFilterTracker in setupOnline. Before correct BloomFilterTracker is set, SSTableReaders use NoopBloomFilterTracker. STAR-13 Introduce a limit on the number of sstables in a compaction and a layout-preserving mode Co-authored-by: Branimir Lambov STAR-13 Log shard and bucket details on each getShardsWithBuckets() at TRACE level instead of at DEBUG level. Co-authored-by: Dimitar Dimitrov STAR-13 TTL-based SSTable expiration in UCS STAR-13 Inherit BackgroundCompactions when recreating UCS STAR-13 Compaction log analyzer Co-authored-by: Branimir Lambov STAR-13 Always create mocked SSTables with stubOnly STAR-13 Fix compaction strategy reload Prevent resetting JMX changes when we create new strategy containers. This patch makes sure that JMX changes that alter the container type (CSM->UCS or UCS->CSM) are not overwritten by subsequent metadata changes that are unrelated to compaction. STAR-13 Handle IOErrors during background compaction task execution as FSErrors Co-authored-by: Dimitar Dimitrov STAR-13 Trigger layout compactions automatically when there are more than F*F SSTables in a bucket STAR-13 Use descriptor passed to ShardedMultiWriter STAR-13 Fix condition for triggering layout compactions in case of non-uniform W STAR-13 computeShardBoundaries handles no splitter and no disk boundaries STAR-13 Check for partitioner mismatch before splitting local ranges STAR-13 Use avg bucket size to adjust maxSSTablesToCompact STAR-13 Limit the number of SSTables to compact in one operation STAR-13 Switch CompactionsBytemanTest to signals Co-authored-by: Branimir Lambov STAR-13 Fix ShardedMultiWriterTest with compression Co-authored-by: Branimir Lambov STAR-13 Move releaseRepairData to LocalSessions STAR-13 Decouple condition for switching writers from append Co-authored-by: Branimir Lambov STAR-13 Shutdown previous strategy container on reload STAR-13 Remove unused maxConcurrentOversizedCompactions STAR-13 Metrics hold reference of a single instance controller STAR-13 Move Controller params check to validateOptions STAR-13 Fix target size estimation STAR-13 Use correct bucket index for picks that contain only expired SSTables STAR-13 Update shard index on disk change STAR-13 Remove redundant shutdown call on keyspace drop STAR-13 Fix amplification estimation STAR-13 Fix calculation of the number of pending picks per bucket Co-authored-by: Branimir Lambov Co-authored-by: Stefania Alborghetti Co-authored-by: Dimitar Dimitrov Co-authored-by: Justin Chu (cherry picked from commit 8cb7905aa079f7e84b480a6f23166192ee94d9d9) (cherry picked from commit 069b6ba99898e8d5841ad16ba849f1416af3e676) (cherry picked from commit d73878b17ddd7b2d4b09190be45fc60385b4da9e) (cherry picked from commit f41152b15c768a6247806b985d8a83b980b3ccd2) (cherry picked from commit fb05a5a4a3114187ca08f45714c2f6a6667fd241) (cherry picked from commit dc58b0b06fcd4b68e7aa55a0e63a21e444d65a31) STAR-13 Fix TimeWindowCompactionStrategyTest.testGroupForAntiCompaction usage of CompactionStrategyManager (cherry picked from commit 8c7c12879a0b0687dac3a6a09f3f3830200ad601) commit a05d1e29941d1568f165beb767dfbcde91459562 Author: Mike Adamson Date: Thu Jul 15 10:30:34 2021 +0100 STAR-813: New MessagingService version for Stargazer (#218) (cherry picked from commit ac0be817fae3c6472532d193a98e2f666ba2cf85) (cherry picked from commit 39fb781162efd1926c21bc22031b72b7a8f8e1be) (cherry picked from commit 1937874c930a204d00bcfdd05ea6d859567091c8) (cherry picked from commit af134b477a4f52e85107c1f8a583285f321d121e) (cherry picked from commit a37767a21988298dff3e4c92ff25674e4df8c82e) (cherry picked from commit c8a0639050d602d6e63489300fe4f2ae92c72c08) commit 3fcc76c0a64de85813090c90ccf2e86a680d2726 Author: Mike Adamson Date: Tue Jul 13 10:11:50 2021 +0100 STAR-815: Make MultiRangeReadCommand usage configurable in SAI (#212) (cherry picked from commit 7d0c209de0915ce223e2371edb870a56808c7917) (cherry picked from commit 071bc05cc20a13e59d649d862a42b99fb80bf9f6) (cherry picked from commit 19ba3ffd401e6a29d2052164e318415ff40eaab9) (cherry picked from commit e2d6362736b57248cc56e452d17a02362a137bf3) (cherry picked from commit 115420664dd344699d596abf86bb3ba1473dc922) (cherry picked from commit 48880009311def50ef52c1e1a52f399773f3e2b3) commit 2e92043c2856ee2ba240118935f71050e0fa9653 Author: Mike Adamson Date: Mon Jun 28 10:18:38 2021 +0100 STAR-228: OR and IN support for SAI (#206) * STAR-228: OR and IN support for SAI * Review changes (cherry picked from commit 5e0a9a6e8b06d03fabb2883b842b05f2d49df81d) (cherry picked from commit 983919321497462c822a4d04330a5559942b3421) (cherry picked from commit 73097328830b4ace8dc52d7a187a2fcc80296347) (cherry picked from commit 23a3cbad776b6174c4ce3c91aace22ce43d33f4a) (cherry picked from commit 0a7cb8aee34a61c1b75def1b7ba46be9a3703e8f) (cherry picked from commit 28c8ca813df3881645f4f44c2a5eb4ac33a64ee5) commit e24cea11ca3aa9381a1bf5724590aaa9d9be72b5 Author: Jacek Lewandowski Date: Tue Jun 22 13:53:43 2021 +0200 STAR-801: Fix ScrubberTest failure (and scrubber itself) Partition index iterator is handled a bit differently than primary index iterator. In particular, it reads stuff on load / creation of scrubber thus we need to catch exception for that instantiation and just leave the index iterator null in that case. (cherry picked from commit d7674a8785bdf1f47f2651aebdfef5cc3a6c9a2a) (cherry picked from commit e9e062aa205a1504c8578b0b36a42105fc07f862) (cherry picked from commit cfe086044b0e4f3d7699297028106852598e9c01) (cherry picked from commit d4b6c642ac6f3f7043dbe04a8d0fb646e4f4f245) (cherry picked from commit 1efb4616abf55eae11ca6162eae5481629a96fa2) (cherry picked from commit 22ce6b4ff3cb487caaed946e552f5ffd2038e890) commit 4e650b4bd5be0b94b9ff308b05affa6ae7d43e43 Author: Jacek Lewandowski Date: Tue Jun 22 10:37:02 2021 +0200 STAR-801: Adjust the failing assertion in SSTablesIteratedTest.testNonCompactTableWithStaticColumnValueMissingAndMulticellColumn The test was introduced recently in OSS to verify the correctness when dealing with static columns. The failing assertion expects 3 sstables to be read when selecting non-static value, which in fact is present only in one sstable. It is correct in OSS that we read 3 sstables as all of them touch that partition however they do not cover that particular clustering key which we query for. The difference results from read path optimizations we applied in STAR-1. That optimization improves filtering of sstables by making it more precise so that more sstables can be skipped. The optimization comes into play in this test case. (cherry picked from commit 1c24e77b52ade9e4bb94016649b9492b7a918cdc) (cherry picked from commit 18a46bd0605a0ddf9b9a9c97c41d4da6577cba98) (cherry picked from commit 2d6824be96b7873b7b7286db104cbc4b66e4dc3b) (cherry picked from commit aebe366f1c7a7a794a17dd8daa9a8182aca880df) (cherry picked from commit 4014ce30670fd7154b987d946195c26553fa3a2e) (cherry picked from commit ad38d54f360487b14809027d1a57ca4d27a18783) commit 8a86ba217029f179d9be3ec12c71fd2926963835 Author: Jacek Lewandowski Date: Tue Jun 22 09:40:37 2021 +0200 STAR-801: Refactor the logic for loading and recreating bf The logic for deciding when to load and when to recreate a bf was separately defined for bti and big. Also, for big the logic has changed in STAR-745 and some tests were added to verify it. In this commit the logic has been extracted to top SSTableReader level and used consistently for both bti and big. Post STAR-247+STAR-745 change It should fix the test failure in SSTableReaderTest (cherry picked from commit 25fa817d823fc317275902d0d156d56f81162adc) (cherry picked from commit 34761d16fc553aabefe2139bfb15947818a0186a) (cherry picked from commit 5db973983a56c4a2992e96619c7f8ceb0785ec20) (cherry picked from commit 30cade33ab37d7a61d725659b4fd2617fea7e45f) (cherry picked from commit 474bb58b805919c2eddb33f4f94c8055ca30f26c) (cherry picked from commit 808240a410f827575b76bc5f7991d28b3ee55e5d) commit 6f398ef9d27ffd60dccfbad1e78e640fcdd2e221 Author: Daniel Jatnieks Date: Mon Aug 28 09:15:48 2023 -0700 STAR-765 Fix conflict in cqlsh.py with CASSANDRA-17914 commit a6993e33356957843a5c4bbc42d49e14adc7be22 Author: Tomek Łasica Date: Wed Jun 16 12:58:51 2021 +0200 STAR-765. Add --secure-connect-bundle option to cqlsh. (#190) * STAR-765. Add --secure-connect-bundle option to cqlsh. This option is required for the cloud connections e.g. Astra. Summary of changes: * When it is provided all the connection parameters are read from it ignoring host and port settings; * can be provided by --secure-connect-bundle or * via cqlshrc::connection::secure_connect_bundle option and should point out to the bundle .zip file; * when connecting to the cloud default consistency level is LOCAL_QUORUM; * special EndpointWhiteListRoundRobin policy will be used to use any of the endpoints discovered after making successful connection; In addition this change: * fixes some potential STAR-432 leftovers * fixes LOGIN command behavior (broken in original code) Tests are added in dtest. Co-authored with Aleksandr Sorokoumov and Alan Boudreault . (cherry picked from commit 92d950a1d1ce592ca842086016ef57b1efe5441f) (cherry picked from commit 4ed6c2858ec020e7b132fa145eb7e922e29d7653) (cherry picked from commit ca22176538733fa7f60eed1ba3efd2e3e43a49d7) (cherry picked from commit 5b8962b178bb4cc0bdc6e19b882d11eea2e41bfb) (cherry picked from commit e412e89a3a315bdfdbb948e3c70c6fe0d743750a) (cherry picked from commit f50bfe61feca3cfbbfaf74e939aa1a61b90eb7aa) (cherry picked from commit 01c45e5da15f929447f3d8d75bd0654b05d1e1eb) commit 2c30387184f99419b5e3dd642e109614287e26c4 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Mon Jun 14 19:44:51 2021 +0200 STAR-795: Reduce test sizes (#196) (cherry picked from commit 05f422a1754c25ecdca1a47f357f5bb99e46b3a9) (cherry picked from commit 756eb7d5dd467539287380670954099d85f2b90f) (cherry picked from commit de936324e0faea1f6ee74a42971af54fc470c5a5) (cherry picked from commit bc0ea5c799e851840feb30bc863bde35c805b8a6) (cherry picked from commit 7d8c5e603d3e25d271236e92aa1ac170092233a4) (cherry picked from commit 1d9d77a8f907645b8759d7b91101255996bdb70d) (cherry picked from commit badc0f3c38867b3d6cdbc55177873e958c5a03d2) commit bd599c4fa66029ecd772d1debae3a2e69bbd2360 Author: Jaroslaw Grabowski Date: Fri Jun 11 22:27:15 2021 +0200 STAR-766 fix testGetScannerForNoIntersectingRanges Since CASSANDRA-5249 getScanner returns EmptySSTableScanner when there are no intersecting ranges. Empty scanner always returns false from hasNext(). Why testGetScannerForNoIntersectingRanges worked although hasNext returns false? Other test created a second sstable for which the scanner was not empty and thus the assertion passed. In other words, the test failed when executed in isolation. This commit creates a dedicated table for the test to avoid other test interference. Additionally the number of sstables is verified in the test. Instead of an assertion based on hasNext, an assertion on EmptySSTableScanner is made. (cherry picked from commit c097960cd4445e3d1243402a4d86ce952a3e6d9f) (cherry picked from commit cfee60b432ac0d944f6c9ea2e3843a66c44a8e2b) (cherry picked from commit a6669ed43f137c2d5e593c1c5104d95df94b4b83) (cherry picked from commit b1e60f6ea89909b564b99d4fdb3085014d335842) (cherry picked from commit ae482ec98fcf310c7428dcbc8408c8bad3c598df) (cherry picked from commit b909656b9eb71e70e807e98274f93cf1d7ac5e99) (cherry picked from commit 5011552c88bdd77d9719b518efa384aa0d5d5de3) commit 2f669da15a79f62ee323e2f544259e28a9561261 Author: Mike Adamson Date: Wed Jun 9 11:24:46 2021 +0100 STAR-786: Migrate SAI microbenchmark tests to Stargazer (cherry picked from commit b12ffc951af4f5ae5bf6b52779750fa3e28c7e28) (cherry picked from commit 0a42d630495f8158625a311aae04757f872f5abd) (cherry picked from commit bda686ff39ae0881fec225aefcc2e5fc661dc9c7) (cherry picked from commit 1e87e9808b1da285f1356bdbbdc0c67a35803e90) (cherry picked from commit bfd058073002615dce8b0296157f73cc08020e1b) (cherry picked from commit 0f8054fea57968fcfc43fb670a74a8ac7d24eaf9) (cherry picked from commit a36e3cd29c0a14bdb476f85a7c347a0fec077b08) commit 1684819e2f95733eb30590f6bc7505c6847f6418 Author: dan jatnieks Date: Tue Jun 8 16:56:03 2021 -0700 STAR-654 Fix permissions race in GuardrailDiskUsageTest.testWriteRequests (#183) (cherry picked from commit 019918125b1b5a2ba2eb7abb7c5d0c4f636061f5) (cherry picked from commit b708759218177d652a3cdb1535888123c96098dd) (cherry picked from commit 4c866c403755f3ae7e8797c09ead5caebac19e19) (cherry picked from commit 87dcb02e4ec9ee879d17fdc8efbce95d3c53fe49) (cherry picked from commit 173993fc095ab99c70d62355561099cbcd3a1f22) (cherry picked from commit 870850df4e8ce036180936a373a0b05dc966a884) (cherry picked from commit 428c2a3a5bdb7df25353c9b5418425f748248a1d) commit c5f299fc95119ee5992807b511af7c432b79d565 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Tue Jun 8 19:38:05 2021 +0200 STAR-403: Implement basic SSTable read/write benchmarks (#187) * STAR-403: Implement simple SSTable benchmarks * Apply review comments * STAR-403: Fix WriteTest and ReadWriteTest * STAR-403: Fix SSTableReaderBench - it was too heavy Co-authored-by: Jacek Lewandowski (cherry picked from commit 2568ab50a47319c34964642744840529d13a6558) (cherry picked from commit b6ca4e39623c8a599ac009e1fc22484750e22b34) (cherry picked from commit b63a8cca6e4cdf789837f779997a08b231fe8998) (cherry picked from commit a43b13a722c6e54ecc36f6b6ba75721cd765aca1) (cherry picked from commit 2fc36704d89fbb51436488afefd3297f3ba17ae0) (cherry picked from commit e5adada5418d154c59ab1d3e10513d7bb15525bf) (cherry picked from commit 8553bba0965dac9a9cf13a1a90e5b093ab271af4) commit ab880d19b9c4dedc551191a6e944d83014fa56a8 Author: dan jatnieks Date: Tue Jun 8 07:07:18 2021 -0700 STAR-543 Port remaining guardrails and sync with cndb (#180) Co-authored-by: Zhao Yang Co-authored-by: Paulo Ricardo Motta Gomes Co-authored-by: Eduard Tudenhöfner Co-authored-by: Andrés de la Peña Co-authored-by: Benjamin Lerer (cherry picked from commit a83e0e87c5521ec24e76de57db03dc0ed093c73f) (cherry picked from commit b90b43306e89172ae7882cb041157abf7e30740e) (cherry picked from commit 36600dfeadaad6affe2b264e3e8f07853b7c94df) (cherry picked from commit abc39d60b9b82178d86cfc86c880ea7978835fe0) (cherry picked from commit 7a263dcf3f827ea68f10de04d964da0f84f851ab) (cherry picked from commit 159a1fa87ed7b6e4696b91bfd60eb58492679e5e) (cherry picked from commit 503767436b374646362c1adfb3abd3ef181d8484) commit 5aed3038a22566a123dd3bc945af2c8740c85f29 Author: Tomek Łasica Date: Tue Jun 1 13:24:08 2021 +0200 STAR-432. Add options for consistency-level and serial-consistency-level (#173) * STAR-432. Add options for consistency-level and serial-consistency-level Allows overriding the default consistency-level (ONE) and the default serial-consistency-level (SERIAL) on the command line. If provided value is incorrect cqlsh will not start. Co-authored-by: Robert Stupp (cherry picked from commit d778304357021c2b9813fd1194227b48266045b2) (cherry picked from commit 81d62c315fd2297b5dea39af38cbd378631a4efa) (cherry picked from commit 00cd8117e05d8425ec8f9f0754c62c81202a6f97) (cherry picked from commit efecea8ca1d6aa9f2ca2a88dd81eace96c6c7286) (cherry picked from commit 56b8fe2197a97125799c4f9c24845efaf94e0813) (cherry picked from commit d3ce6ca5fe80d50c1967ffd35d43324e69f5d5f5) (cherry picked from commit abe4d5ee24dbd959204558c2090c77decfd48b4d) commit 74830603b26a667e6e6b23729233a7ee56675525 Author: Matt Fleming Date: Mon May 24 15:27:45 2021 +0100 STAR-429 cqlsh disable cqlsh_history logging (#165) We can disable saving of the history either via command-line parameter --disable-history, or by setting disabled = True in the history section of the cqlshrc. Both options will read existing history, and just won't save new commands. Co-authored-by: Alex Ott alex.ott@datastax.com (cherry picked from commit ff1bc1f395bdc00c7c54b8e763d595366a1a3f72) (cherry picked from commit d478c9ff8a416eb2b732347e1ec3a7d296407288) (cherry picked from commit 1a3f8b3fc335b72c84c34cb156a254791c3539db) (cherry picked from commit ed21da3702268abeb8b931f0ccba26c40109fb96) (cherry picked from commit 17fab85eab1ff75984d6307110b9aff0cc01ec14) (cherry picked from commit 62d43e12b6428a02148864b0bf2d40bcbadbff80) (+2 squashed commits) Squashed commits: [f11e9c7e2ea] STAR-431: Add option to prevent any file-I/O from cqlsh (apply from correct version of cqlsh.py this time) (#170) (cherry picked from commit 38a49150cd8d2ea7acbe54095a7fd8c9c3f5d5cf) (cherry picked from commit 0b65dd40abea3d2493083cca603dae658bb46eff) (cherry picked from commit 271f1f1c306beaeb777d869dbbe5759b0880e170) (cherry picked from commit 29b8d06c424b052ed104b185061c7bc6bf8bfa86) (cherry picked from commit 567c6c8710ebb38668527d3cd145dbe2d35756c3) (cherry picked from commit 1060c56898fbd16cf893172ce777c41ca6418c20) [da773eb67ed] STAR-431: Add option to prevent any file-I/O from cqlsh Co-authored-by: Robert Stupp (cherry picked from commit d7e5ff43b037684c8606cfb2b7c515bb8d51cc20) (cherry picked from commit bd6096e7c7c9a587976bb3ff49f4ac8cf57d5846) (cherry picked from commit 0d7bdd7e8dbdd839dc7e0d2e3b138d009e276980) (cherry picked from commit d3fe3d39bd48e53894f4f0b39f7f03b1efa00ebd) (cherry picked from commit dbbd7bbd17280ca10b9c417d85e4610d451b88de) (cherry picked from commit afdc4fca6cc7bbf2a05118c8943f2f553e463fa1) (cherry picked from commit 1b586e9381f4a79d2ef753e15ba6abd50a235fdf) commit 6705e9b8c3fd59ae9800d7742d34ccc70d3bbda5 Author: Matt Fleming Date: Fri May 21 11:35:37 2021 +0100 STAR-674: Enable trickle_fsync by default like DSE (cherry picked from commit fc60821edb0e61ff0d1c6e5ce52c8a8e3d455e78) (cherry picked from commit d51809875b936f5919bd6d382c570f0483f7eb5b) (cherry picked from commit 6d5d3bb0868bf017a6bda58bb8726c2a8735e050) (cherry picked from commit c14ba2b0e10ad9a4f27d484bb250982fcb3c62ca) (cherry picked from commit 1535ae4d3168f7af7b1dc616480de437f9ffde6c) (cherry picked from commit 19a1efd6823af8d48ea9fbad2a3840d8ce2813ab) (cherry picked from commit 8785ce165400a12ecbeba55a75b4de0de8969669) commit 6ad58b9a7fb49f848cb6465d73ad42149caea616 Author: Jaroslaw Grabowski Date: Tue May 25 16:20:35 2021 +0200 STAR-725 Bloom false positive rate includes true negatives (#164) This is a port of CASSANDRA-15384 that was partially ported in the last oss -> dse-db port. Only the corresponding dtest was ported and now it constantly fails without the actual fix in dse-db code. Patch by Jaroslaw Grabowski; reviewed by brandonwilliams and paulo for CASSANDRA-15384 Before this change the bloom filter false positive rate was calculated without true negatives which resulted in high rates. In an extreme case, where all queries return no data, the false positive rate could go up to 1.0. This change includes true negatives in [recent] bloom filter false ratio. (cherry picked from commit c0fa15431d3a27677ec1e74da0b419f628356d9f) (cherry picked from commit bd77281c7a61114fc95e298ab2aa31be25142b4a) (cherry picked from commit d859bc9b90e02b1b2aee590d2ca1e46e094aa43c) (cherry picked from commit 31f38e9292af05e31b15111c03716d9a218ae3b1) (cherry picked from commit 17c340d7bca0977bb002e5bad5d12fcd6af18b02) (cherry picked from commit af5cf5e28630a7f0251f034f0d89bf73243360e5) (cherry picked from commit d4ecbd21a1ab82ce80072c13f5b8389e198b2f4c) commit 2742b1b1833fc176f1412585492b3006dc806071 Author: dan jatnieks Date: Tue May 25 07:15:43 2021 -0700 STAR-426 Add specific CQL statement to record dropped columns (#156) Co-authored-by: Sylvain Lebresne (cherry picked from commit 2a48ae1d278e7e5df2cdea03186b790935d7c953) (cherry picked from commit 7ce7a83f38cfe7b9519c85525f8255451d3530d8) (cherry picked from commit 92c8f6c79ad44b45587dcfc0890d27375f039e37) (cherry picked from commit 330f0366fa318f133d26c9953af14af588060f9c) (cherry picked from commit 5ba47c825d9012e0c644079ca197089ff41c5229) (cherry picked from commit 6a980f41b958b9c1962bb325fd40850610ddcb98) (cherry picked from commit cb8994cc60eaaf0765f5030f5c882f233baa0e00) commit 93481b65af32f3171916476953957587d8572efd Author: Matt Fleming Date: Thu May 20 14:16:06 2021 +0100 STAR-674: Switch from CMS to G1 GC like DSE (cherry picked from commit ccf9d94c7e13f3e05118ac83cb1033156fe6b479) (cherry picked from commit 580bd44d0322e92e954f1736bacac10d3f924edb) (cherry picked from commit 80abaa774b91c1178ec76c235948c2c8a491b1de) (cherry picked from commit d3aeab71f6dc2dfde0e0924a1b97e8ea6be3e299) (cherry picked from commit 78e2f408ab9d76871cb997e189fc2228ee83e391) (cherry picked from commit b682c470938cfc3888fc2e45c0679863e21f81b4) (cherry picked from commit cf010b95c3097d072f184f0c2b142dffb0bbfc03) (cherry picked from commit e1fc26311a4fa264c7f0f66fefb6a46a1d2855e4) commit 643db276133c2c391ef48d6cf77c74e49131e789 Author: Matt Fleming Date: Wed May 19 16:07:56 2021 +0100 STAR-674: Use object_buffers for memtable_allocation_type like DSE We also need to reduce the GC logging level (from TRACE to DEBUG) for some events otherwise we'll overwhelm the system as it attempts to write events to disk. See STAR-658. (cherry picked from commit 1c38f5088d61a45c4a0b06412789208e3d665749) (cherry picked from commit 122bf53bdd4e68f070cd0a0f9bde14eb45e89af1) (cherry picked from commit d529aa339190c1591de1a372ed775b06439fc69b) (cherry picked from commit 94414c1bee4a08246b093b7e92a2a1e1e762a4cd) (cherry picked from commit 19ca6c31f42756d3a1790359355cc2246a0b8202) (cherry picked from commit 3b7cb0054e4c9df2d90f076c3f76143d9135fae8) (cherry picked from commit 47ec658cca36ebb922171cee3a541b653314f3c4) (cherry picked from commit 0a92a8241a0fee59ca252c13570f8e10ba3e8ff6) commit ac27087624494bd1da0cb910ab5786528b8d6f3d Author: Matt Fleming Date: Wed May 19 12:01:58 2021 +0100 STAR-674: Map disk_access_mode auto to standard like DSE (cherry picked from commit a76fd0a98796488f454980f3100dda3477f4c238) (cherry picked from commit 3d3e7a563c35df52cab9b94f09d50a4e3cb63062) (cherry picked from commit c7f0c550d3b09394f23d4a17d7934a23657866d0) (cherry picked from commit e9c6966bc618aaeb3810aa572fad0856616b9625) (cherry picked from commit 5be334ca3a3525fbbc0e240c9845e0b7e3db5cda) (cherry picked from commit 8afb030bf587d9a5227a7d56734b04c33cd4c89e) (cherry picked from commit c7305f8235204e58d4e8708bbf1da04175b74a69) (cherry picked from commit 253e6f76b53cde61e0c5ef258cf30be3b3d56fbd) commit 5f8f130ba6d9db65441a334c36ae3d29c66cca79 Author: dan jatnieks Date: Thu May 13 13:18:08 2021 -0700 STAR-515 Add guardrail for SASI and STAR-512 Add guardrails for total and per-table indexes for SAI (#140) * STAR-512 Add guardrails for total and per-table indexes for SAI Co-authored-by: Paulo Ricardo Motta Gomes * STAR-515 Add guardrail for SASI Co-authored-by: Zhao Yang Co-authored-by: Paulo Ricardo Motta Gomes Co-authored-by: Zhao Yang (cherry picked from commit e2ae18ead10d7186a76bb1a8299bfcd4ddf3820e) (cherry picked from commit eaf8c59b3820a575742370aa6b4936d3783bc49b) (cherry picked from commit 753fb6916ba32673a479b8c8e39cec447aa97012) (cherry picked from commit 0fafd7b053fc2520219ab8a0243362539696baf8) (cherry picked from commit 41d26d3e2dc86f8be5e1639ac68d684eb3e57772) (cherry picked from commit 3a40eae385b31cac03b61cb81fd5042c0f93964c) (cherry picked from commit 203e112c8ac22fb9141913b15e3b9d9b5b36f30a) (cherry picked from commit d508237f522f941c33c00c5ad97beb9d7ab09aab) commit b4fa6f3fef8aa422ec999209a3b0c47e73843885 Author: dan jatnieks Date: Thu May 13 09:55:31 2021 -0700 STAR-514 Add guardrail to disable counter columns (#139) Co-authored-by: Zhao Yang (cherry picked from commit 82351749bd01e346441e747aac0c7d96f10b9239) (cherry picked from commit caf45c16690dd41fdef88cfd02be0aeb89ffe289) (cherry picked from commit 164f30f9185dc21451e86f4b2eb8db2a8e9b5f7c) (cherry picked from commit c0cecbe28c94070e509dad67bbe45e0e1549fab8) (cherry picked from commit a6c5c6606d01fcaf10cb44b4b6b4785cc37e09a6) (cherry picked from commit 0b87207de8558a171acb5fa8a69f15343f962aa7) (cherry picked from commit f6c41df99ddb43645aff7eaf013c05648d7f0aa2) (cherry picked from commit c9a4b477d306e740ae766fcfcc8801ea0f449f83) commit 67ba33904ecd05a5ba7d3f1c52c5faff0d0755c5 Author: dan jatnieks Date: Thu May 13 08:58:25 2021 -0700 STAR-513 Add Guardrail to disable TRUNCATE TABLE (#138) Co-authored-by: Stefania Alborghetti (cherry picked from commit df12ae10368526485dcc6e4314c2b2ce1a9efe9a) (cherry picked from commit e0ffa4c68835d8ced03c18b9889e88b688060df0) (cherry picked from commit 0bd7ea3e9298485cf26c61dee4e76f08bbee4b76) (cherry picked from commit 226a4ed4f6cf6a60aff5a4a3aea9d30623952073) (cherry picked from commit eca25664daadfe104449ba954856f66ac1c3d06d) (cherry picked from commit ac4e51f5ae1e812b76fc1d245eb3cbe04e2d7568) (cherry picked from commit 3ad6b59a0ff94d2904bbe907efaf0f483544b65b) (cherry picked from commit 9977eea8f7f8b3af1d528b40b64c92d6a82c97c8) commit 3f20dabf8c3a1c32248284adcb944cc21f8e05fd Author: dan jatnieks Date: Thu May 13 07:49:50 2021 -0700 STAR-511 Port guardrails: ignore table properties (#134) * STAR-511: Disallow obsolete table properties Co-authored-by: Aleksandr Sorokoumov * STAR-511: Add new guardrail that ignores table properties Co-authored-by: Sylvain Lebresne Co-authored-by: Aleksandr Sorokoumov Co-authored-by: Sylvain Lebresne (cherry picked from commit 117bbc92990c7db571320e294990406087bd2fea) (cherry picked from commit 301b93dca10ce9a65bc5b42443a6a527d7110d3a) (cherry picked from commit 1f5b4246b2332bf8266db8e2d9ec3bc4d310b62e) (cherry picked from commit cfe2474695bd4a4dbf2180b4024b63186a63a9d0) (cherry picked from commit 8c4953bf7334c7159e553c7f89d6280025fa2adc) (cherry picked from commit de1e713904e4caead522645db91926a5d88ec303) (cherry picked from commit ca3dad621c98071ddb9d7459bb7a8515577515d5) (cherry picked from commit 086985e584c5bb9611ee1d2de555e4f24b2ac948) commit 6ddce1d4600deeba53efc9dc590ee0e1711eabec Author: Jakub Żytka Date: Tue May 11 10:01:12 2021 +0200 STAR-555: fix commitLogUpperBound initialization in AbstractMemtableWithCommitlog Fix commitLogUpperBound initialization in AbstractMemtableWithCommitlog so that it is always available when writeBarrier gets initialized Harden Memtable API so that it is apparent that getting commitLogUpperBound is valid only after it is fully established Co-authored-by: Daniel Jatnieks (cherry picked from commit e2fb3a1532967de62dc2885e14551f356793196b) (cherry picked from commit a4482ec698472b17f2549d7c6785d311a3be224a) (cherry picked from commit c5804252d67971a6860124bfe904987ce4830af4) (cherry picked from commit 942621a384258d9a206cc5ff3ec010b71f626b14) (cherry picked from commit c71be0e21bdfbf230e576f36c02e982591fff154) (cherry picked from commit 5000060af0e6f473818a283102190aea3d34d823) (cherry picked from commit e0a5f2f7c706311af9931f833a68d9d78d40191e) (cherry picked from commit 5407fa237b738325b6208af9e5c395047e36db75) STAR-555 Fix conflict with CASSANDRA-18118 commit 7d4879ba614592acec1fd77cfe2e21640f2e7051 Author: Jaroslaw Grabowski Date: Mon May 10 17:07:24 2021 +0200 STAR-530 SchemaKeyspace is compatible with riptano/6.8-cndb (#133) * STAR-530 SchemaKeyspace is compatible with riptano/6.8-cndb Add missing columns in SchemaKeyspace, which results in sstable compatibility with riptano/6.8-cndb (used by CNDB). This makes full-stop upgrade from Astra serverless to dse-db possible. Additionally fix reading commit log coming from riptano/6.8-cndb. Commit log version 680 is read with MessagingService.VERSION_40 messaging version but in truth it doesn't matter as the version is never used in deserialization. * STAR-530 add UCS mock Add temp UCS mock to support full-stop upgrades from CNDB to dse-db. The mock should be removed by UCS feature PR. (cherry picked from commit 03ad8bab34840501cacf1bcee2028ab2a62cbcdf) (cherry picked from commit a4598b5346c9fbf3f46a1ffe96e004df412f3310) (cherry picked from commit 2492b498d13076fd78bc96d4d09295e359b982bd) (cherry picked from commit 805ab0551420c4cbf0ef14e5f8c06a224f7c84aa) (cherry picked from commit a4eec2f4ee5ea6b4522572fa8f76d6e07a19d2e9) (cherry picked from commit 126aabc5666a5ffed7b88e555ccadf89acaf1904) (cherry picked from commit 16620ed3ed18522fda1659fbcf08f6fa54954ac5) (cherry picked from commit 206504fd0a108ed6f05c7af9c42751fbe3358ed5) commit a441beee397c19e32d24d05f0e2efdac60ba7add Author: Jakub Zytka Date: Fri May 7 11:05:50 2021 +0200 STAR-553: update last flush metrics in TrieMemtable + cosmetics (cherry picked from commit 5081fa14af122ebac82bfff997be7ff483bfe2d3) (cherry picked from commit a3280794778883852d1b84fe3123b609af8a9160) (cherry picked from commit ce958c67252e8c5fc75f5f59438b8d31209462dc) (cherry picked from commit c1a564ab32b803fed563ac22bbbf7a378a3fe406) (cherry picked from commit 22f7dc41846f389e79920d93b6a3e202998e33c7) (cherry picked from commit 658ccd295df8d3eb86d49baf9244e5b67b64a83b) (cherry picked from commit efcab87e341526a01bd47cc5305d86652a61a4a1) (cherry picked from commit ac37afe5ece570194a49a4d3eaf16a8a3fc837bd) commit 5cf83634698915c65de6ac1b7e1633ecc7b354f2 Author: Mike Adamson Date: Thu May 6 16:07:51 2021 +0100 STAR-549: Only register index components that exist on disk with SSTable (cherry picked from commit ee59af85f5a4bba09892fe28ddf7112a0921d781) (cherry picked from commit 3212a6674cc61c3f0c60cd22f6a754ed45f104f7) (cherry picked from commit 4c29fc971b999d2356458b7481540cc1ba0905a4) (cherry picked from commit 6e587695bcbc50d41ba0beb959eb84ffc03ab514) (cherry picked from commit 8f00b1feb9f350bda4e4bbf3ecf9eead36eb6971) (cherry picked from commit f639bc1c21011754be79c94af027d865d29c8ac4) (cherry picked from commit e3489251161985c9a4cadefe4e987d258d2db3e7) (cherry picked from commit 09f5dc23a0f77f96322dde0d67d4ffc6f2b8d834) commit dc7899c826f8aa9ff2a69a07704a6e8f6ad5bcd0 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Thu May 6 08:51:37 2021 +0200 STAR-539: Extra fix, missed running bigtable tests again (#143) The problem is that we cannot test serialization / deserialization this way with null clusterings, because they get truncated during serialization. Thus, limit serialization / deserialization tests to the cases where there is no null in clusterings (cherry picked from commit 0a5fb20341a7daf41e8f4dbcb1b34d547d1b8715) (cherry picked from commit c8964c64efda766a77e445500c24e22198e9c7ac) (cherry picked from commit 7e081acbd6c4fd95a1263bc9840b523ac388a2d2) (cherry picked from commit e03747e0cde93cefc65842611d0e2416ced402b0) (cherry picked from commit b1ba4e1b6153a2976457d4383a7a36c759a687f0) (cherry picked from commit 16d937ba7a1e9b2f622154564bc0e0ca90c9a9c2) (cherry picked from commit 6f192f01865d96ff1e2495724fef586f028b117e) (cherry picked from commit d67428ea381ef1ce571a4deae50796db6f01b2cf) commit 10fc64b27327d1ed6a3418447c584f0207e5e2b5 Author: Jakub Żytka Date: Wed May 5 12:16:46 2021 +0200 STAR-396: Multi-buffer, sharded memtable trie (#131) * STAR-396: TrieMemtable sharding; Multi-buffer memtable trie Co-authored-by: Branimir Lambov (cherry picked from commit 62ead9eb0fcf22ed39ad449398b56aed0a23505c) (cherry picked from commit 2eb26043abe8b882d9790abe187e9ad7019d21c8) (cherry picked from commit 00da732ee5b3507902e0a73a6f3f3ad7021be8b3) (cherry picked from commit e2af3b8ee8489664368f3d5544cc298881889e7a) (cherry picked from commit b2d0e0d11a0e6cfb21b95f26608e069b421f653d) (cherry picked from commit 8877953723939d5de60435dcb26e3980af672b99) (cherry picked from commit 0fd098cbca20cb376b7ffd32a054c11e589fa1c9) (cherry picked from commit 45813dda52a1004ac46a1295ec40b56d5a257d83) STAR-396 Fix conflict in TrieMemtable with CASSANDRA-18118 commit bbef96c8a576ce4dc163545429d64004c4c231d0 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Wed May 5 00:09:54 2021 +0200 STAR-539: Fix storing stats metadata for big table format (ds-trunk) (#141) * STAR-539: Fix storing stats metadata for big table format The problem was that at some point we refactored the code a bit and one detail got missed - for the min and max clustering bounds we should store clustering components until first null component is encountered. * Improve test coverage (cherry picked from commit e0c0accb61dede076954f7d39ba494e5f2249551) (cherry picked from commit 88d7d00d112d8e09476512765e385684490e43e1) (cherry picked from commit b91909df3f54ef1a97517838ce629a1a09aa8a86) (cherry picked from commit a07dcb3fe1f8521f2f7a72d3102e5e1b8247e728) (cherry picked from commit b9f8c58b2de56b079017356e4d33cff816f853fd) (cherry picked from commit 024e87f621985049cd3c18c1bfc51ea3f0f802c8) (cherry picked from commit 4fc8efe3b950db50c5228d4048ac56d7a8cad1ae) (cherry picked from commit eceb6a7b54a97be9042ee615270d5391fc9945ae) commit 86608125acc253616a712ea1c3606e5f36dc9d3f Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Tue May 4 08:57:08 2021 +0200 STAR-400: Add a smoke test for mixed sstable formats (ds-trunk) (#136) * STAR-400: Add a smoke test for mixed sstable formats A smoke test which operates on a table whose sstables are in both formats - big and bti. We try to read and compact those sstables. * address review comments (cherry picked from commit def65b86df31010ef581729153544c91f871b7d8) (cherry picked from commit 5d3739b69384243700e8fb806dedecf1b1fc4263) (cherry picked from commit 4a304000581ca382cb54c949c3cd2a935f7eae0e) (cherry picked from commit 69b7f8fc98cbfff2d42bd0e22473b15de15a3ec3) (cherry picked from commit 07c1d27a177f549d243eb9d9e5a1ed0c18d065e1) (cherry picked from commit d4de9520270fb64356a69ba873a673aaf4bbea1f) (cherry picked from commit 3d6806028902d4a5e536dad3cf0fd9f643e4ce5f) (cherry picked from commit cda600fa22e03d2d9da1cd0fddfb9e87b1e657b4) commit a178f4d5dcd11a0e6ffffbb7264fb4fc3e0981e5 Author: Jaroslaw Grabowski Date: Thu Apr 29 10:58:47 2021 +0200 STAR-455 ignore VERTEX|EDGE LABEL in CREATE TABLE statement (#128) * STAR-455 ignore VERTEX|EDGE LABEL in CREATE TABLE statement The DSE-specific graph properties are added to CQL syntax. Once detected, a hint about them is provided to the node via table properties: dse_vertex_label_property and dse_edge_label_property. The node returns a warning to the caller, ignores the properties and creates the table. * STAR-455 ignore graph_engine property in CREATE KEYSPACE cmd the property is ignored, the keyspace is created (applied == true). A warning is returned to the caller. (cherry picked from commit b2a8a809d8cc11580e6027bc5838394f00f86481) (cherry picked from commit 25389746ebb04c383ddcd768760add9fe95735c4) (cherry picked from commit f401273dd53fe6273d3d95da572e8626d4f09747) (cherry picked from commit 6473e019dc6e579aebc9e27d8ad8d227cbeedd60) (cherry picked from commit 60f6e7112c86e0c6b8e959e07b50ef515dd95b11) (cherry picked from commit ca14d11cde16531477857d71c0ebc62ced3e0dac) (cherry picked from commit 721762d580fe4d65b1fb32c0cb3774a8a7928869) (cherry picked from commit 796477199fe57be0371d6edf61e059c032571776) commit e76a64232890169c2c1736fa2bb5dbd28a4751fa Author: Jaroslaw Grabowski Date: Thu Apr 29 09:55:12 2021 +0200 STAR-454 ignore 'nodesync' table option (#126) The table creation command ends successfully (applied == true). `nodesync` table option is ignored, additional warning is returned to the caller. (cherry picked from commit eaeb61283ce845a345954efe92a3989e2990c1be) (cherry picked from commit 47bc95e77f2138c86fdb5cdd0af8488824b6c8f8) (cherry picked from commit c0100b281aa1b61a35b3336c21b0b216a1139d15) (cherry picked from commit 3085dfb8a619454afc752e0b768d7684b8d26b9f) (cherry picked from commit dd707b7703b3b4fd4c3f634f365feec517f6f753) (cherry picked from commit e3e915ea09e18280ec55775ba438922f4b73eabc) (cherry picked from commit 950d3563c1e514b82b0ca679a86e7ccf25a5002b) (cherry picked from commit 01f6cf05412eb55cee35106ed97b6bfdfa4874d1) commit 2eae0a6e39c57633bb65f7d42a9002851b146de3 Author: dan jatnieks Date: Tue Apr 27 16:02:43 2021 -0700 STAR-409: Port guardrails from astra branch (#124) * CDB-15: Introduce basic framework for Guardrails Co-authored-by: Eduard Tudenhöfner Co-authored-by: Sylvain Lebresne * CDB-18: Introduce Guardrails for MV/2i Co-authored-by: Eduard Tudenhöfner Co-authored-by: Zhao Yang * CDB-19: Introduce guardrails for Table number and Table properties Co-authored-by: Eduard Tudenhöfner Co-authored-by: Zhao Yang * CDB-22: Introduce disallowed write consistencies guardrail Co-authored-by: Eduard Tudenhöfner Co-authored-by: Aleksandr Sorokoumov Co-authored-by: Andrés de la Peña Co-authored-by: Zhao Yang * CDB-21: Introduce Guardrail for partition size Co-authored-by: Eduard Tudenhöfner Co-authored-by: Aleksandr Sorokoumov * CDB-20: Introduce Guardrail for partition keys in a SELECT Co-authored-by: Eduard Tudenhöfner Co-authored-by: Aleksandr Sorokoumov Co-authored-by: Andrés de la Peña * CDB-17: Introduce Guardrails for CollectionSizeWarnThreshold / FieldsPerUDT / ItemsPerCollection Co-authored-by: Eduard Tudenhöfner Co-authored-by: Andrés de la Peña Co-authored-by: Zhao Yang * CDB-3: Redact guardrail user data in error messages that are passed to listeners CDB-16: Introduce disk usage guardrails Co-authored-by: Eduard Tudenhöfner Co-authored-by: Zhao Yang Co-authored-by: Aleksandr Sorokoumov * CDB-30: Add Guardrail for cartesian product of IN queries Co-authored-by: Eduard Tudenhöfner Co-authored-by: Zhao Yang * CDB-31: Add guardrail for user-provided timestamps CDB-23: Introduce Guardrail for read-before-write list operations CDB-3: Generalize/improve conditions on which Guardrails are suspended Co-authored-by: Eduard Tudenhöfner Co-authored-by: Andrés de la Peña Co-authored-by: Zhao Yang * CDB-3: Make sure tablePropertiesDisallowed Guardrail is also applied to MV creation/alteration CDB-3: Fix remaining test failures CDB-3: Fix CompactionStressTest / CQLSSTableWriterTest Co-authored-by: Eduard Tudenhöfner Co-authored-by: Zhao Yang * CDB-3: Add missing guardrail settings to cassandra.yaml and missing enabled check for tables limit GR Co-authored-by: Eduard Tudenhöfner Co-authored-by: Andrés de la Peña * STAR-409 Fix ClientRequestSizeMetricsTest by ensuring session connected before clearing metrics * STAR-409 Add call to SSTableWriter.guardCollectionSize in PartitionWriter.writePartition Co-authored-by: Eduard Tudenhöfner Co-authored-by: Sylvain Lebresne Co-authored-by: Zhao Yang Co-authored-by: Aleksandr Sorokoumov Co-authored-by: Andrés de la Peña (cherry picked from commit 0676af036fc17d1e9d9aa23e1305ea1dcacd2040) (cherry picked from commit d187a01bfd17aa58ef109869073f674a57ca0d1f) (cherry picked from commit 4c99b0fdfbe690ad6e180a904d34069f8807e753) (cherry picked from commit 2dfc86070d0211a19797ec345c1473eef4fb4e4d) (cherry picked from commit 604b4ba03f0c1f9fbded3383213b0be8e8adb236) (cherry picked from commit 9ea01995b6f29c18c13876e993285c2b3643de08) (cherry picked from commit d9a84f066cb15e831ea35dc011636a467affb597) (cherry picked from commit 29c90ac629a372cea0bf887524e5719cf54b1ebe) commit fc0a7eb0954dbea7583525ef037d7bead66a353f Author: Aleksandr Sorokoumov <918393+Gerrrr@users.noreply.github.com> Date: Mon Apr 26 17:04:21 2021 +0200 STAR-410: Define compaction metrics and make them easily accessible on Fallout (#101) * STAR-410 Refactor compaction metrics Co-authored-by: Stefania Alborghetti Co-authored-by: Branimir Lambov * STAR-410 Rename CompactionInfo.Holder to AbstractTableOperation and CompactionInfo to AbstractTableOperation.Progress CompactionInfo.Holder was implemented by all table operations, not just compactions. For example index and view building were reported as compactions. The new naming reflects more correctly the role of this class, which is to report progress or interrupt these table operations. CompactionInfo was a basic class capable of reporting the progress of an operation, for example how many bytes (or keys, or ranges) have been currently processed out of a total number of bytes (or keys, or ranges). The new names should remove the previous confusion, and allow adding further compaction progress statistics. Co-authored-by: Stefania Alborghetti Co-authored-by: Branimir Lambov * STAR-410 More refactoring of compaction-related code: - Added CompactionProgress and refactored CompactionTask into CompactionOperation - Extracted interfaces from AbstractTableOperation and its progress inner-class, so that they can be extended by CompactionProgress - Added compactions in progress to the abstract strategy for a basic aggregation of compactions in progress, to be extended further Co-authored-by: Stefania Alborghetti Co-authored-by: Branimir Lambov * STAR-410 New aggregated statistics for compaction strategies added to nodetool compactionstats and compaction logger Co-authored-by: Stefania Alborghetti Co-authored-by: Branimir Lambov * STAR-410 Rework compaction strategy code to remove repetition and clarify Co-authored-by: Branimir Lambov Co-authored-by: Stefania Alborghetti * STAR-410 Minor refactoring of compaction strategies and fixing of CI failures: - Refactor getEstimatedRemainingTasks() and getMaximalTask for STCS, nits - Fix a possible deadlock in CompactionStrategyManager - Fix strong self-ref loop A strong self-ref loop of SSTableReader was caused by background compactions because the compaction strategy manager is an observer of the tracker. So long as the txn sstable tidier references the tracker, then the tracker references the csm and the sstables via the background compaction aggregates or picks. Co-authored-by: Stefania Alborghetti Co-authored-by: Branimir Lambov * STAR-410 fix compaction interruptions, refactor compaction iterator to return a separate table operation Co-authored-by: Stefania Alborghetti Co-authored-by: Branimir Lambov * STAR-410 Undo some changes in ActiveOperations * Store all active operations in the same set instead of a by-table map * ActiveOperations does not extend SchemaChangeListener * fixup! STAR-410 Minor refactoring of compaction strategies and fixing of CI failures: * Review: Use throw Throwables.propagate instead of hrowables.maybeFail when catching exceptions * Review: fix wording in BackgroundCompactions and CompactionTask * Review: Remove unnecessary maybeReloadDiskBoundaries() call * STAR-410 Compaction strategies filter out SSTables that are no longer in the live set Co-authored-by: Stefania Alborghetti Co-authored-by: Branimir Lambov (cherry picked from commit a8be76dfbf72f46f24d84cc1b62188280deec893) (cherry picked from commit 35e12f4887c615c3e0423d86dbe40324a42a9fb4) (cherry picked from commit 1f456e2b4313a4beaf3e091f8319bde047e890ce) (cherry picked from commit 32b374a6fce9291cc5c2a38e5ef82432858b38d7) (cherry picked from commit 29500e962943a0c582595e42f519b3e7fc1ec9d6) (cherry picked from commit dcb9cce8d17ceb4ed8e70cfd6e6bf76bdf95b94c) (cherry picked from commit 3d134e119bfe3d8801f7a7b433dcad60058c69c1) (cherry picked from commit 41b49c90fee98d5de4650870c6f19874ee5eda42) commit ffe2164f2738a6785f9a1fe89b3527b6a054e152 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Fri Apr 23 14:10:05 2021 +0200 STAR-493: Fix Scrubber (missing bad rows incrementation) (#127) (cherry picked from commit 18b2a69f1ab968007784e569b69d9898256b738d) (cherry picked from commit cf264ff0c378fbb77526b3b5a3c9e44684f5e765) (cherry picked from commit 60d2fbe4dd005009468654c4a276ca41a121d83e) (cherry picked from commit 7254d95ee7ef512b459643a06c098cd8c6a5654c) (cherry picked from commit d39f025e5e16cf97b6d3ac0e6042d5caa6ccbc24) (cherry picked from commit bdec286a28f2826ae73ec6f66e3b1f4ce1a80bb1) (cherry picked from commit 8267ade5c44561085d022142b2288c93dbbd6f02) (cherry picked from commit bdd16057a872d977e180b6efae6a40c1900f23de) commit bba82ba27dd751560dee089346c65e54b96ea233 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Fri Apr 23 11:23:40 2021 +0200 STAR-404: Add maxColumnValueLengthsMap support in StatsMetadata (ds-trunk) (#116) * STAR-404: Fix sstable versioning - maxColumnValueLengths is not present any longer in ca - removed nb and nc versions (for OSS compatibility) - update legacy sstables data files for testing (cherry picked from commit 5150d26ba2b06c4ae297b2102c9f882e4589ebaf) (cherry picked from commit 295e5d0356c4434ce76298ea7b53584f65b153f8) (cherry picked from commit e0fa04822a8d5d9abee11565aad1146c49029504) (cherry picked from commit 8e2f732785b847e0ba327ed5111df16b51722830) (cherry picked from commit 4c5d0eb1481df69b0e8cdd5c22b6bd25865f1c04) (cherry picked from commit e703d3c56e63c100266069352882c8279b34290d) (cherry picked from commit 8ca841d3ddce09158b29c9625ad216493bc4d778) (cherry picked from commit 80dd880e3109bf7d94da859e85af8ffbdfdbc723) commit 23cc851dc4edf9bac383afc5fcf5646a15c426e0 Author: Mike Adamson Date: Thu Apr 15 09:56:37 2021 +0100 STAR-184 Endpoint grouping for distributed range reads (cherry picked from commit f108eab68edb1ddd20eb7723194cf69cd84ab2d0) (cherry picked from commit ba54245e6c992fdae362c513fb3f4dc78cfa24b2) (cherry picked from commit 9f5f99726a0cccadb8db369f951ddc0f9c98b879) (cherry picked from commit d079c3363648fc1fbf62a0d483e484536a4df71c) fix RepairDigestTrackingTest (cherry picked from commit 96e8f95e850bf66a18f2b45c606caf7019381fd6) (cherry picked from commit 8f7f20004625be2eab74d90d57b39470c98a93f6) (cherry picked from commit 8ba99707468e7f147dabdb2f4d4a0b5c0b7293bb) (cherry picked from commit 66cf9a9d5457bc9bc96dd55c7fd5cc584b796544) commit 352c8f1287b32848c322cc2e69e0324fbae7dfe6 Author: Jaroslaw Grabowski Date: Thu Apr 22 17:32:17 2021 +0200 STAR-451 override unsupported DSE compaction strategies (#122) * STAR-451 override unsupported DSE compaction strategies Create table command overrides a recognized unsupported DSE compaction strategy (MemoryOnlyStrategy and TieredCompactionStrategy) with default Compaction Strategy (SizeTiered atm). The table is created successfully, applied == true. Additional warning is returned to the caller. (cherry picked from commit ae2eded1eb1f2db40d775bd0245f215907a539d5) (cherry picked from commit af68a13f7494db1a4908b8a12bbd2649befba315) (cherry picked from commit 3ad7794f486985a216030f3643edabe6b1e142ad) (cherry picked from commit 8e1ceb86b517671c14e217f5b8ad0bb8d5c2497f) (cherry picked from commit 67b943be00c7beffc3307fab06132adbf5294b9d) (cherry picked from commit ec021129b106c61f61ad9dd0562d0be77068a0f3) (cherry picked from commit 018d061e4b6c7795bdd9c6c15ed48daf596224ab) (cherry picked from commit df049df9b59f754776fdc6a8e4eb819cf9e183a4) commit 6a9d14073e062614744b380d62f6dfc7fc9647da Author: jacek-lewandowski Date: Thu Apr 22 10:19:39 2021 +0200 STAR-501: Fixed number of CPUs = 2 for testing (cherry picked from commit 1c4867c35703063dc4162b2a7c8c213763c91283) (cherry picked from commit 1b4b2252cbf46fa86ae03aeee45b60401a07590d) (cherry picked from commit 79e4a70fec0b5170e7b16a44f66057b0a642a1c5) (cherry picked from commit 86f7d99837e115c7e7c50cb9910c620eac267556) (cherry picked from commit 805e890a9a8e78345517b3f41afb2457210c7612) (cherry picked from commit 92050668d67ee28963401675fb3893c3d4b2fca9) (cherry picked from commit 3270fbec55fee06231e11d8fd18aa2359bdfee41) (cherry picked from commit 798c68121d1623538521f3b59f0b6add9cbd3e66) commit c0126608678ea9adfc29f08258168c3061fa8ace Author: Mike Adamson Date: Tue Apr 20 12:57:18 2021 +0100 STAR-343 Frozen Tuple, Frozen UDT & Boolean support (cherry picked from commit 064d9eeb27c327b7b9a3ae55ac10137e3e30eca6) (cherry picked from commit bb40cd0ad2a173c260328e25b1941d8c7adbd705) (cherry picked from commit cb00ffb26198a32c282ce1cff273c6ca2b123cc2) (cherry picked from commit a7087ef4c23dd0d83f51dc26cbdd1c4ada7cd0c5) (cherry picked from commit ac9b173fab31ad838e87e820de3376b6d10e1a6b) (cherry picked from commit 240b7aef04972003077ebda3f89a6423415ec799) (cherry picked from commit d415d93e29f6c77dc54e1a5b96054b94d398ec96) (cherry picked from commit 8ddab91668b7415b0b4f16526368a9ebc9fe0868) commit 65001facc4541976f85fbbfe656e25b07b5065f0 Author: Mike Adamson Date: Mon Apr 19 18:19:58 2021 +0100 STAR-476 Fix TrieTermsDictionaryTest for duplicate random values (cherry picked from commit 85477999f65ea601a9ba5c69c4955019c219f92c) (cherry picked from commit a8fd86c2f8635056e5fa4eddadd1c7318de1833d) (cherry picked from commit 59c12a978f2cf23fe57d85fa91bb3be416ae5f4c) (cherry picked from commit b027eb202577153ca40eaa46cd7a0cdd0679c285) (cherry picked from commit 011d597bd6c0fb3f996770eeb923dacacdf73cc0) (cherry picked from commit b6e09d67b1b904c408951846e2c1b63f76d9708c) (cherry picked from commit fa862cb700e0f2484f668df1f7d061559af1d4d7) (cherry picked from commit dae9b684d963781096c8f0b9a55c454972efdacd) commit d976fe5ff6c4f5f7937d53983072644b4b486888 Author: Mike Adamson Date: Tue Apr 20 16:40:08 2021 +0100 STAR-297 change dse.trie_size_limit_mb to cassandra.trie_size_limit_mb (cherry picked from commit e198beb23337227209ebc93da03025c4c731a844) (cherry picked from commit 31b6eb79e480746643f1a5b28e5045ad12a0f8a2) (cherry picked from commit a757e2f26318dd63cc3d264f6c082714d2d193a5) (cherry picked from commit 04c42997a7cde0d3a75ca4d6ea5b32e0509b5e27) (cherry picked from commit 8b9742844238cf14df345576fc5b014c0a5ef33b) (cherry picked from commit 25737120b2be693536fb74f1fd19af85c12d3a1e) (cherry picked from commit 37c9ef049fb859a066e37464f8e14a13c6b301dd) (cherry picked from commit 49e755e6cb7cbd4126bb352ab92bdf6ddd7f73da) commit fe0d103f729fbb34877e4981850539028c6aa336 Author: Mike Adamson Date: Tue Apr 20 17:23:15 2021 +0100 STAR-435 Make MmappedRegions.MAX_SEGMENT_SIZE configurable with system property (cherry picked from commit fe40734503feafe3123632f711bfff75c74f1c36) (cherry picked from commit 5e06b7f4728e481f1c68425b4c06fbd99bdc32d6) (cherry picked from commit 5c12f8726dc1e72adf3714eed1b3b8af1c71465a) (cherry picked from commit b4f55adc4112a1dba1b5e9983e8a27be2b1d6e13) (cherry picked from commit 2c0254e9dab161a072520b315103a6aa19cb8a6b) (cherry picked from commit e652228d8a37fe164e0470a15cce1ca5a8b935a5) (cherry picked from commit 25974d84fc701f383ea01d6112bacd17c5709f15) (cherry picked from commit 03f4ce3a3474558cf62ef96c4a77b4428d325ae9) commit 4f509742f97fdb31dd4966127226da07d2282937 Author: Jaroslaw Grabowski Date: Thu Apr 22 10:02:16 2021 +0200 STAR-497 fix pycodestyle (#123) (cherry picked from commit d36c1104478b3afb1fb9403d004c7eb9a3425ab6) (cherry picked from commit f1eb25c7214e2d4803041cf29c22fd3b0d2ad263) (cherry picked from commit cf205511d4722bea92fab7391dc6809e86565fae) (cherry picked from commit 813359d7e8696fef7a595c854892db10dad00dbf) (cherry picked from commit 5294ca8aac8c9b77724596dbac6c77852d58f60d) (cherry picked from commit 9c927fb072ffb406e9264ab91bb63e460714b715) (cherry picked from commit 0a31f068621029ca2a565b2eddbd96a884a8ccae) (cherry picked from commit 004c2b96d2fcd29d8404d1375b789f7afac9e52a) commit eb6270b1982162a2e67b92aca95806f82ad305c5 Author: Jaroslaw Grabowski Date: Wed Apr 21 22:18:53 2021 +0200 STAR-453 ignore DSE index creation commands (#117) The commands end without an error with applied = true (in the same way as "IF EXIST" commands when the entity exists). A warning is returned to the caller. The index is not created. (cherry picked from commit b89547e26beb4109a2d1e6b6dbac9c70e8e224f1) (cherry picked from commit ef28215e04e378907c1ce6695ee32f80ebbf5c74) (cherry picked from commit 1b5bf825486dac89433e8fe8d08917325dec9cda) (cherry picked from commit 2dacdef70c138d148e51c439b3d4fe073af8c5ee) (cherry picked from commit 4e8687c87be8896e7c9a7581a00bd502ec837f44) (cherry picked from commit c1129a3c7afbe80c5da068aacccd26f4453cb623) (cherry picked from commit 3fda0487d60a5aaabe9316ad1347c7769cf62b93) (cherry picked from commit cf55300369cbaff9e63461a23bb3379dca92f570) commit 0a23d9ebfb3a17be60394a52c6ea24fd056d266a Author: Jaroslaw Grabowski Date: Wed Apr 21 14:46:36 2021 +0200 STAR-452: Add EverywhereStrategy (#111) * add EverywhereStrategy classes Files copied from DSE 6.8.11. * adjust ES to OSS Replication Strategy interface * port DB-589 (APOLLO-589) Only perform drop below RF check on decommission for non-partitioned keyspaces (cherry picked from commit 25609fc5dbbf9e2666611b06b389c2705110d6ad) (cherry picked from commit 9fd515678a2d4a40353db928655bf6b00ba30b25) (cherry picked from commit 5bd625c7691efd9b4e80b96597c80499659e9711) (cherry picked from commit a030f411de5bb46ebff3ee4978794bfc307dbeb2) (cherry picked from commit 554d303ffb4cc1eaa21d21bfefeccf1255523eb7) (cherry picked from commit e6d77ef8d5a39bd749e09eaff47c79c6a7da36ca) (cherry picked from commit b5478bc6ec2f0883554b5184bb9f0345a075eb9d) (cherry picked from commit e942e31a7b3e705fd654a274d3bc933313dea68c) commit a145a7f6d727c69165ba687a00d630366ee175d9 Author: Aleksandr Sorokoumov <918393+Gerrrr@users.noreply.github.com> Date: Wed Apr 21 11:24:08 2021 +0200 STAR-391: Reduce garbage when debug/trace is off in CompactionTask (#94) Co-authored-by: nitsanw (cherry picked from commit 03179457fd826cfdb35787dae3ef374be3700d13) (cherry picked from commit 9face890cf4c33545ace09bee1c7f7620ab6c6f3) (cherry picked from commit 28bd750b5d6084689f5edad48b2a767a1398809a) (cherry picked from commit 8f0cdc92d3fee93196ef556d5c5d2faf487c8f53) (cherry picked from commit 133fcf71f08d116569019b2c70e616d7f3f5ca37) (cherry picked from commit efe3e18c9b1aadf8449d04492cacc6a63185026c) (cherry picked from commit 6904100de0e5ee50b53e4b1b20575b351eaf422d) (cherry picked from commit 74b96a8f69c9af991c253144f29d6d930527ecbb) commit 0d1707e5bdfca1b1aa05f71334a212d0eabc0ba2 Author: Aleksandr Sorokoumov <918393+Gerrrr@users.noreply.github.com> Date: Tue Apr 20 18:10:05 2021 +0200 STAR-392: Reuse compaction task ID in compaction history (#93) Co-authored-by: Brandon Williams (cherry picked from commit 10cc26d4a39d5e7f73d1bda644d1399f8d8e1edd) (cherry picked from commit 3c272015bd1b0ddb9c8bd822fa500d8f99529527) (cherry picked from commit d713b7e659feffb53c0f6e6420f76c29458b6b04) (cherry picked from commit f481850a56ee15830425ac7204f38d977fe527aa) (cherry picked from commit 02367c1f3637d8ff9f83972d05110765313ce876) (cherry picked from commit 718d91ba263f0ec58fa8d023c89e88e7f69cdc90) (cherry picked from commit 7302f7156eb3bb7e5272fccc8ec1488c9691b21a) (cherry picked from commit a4858b1bcd2cc44d6c56d6e22ae01a56096d6006) commit eb853f3f3956451da212fb23a76a20251c832219 Author: Mike Adamson Date: Tue Mar 16 14:55:05 2021 +0000 STAR-196: Multi-node tests for SAI (cherry picked from commit 27e5f761cadc06c489dc071c86e633ae3de78ca1) (cherry picked from commit c2c1c18821ff404de6829291a536d4a6a3f609d6) (cherry picked from commit 9e9b9e97900b0c5dab8194b985a0ce00ecb6229c) (cherry picked from commit 5019c1f5a9ffe2632c0f977c1323507a4ab73d99) (cherry picked from commit eccc0a2e102b99b0785ba84a39f86f4ee5a172bb) (cherry picked from commit 86cd08a1548db014d206021b920ac370b48b2c2e) (cherry picked from commit 67e005caec158eddaecc661ce600941e9e157972) (cherry picked from commit 2231b1b6d98476eebd0498e2cb7553d1785334cb) commit de5d9284a1a3a592fc6570ea6770486eeb3d336d Author: Jaroslaw Grabowski Date: Tue Apr 20 11:11:38 2021 +0200 STAR-254 Port DSE custom CQL types (#107) * STAR-254 copy DSE 6.8.11 additional types * STAR-254 change DataStax copyrights to Apache * STAR-254 add esri-geometry-api 2.2.4 is currently the latest release. This is a different version then the one in DSE 6.8.11 (1.2.1). This one doesn't have the following dependencies: org.codehaus.jackson:jackson-core-asl and org.json:json. Instead it relies on com.fasterxml.jackson.core:jackson-core that is already present on C* classpath. * STAR-254 fix DateRangeUtil by moving some Apache Lucene™ methods. Using the exact same code as DSE ensures compatibility and at the same avoids dragging in the whole lucene jar to this project. * STAR-254 adjust to OSS serializer interface DateRangeSerializer was adjusted to the new interface. The instances are created without slicing a new ByteBuffer, unfortunately it implied manual offset counting (duh!). Geometric TypeSerializer uses a sliced buffer as using the ValueAccessor is not possible in this case (the buffer is eventually passed to an external library). * STAR-254 fix DateRangeSerializerTest Adjust to Cassandra test parametrization. * STAR-254 fix tests * STAR-254 fix GeometryIntegrationTest Allocation types parameters are removed, a dedicated tests for endianess are added instead. Fake (!) endianess logic for OGCGeometry was simplified. * STAR-254 fix geo tests by adjusting esri Polygon assertion was changed for the equivalent polygon. Esri is now more consistent in results of to* methods (polygon points are always listed clockwise). (cherry picked from commit af6e4e6325615afc57c1c416f55b462ab29e0401) (cherry picked from commit baade55bb28ca02e83e15756b809702ab46bb946) (cherry picked from commit 98d8c278caecf1592b1a3876edfbcf1db3421253) (cherry picked from commit 681d37593bb246c917725cbee5671f82829b9737) (cherry picked from commit d6e7fd2af1566faf5ff2fe1426a7a4366637e1e3) (cherry picked from commit 1c1c7f29adc652be64a97244cf475ebbb50b2a66) (cherry picked from commit 0ee1c6e8c20c0490f726dd9bbcbda32196ff4c2f) (cherry picked from commit 9525c5b4f82480901a1556a45d9dfae0ddec7181) commit fb1ab467b906a4854b4b68f8ad751d3df8582168 Author: jacek-lewandowski Date: Mon Apr 19 10:08:51 2021 +0200 STAR-247: TrieIndex SSTable format implementation [ddc51ba35cd6312eb2529e48dfd659cf1f39506f] STAR-247: Add bigtable unit test configuration [afbedc62b3cb097901d0eaf11b44900a75721f34] STAR-247: Disable potentiallyUnclosedCloseable in eclipse_compiler.properties [fcd91fb96e42b7010874386f852cebe7e5f20c8c] STAR-247: Minor methods and fields signature changes and non invasive refactorings [a808a209628547049076a64d889010acc14d08df] STAR-247: Move and rename BigTableZeroCopyWriter to SSTableZeroCopyWriter [ffe6182b3bd4dad5d1e4a6bd15ee294b4a97c426] STAR-247: Don't hardcode sets of components of an SSTable Use sets predefined per format instead [a4fb1f14133824cdf62e582775aa02dff0eacfa2] STAR-247: Refactor creation of SSTableReader, use factory where possible Previously SSTableReader was created usually in the following scheme: static openXXX method -> SSTableReaderBuilder -> factory method -> constructor Instead of that, we will use the scheme: factory method -> static method -> builder -> constructor The intermediate static method is left to keep the amount of changes minimal [c27c2b0a64a822aa01e7fe6a7f8335f29545acd7] STAR-247: Update SSTable version - introduce DSE-specific components Those components will be ignored for now, but we don't want to fail loading such SSTables. [f8efff91bd3108230b0b39855a6861eddbf384e9] STAR-247: Change contract of SSTR.keyAt and SSTFO.startPartition The previous contract states that those methods accept a key position in the primary index file. We need to change the contract as primary index is not an obvious component as BTI format does not have it. Instead, those methods will use a key position in data file. [d7a2268265ec939ad97d446f8223beafad88d6cc] STAR-247: Refactor usages of default file handle builder In particular, index file handle builder will take a component as an argument [ce68cb7fe3f90b15ce040ee776fabf25cc93e05f] STAR-247: Fix in Walker (add to STAR-75) [17ad86e983f7f44054eec2c8b6f2d4d15c5ec92f] STAR-247: Use RowIndexEntry where possible [13a6e62550eb0e22ebfe807f01022d7be6d36c37] STAR-247: Bring back ScrubIterator [7e09ae0c948f45783c4ea37aa35faecbb2667d19] STAR-247: Add SSTR.hasIndex() method [6ac6f1f7788627c867bcc850f6d558229d48097c] STAR-247: Refactor Scrubber [b3383b16b07fbd669e974260db1e599bc1e8a0c9] STAR-247: Refactor IndexSummaryManager and IndexSummaryRedistribution so that they are used only for BigTable SSTables [7d18ef94652a91564af3888ed37530a730bc4373] STAR-247: Refactor instance tidier [14429e6b5a3fb60fff483360628bd22f7113b4ae] STAR-247: Don't assume any format in CassandraStreamHeader [9cef1d57193b69202c7655614580f7b326b7bd27] STAR-247: Pull up maybeLogLargePartitionWarning [e80117d52146d6a0f62b13192d4050ec02da5e4e] STAR-247: Refactor SSTableWriter.openEarly [092d615af68f4844a17e6de5edac5936efb62026] STAR-247: Add SequentialWriter.requestSyncOnNextFlush [93ebc76e5a1439c8da99ab8e9188b4e5929a3d3e] STAR-247: Refactor FileHandle [4194c58ec022741276271fa90a9e5b92ce43ea47] STAR-247: Use aligned rebufferrer when it makes sense [b1464106170b512b86ca8ca1439de7652d45f86d] STAR-247: Refactor stats collection [dfa458278cfab40458cf7475c9e6b64285b01c88] STAR-247: Refactor SSTable verfication [36ef55edcf11edf7b1f16ca6b6cdfe14c86855b4] STAR-247: Various small changes and fixes [4cfcb8d3611d655e67d2dd2c30b4cd9ecfe5347b] STAR-247: Assume big table format for big table format specific tests [4090410205d150c69cccf6b94fbd83818d3771c7] STAR-247: Generify some tests to work regardless of sstable format being used [d6dd8ea54c7baf106ce03e8ffee7171a2d2dec83] STAR-247: Exhaust row iterator before going to the next partition in tests [87328ec17dfa907be63610e6310c874ec61d9ff1] STAR-247: Refactor ScrubTest [a693753e5a37fe9de39aa370e9342fb81e3b097e] STAR-247: Comment out a part of one test case in SSTableRewriterTest It is not clear what is the contract - see the comment in the code [d1fb135b4359a802fc556252c59aaf30180f9acf] STAR-247: Other small fixes and changes in tests [0c2095cd333c0638ddb92131f82d0048ad4972ed] STAR-247: Apply changes in OSS for use additional BTI implementation [02df51813dae659cfe40d28b8ae55a37fd6723be] STAR-247: Copy BTI implementation from DSE [10946deec4fcb8c43fbb6c3bfacdae50e99251a4] STAR-247: BTI specific tests [5427eb03d56e4425ba14d9b1aea899bca651f69d] STAR-247: BTI - minor adjustments to OSS - removed notions of reader constraint - removed notions of zero copy metadata - removed notions of file access type - adjusted generic types - renamed usage getSeekPosition to getFilePointer - changed usages of getTempKey to getKey [e80f503759e5e5942486980f3ab434506f95ae91] STAR-247: BTI - Add missing methods to TrieIndexFormat [71b374ebac8dbce78931472209940472b073226f] STAR-247: BTI - Fix inheritance and visibility of some classes [af3475a59b52c4b5b25efde749a6ce73a1ae969f] STAR-247: BTI - Fix TrieIndex version, add missing fields and remove unused [371db79b7464cc741140195a229cd89ff3e0ced5] STAR-247: BTI - Refactor usages of update fileFileHandle [9c7efbd9612081966c1ffee493174b53409a2b2e] STAR-247: BTI - remove index encryption from writer [8d2b9985e479f870c7f36b9ba57ded7a43b5b90f] STAR-247: BTI - refactor usages of some methods which have different names in OSS Also, when using default file handle builder for indexes, don't use compression [7ca897a51bd5bca9a5af1521e5508fc1150a188d] STAR-247: BTI - remove maybeLogLargePartitionWarning as it is defined in parent class [2efbf86d96cb1e77699472b479d1023f37d69832] STAR-247: BTI - update constructors of reader and writer [de983f6d1c41682d24444e589a9a3b73fbbb6b93] STAR-247: BTI - Update clone and internalOpen method, add a new internalOpen method to reader [efae04bca106455197d5874f80bf936dabafa374] STAR-247: BTI - Add bloom filter related methods [c67bc891df77c576e3989dee29dd0c6524f583d1] STAR-247: BTI - rename / remove methods from ReaderFactory [103bb1d5575962b5cdd64c8641b8db29812999cc] STAR-247: BTI - add a new open method for reader The method loads indexes and bloom filter Also add missing methods to the reader factory [4134ab07a33784a454401a228152c90acea6b37f] STAR-247: BTI - add missing clone methods to the reader [575eb7781021d41e92b2d3fd87ca632eb0d01b6c] STAR-247: BTI - add missing iterator and scanner creating methods [6d062208a614225b57d1fbebb44cf953417a157b] STAR-247: BTI - add missing implementations of index summary and cache related methods and remove methods which are not needed from reader [fade402a4f60c32792c3a339f653782541c4061d] STAR-247: BTI - refactor components verification - it is already implemented in the parent class [98da04446b77fbaa143a82c05aa35d0513f152c5] STAR-247: BTI - refactor getExactPosition methods and add missing filterFirst and FilterLast [3a7a1045678d3620cd5b950d0607daf72e6b580b] STAR-247: BTI - add one more implementation of coveredKeysIterator [78d6d9e7b30cf5013f6727cd8d6a5c588f533a4e] STAR-247: BTI - add/remove remaining missing/unused methods in reader [45b76a25aa3950ebb931752d933f63ef79fd0719] STAR-247: BTI - update writer factory [162d580c1f5a619b8afa4f25d9ff7d0c0cdc5f1b] STAR-247: BTI - tidier usage refactoring [0aa9cb1e438be5b7b34c094b2111fe6bbcadba8f] STAR-247: BTI - add missing methods to writer [819ed6a5b363e88e18f352735f35f57120c753b3] STAR-247: BTI - refactor TrieIndexScanner [9bdbcb3262c58dd0104605a2e5e8b8e38ac5587c] STAR-247: BTI - refactor PartitionIterator to conform the interface [0150544fcdc1e66cc47edddfdb273b133a40ee27] STAR-247: BTI - add/remove the remaining missing/unused methods [846414f267f94398c5a4f9fee338af66bfac1d0e] STAR-247: BTI - rename rowIndexCount to columnIndexCount in TrieIndexEntry [2f2539dd2dd6032b0b275ba1c93d201dad249877] STAR-247: BTI - renamed TRIE_INDEX to BTI [29caf8bf4637a924f39c929f3e44b45b4f1767c1] STAR-247: BTI - apply the remaining changes and import optimization [26f797aa86b15dc2bd1064fbf756f488bff88aa7] STAR-247: Review: revert changes in Refs/RefCounted/SelfRefCounted [213c5bdb7c98e97643d813645d944013f41fd006] STAR-247: Review: fix key iterators It was actually a bug that I implemented KeyIterator so that it can returns something quasi-meaningful other than what 'next' call can return. I removed some unused methods and whenever we needed something more than a key, we use PartitionIndexIterator. Also, I've introduced the notion of "preferred component" for keys. This means that SSTableFlushObserver.startPartition, PartitionIndexIterator.keyPosition and SSTableReader.keyAt have documented use of 'position' argument as key position in preferred component according to SSTable implementation. Big table uses index file and Trie Index uses data file. I paid attention to never explicitly say which component it is when operating on abstract level. [f9fedc21c2cfe3431ba5a971ab2c9c5c1b016c0d] STAR-247: Review: fix description in build.xml [d3406a60afeb12abb0de0fd3ad893b9b9fe35e89] STAR-247: Review: remove me and nc formats [fcc7bede4163a52a8f9dcc73b6396aec7e2661c7] STAR-247: Review: fixed releasing a ref According to the suggestion in review comments, regardless of whether closing component readers is successful or not, we continue and try to release a ref Errors are reported accordingly, stashed and eventually thrown after passing them to JVMStabilityInspector. [391073c0a530e1e679ad8631c32cc9ddfea57ee9] STAR-247: Review: fixed indentation in SSTableWriterTest [491b714acd0fd7ae23f7624733be5290ba531315] STAR-247: Review: Add a comment to ScrubTest [90b29e89c8c4216bb9cb041f6814d4960a292235] STAR-247: Review: Removed MetadataCollector.addKeyHash and DecoratedKey.hash2_64 [166450cb534752db5e4ad83a26c5fe2dd5868cea] STAR-247: Review: Remove some code related to DB-4159 [a9eb3fec6f02e5e7a01ec4908fbab06951255856] STAR-247: Review: Revert some code related to DB-1220 [81f19103d9a94f13c7ff0d28ef72703b0d86f082] STAR-247: Review: Reformat TrieIndexSSTableReader [e9f338b46c552172b1b21e50afd4aeff67d0042b] STAR-247: Review: Fix tidying SSTableReader [46aa1fad1846b7f0f45a9aa32e6a35a86205e445] STAR-247: Review: Add assertion on permitMatchPastLast [cb4f3d3fea130785ab6a6f1fc118c0ab530588d1] STAR-247: Review: Add a comment about moving validation to other place [3669667a79b64659464aea43efa9b54619ccc75e] STAR-247: Review: Remove supplier from simpleIterator [e3fbd93d94eb16f7c8d991065585984b75ff38c8] STAR-247: Review: Bring back PartitionIndex.dumpTrie and add some test to ensure coverage [9d3d07e140d4c3aef1562ee79a8e1465f3a9280c] STAR-247: Review: Removed one stupid todo [267447cf95e3c472c05a043f67e9965b094b5148] STAR-247: Review: moved keys.advance method to a separate line [8101526f2da23985ac0445e2062d37d90ef456be] STAR-247: Review: Updated JavaDoc in PartitionIndexIterator [ef9682b8710db624cdeff05533772370ada7e249] STAR-247: Review: Refactored keyAt and added SSTableReader.openKeyComponentReader [8163cbb5a034a1a7835fe87d4379e0c6d2eb507d] STAR-247: Review: Removed notions of NotInCacheException [1db85a55b5db307cef45d766377ba1eb31a37bf4] STAR-247: Review: Hopefully applied the rest of the comments [67ebe91a97e4265fd8136bf3d16a372d855d76e5] STAR-247: fix JMXCompatabilityTest [cf4a3d6ef0a99fd08ec013819df25c17843136b9] STAR-247: Post rebase changes: add missing method to TrieIndexFormat [ef6e7f299a947487c294c08ee792a3aba47d0612] STAR-247: Post rebase changes: fix compilation problem in ScrubTest (does not fix the test though) [c742b4264826494d1dc7ce4e8e7a57dace8771d1] STAR-247 fix ScrubTest [dd264390237b38744249f407a8e799d60d24fe3b] Remove unneeded generic type argument of RowIndexEntry [7b2b269b114e3da1c6c02c5fc7d7594f39402328] Remove the TPC-imposed Reader structure and replace it with iterators [7457283eafede5b45fcc3dad0597092074947199] Combine AbstractBigTableIterator into AbstractSSTableIterator [c73d9bf4e20df81dce2bf8d2bbf27be43d4f689f] Fix forgotten maybeValidateUnfiltered call [bd8d36910d10a229f89d125c87ffadb9714357c8] Fix skipping over blocks with no content and reporting slice end tomsbstone [9ea8af38209069fa71aa3402a3fa1155190610a4] Applied the remaining comments [f566e336df4c16a9995cdd9aeff90e1098db70f1] Applied the remaining comments (2) [d3aa9a46bd63757b27d926bcda21ca8fddbeb17d] Update LegacySSTableTest and add other formats to test [109f509d786cb98c4ad91ecde54a3dba1e927b36] Suppress resource in default handle builder methods Co-authored-by: Jaroslaw Grabowski Co-authored-by: Branimir Lambov Co-authored-by: Jacek Lewandowski (cherry picked from commit 48104fc268869dffd74eb8382a0daeb97469d97c) (cherry picked from commit 568d8ed4e8aaab851f8c74f3d39d56b20fadd02e) (cherry picked from commit 1d642978df157294759d01d1a728233a59c402a1) (cherry picked from commit 5ec0dc9d7c3016c8bf640d7a245d52a3b8c5c001) (cherry picked from commit 5feb3c7b48c9cf9165905b9d3eece1f29833cee5) (cherry picked from commit a73bc28519716efa873539d34a5b752111bde786) (cherry picked from commit 1f9fc75f80796cddaad039d2d51ac1c384cefbb7) (cherry picked from commit f48bce618fd88a38f0b53eff3e308ec25c793af6) commit ff2a187b7084f2158ad2499df0d91b4dca31926f Author: jacek-lewandowski Date: Fri Apr 16 10:23:29 2021 +0200 STAR-489: Add missing expected 'memtable' param of column family in cqlsh tests (cherry picked from commit dd5d05cba1f8e38fdf9dad5df1396a8268028074) (cherry picked from commit 96c1f63022dfd5da3405f973e0f7ed78eff8f98c) (cherry picked from commit 55dbb8e71d537141b244e7ccc98fec36c2237469) (cherry picked from commit 7727b11e3a8e9de9ec4bb2ae4d43c5b2b9f8bc9d) (cherry picked from commit a5b26bee8cf1f4da232fb1c960c625c6bb6d24a3) (cherry picked from commit 714ca60da22f85b40e1ef82bef6c8f319b33f677) (cherry picked from commit eefb6f44ee759bb4cc394f7055488c31dde0f2b3) (cherry picked from commit 732d86862aae6410ec1b152237cf8dd516a0ca64) commit 6285dc31961f220fb2ad1679bbc8bb27f23fe315 Author: Branimir Lambov Date: Wed Jun 9 10:59:30 2021 +0200 STAR-14: Memtable API Defines a pluggable memtable API which permits selection of memtable implementation, which can be selected in cassandra.yaml or individually per table. In addition to replacements of the typical memtable functionality, this also exposes some control over flushing and commit log operation that should make it possible to use persistent memtables. The main API is in Memtable.java, it includes support for altering the implementation of the current memtable functionality, but also expands it with control over flushing and commit log (where support for changed-data-capture and point-in-time restore, which rely on commit log, can be turned on or off), plus implementation of functionality that normally requires a flush (streaming/repair). The API comes with three implementations: the legacy ConcurrentSkipListMap, an initial blocking version of a trie-based one, plus a skeleton for a persistent memory solution that demonstates how one could be attached to effectively replace the storage subsystem. Also adds logged FlushReason to every flush request and improves JMH benchmarks. patch by Branimir Lambov; reviewed by Jakub Żytka (cherry picked from commit 04b02431b97beaa33366af25e03ce977e09d4c5c) (cherry picked from commit f0c0b5f5db1f2ad8dc663ae8bb0e0c7c50e15145) (cherry picked from commit 7d40a6c62ea527faf2ead0e9530d6bc0d749270c) (cherry picked from commit ebd54822fb6027c64a887458e2fefc0b527b1bd3) (cherry picked from commit fb7691bfac808497563f9398ac6e9b0859e6408e) (cherry picked from commit e630f65df05b70513ff8f799862e4eac6c29e150) (cherry picked from commit 319fdb74406a6cf459c139ff67c90aa3f02bda0d) (cherry picked from commit 1f9e56a62f83b21b8d4831833d32872320999289) STAR-14 Fix conflict in ColumnFamilyStoreTest with CASSANDRA-18118 STAR-14 Fix conflict in AtomicBTreePartitionMemtableAccountingTest with CASSANDRA-18125 STAR-14 Fix conflict in PartialCompactionsTest with CASSANDRA-18507 STAR 14 Fix conflict in ImportTest with CASSANDRA-14013 (cherry picked from commit 33b8c8fc93387f0eee25003f8aafc23b9a14d8d3) STAR-14 Fix AtomicBTreePartitionMemtableAccountingTest due to BTreePartionUpdater missing updates from CASSANDRA-18125 commit 89803e0012cbcb22b8c711c8b4ce7eac618ff631 Author: Aleksandr Sorokoumov <918393+Gerrrr@users.noreply.github.com> Date: Mon Mar 29 14:13:30 2021 +0200 STAR-374: Avoid redundant calls to RateLimiter in CompactionTask (#92) + throttle RateLimiter acquire in compaction operations Co-authored-by: nitsanw (cherry picked from commit a90d75cda3045929c98c30b2a00ce178e5b77227) (cherry picked from commit d70f083dd6db19eb814d1fbfce6af26579b68922) (cherry picked from commit e7984601cb62f6369148c81a0383fcdefbba4661) (cherry picked from commit 08744a5cc79898d216c19e179f0ba0b0401991dd) (cherry picked from commit d3357d2d04001d6617c39d7fdaf1ecc3a7ef851c) (cherry picked from commit 68e036031aac260acc28c10492929dcf0ba223aa) (cherry picked from commit 2289f20b7b9e07ccf5e6e57d2987e39146ee4d23) (cherry picked from commit 82cfca6f6058382c1298fbd994f5b5bf6a16634f) commit aaa1ef37fdcf14ea317c612f5e29f2e6fb9d234d Author: Mike Adamson Date: Thu Feb 18 13:11:46 2021 +0000 STAR-348 moved postings decoded to mark on postings reader close (#87) fixed up PostingsTest and others removed new line (cherry picked from commit fbdbec306ebd383aa3ac05236e93c21ce869cbce) (cherry picked from commit fabfca15c68c05ae9a5903767ca150db03b1a2d5) (cherry picked from commit cf7714e5be99b2ec77d34b44b4aff551516d7d09) (cherry picked from commit aea6839269dde72b2ee586e4ee4728e271773d4d) (cherry picked from commit 520d1fa339af36d423aeaa0a6e2eb42b014f3fd1) (cherry picked from commit e5a1da626247621412b33250ef6251992f38dfce) (cherry picked from commit 3056f403047412235e13f16aced385dcaef4ab89) (+1 squashed commit) Squashed commits: [7db49104811] STAR-360: Change compaction pause to be more specific to rewrite (cherry picked from commit 16b3e35db2a8d6c66bd5fead3c1240116a3ae8c4) (cherry picked from commit b913058c6cfe47624ced944255877630eb8e3c3f) (cherry picked from commit 641a5cfaa403fc60ec45110518ebc30810b5eec6) (cherry picked from commit 43bbcfecb5b7fdb83d6f0f0ce8986e2ea64d3250) (cherry picked from commit 6f80df980ff5e45b9ceaf2d0fee3edb25732ef2c) (cherry picked from commit 745de8275e7f40068a0331fdeebabea1a5498d19) (cherry picked from commit 96cd4544c84bb161e26b18c4f362e49ca28ab7a1) (+10 squashed commits) Squashed commits: [1ea07eff83f] STAR-361: Query all ranges at once for SAI distributed queries (#85) (cherry picked from commit 5ba3c6bd57ea437e670c9f3734d36cd5f75822db) (cherry picked from commit c7e186dd5648ac96472c1bf519d35082c37d6d33) (cherry picked from commit efc66d1c1f7a729ad14a359963d968290286b88c) (cherry picked from commit 7d5630f14aa4f915845e4f5738fab899477fa158) (cherry picked from commit 9e684651ddb72012692210fd04b03bb7b3f9161d) (cherry picked from commit 57103ba49a9f6d55d7b8df4e9102f837454f9da4) (cherry picked from commit 1ae42ea6e31b15bc0b7bf15c7aeefdd91768497c) [e0804e871a1] STAR-347: Fix FailureTest failure (cherry picked from commit f090eb6dce1e0e425a3d0ba209f8cb42e0ac540a) (cherry picked from commit ea105ce23fb105b0383c485617b5ddcec81a5a9d) (cherry picked from commit 8d187208d907bed215fc5baf5814f2125b540fc0) (cherry picked from commit 3c31759f010106c01413ab99152cd244f331b46d) (cherry picked from commit 88ee842b6508882870ad7ec45d429154041ec18f) (cherry picked from commit db0ab05488bd57f1bff0e3c0fe2c9499e58c6b96) (cherry picked from commit 5f2c269ce033757fdae3cd628324fb17e5feaa3b) [1229d4c1640] STAR-121: Storage Attached Index implementation and testing (cherry picked from commit 430c24026fb2a4afea735475408f0ac34944478f) (cherry picked from commit acbfeb80cf865ebe231146ad8ed54e03fef06cd3) (cherry picked from commit 54af59d4f31bff6ee7cbf8f670e7d82e94cc00d2) (cherry picked from commit 30425c49353bb88e756f1ea33cdb50d8932332b4) (cherry picked from commit 52d50e8ebfea9c6ee3cec6f170a1af26dd9a4310) (cherry picked from commit 94773ba88c6c7cf13e9f859fd75aef2d77f44cab) (cherry picked from commit ac2f66c401a45d0437198b6139efab74260c4ba5) [affe267a4ac] STAR-121: Test support for SAI (cherry picked from commit 6501503cf75a9a1d2f0ace5eb5d56bb9f28bb361) (cherry picked from commit f4d12eae47d3585897798df1b437bb4e4177391b) (cherry picked from commit c1cad34f418e4dec97bc02feb373980a1d6cd485) (cherry picked from commit 27ac10a37e9a7879baeb355b9a4cf5660f61a802) (cherry picked from commit 47becde9af455009334515e3e6032d3683e30838) (cherry picked from commit dc715880c196af11e2c4ab9206989aa75ff6e167) (cherry picked from commit d35ed8291f0a74fc2f21e04fb0ce11f9c7c3d263) [ad7cf475338] STAR-121: Key iteration and utility support for SAI (cherry picked from commit 36c018bca0a27c53c9c91a40bf0621e56fc16152) (cherry picked from commit daf5f712e38ca71357112d496ecebb370b47d90e) (cherry picked from commit 868c0a730f93a1b1f59c7c0b08345183fc4add3d) (cherry picked from commit 35f3d50cb868829c4edbac9553a355ff5e3561d3) (cherry picked from commit 97e0bf6e93c55b800803833a317d1583c65bfe9b) (cherry picked from commit 3dc326a2dc37906594892f9895fcdc1872c68d1c) (cherry picked from commit 2e7f9ed23ea03e5e2d068998262538933279b4e9) [c9d2d226334] STAR-121: On-disk component support for SAI (cherry picked from commit 89668cf917f0ea6a663f88e3081865644daae150) (cherry picked from commit 906845012771c0f10d6e051138883915f592ab8f) (cherry picked from commit c80316cada90998a0f558b582425c2ac40899d40) (cherry picked from commit 39f03c68628b9a5b7e584fbfea34bac67677f03b) (cherry picked from commit 3ebeb25918325894a05032b2b96bcde9482982be) (cherry picked from commit 7fd43c04c63caf932bddbdf42d001b1852027f18) (cherry picked from commit b3481de70372eb4e53fe051d7961311942f480d5) [6a3d288056a] STAR-121: Config support for SAI (cherry picked from commit 5db0b08fc6eee05fe914261600de51c1524685d0) (cherry picked from commit 24192212f5b456ee16aed5a4c065da43b393c46c) (cherry picked from commit 36d022e3d553f26769a7f7a7aa6fc067026e8799) (cherry picked from commit 37dd663047408b2a4d957d2f82ee6b03ae47212d) (cherry picked from commit 8af888ff6f402c8962c10bbe888878a99f32e178) (cherry picked from commit 7721e9b3aff762da6ebdd818420480e158ff7b13) (cherry picked from commit 7df7779fdf5b1af99cab15e7e2ecc8cf08b5d118) [bdaa00f65d4] STAR-121: Build dependencies for SAI (cherry picked from commit 32580250852ce46f2f71e0631241eb64e51753a8) (cherry picked from commit 77850e3a2d37db82a6f0b52aa420bf3970266dbb) (cherry picked from commit ee4575d13405ea40443b3ce3fa68ed76d7b00024) (cherry picked from commit 65e02b3a08844fff0aabfabe5103fc8b91d039bc) (cherry picked from commit dfd29dc4021deb64546caf4049b7ab7d401f5470) (cherry picked from commit c74939d2cd129041cf58c61e6af73af424a1038a) (cherry picked from commit 13fde650cbe142ea35ff769b0b1145d9f37802b9) [3e97eb5a4b7] STAR-121: Add index support to select statement restrictions (cherry picked from commit abe04b3f031d01f543eb8f4a1f0c3665e710e96f) (cherry picked from commit 0249d0559e3015cdbc232d05e4c88e22ee349c61) (cherry picked from commit bbc9ca4222bf7a2e57c7ab4ec777ce6ad0e49542) (cherry picked from commit 925a312b4627f7e41b1d8a2c7e9073be578fe1e6) (cherry picked from commit fa021cdca7ed7ce62df2dd2e9cfba9ebde5d6b52) (cherry picked from commit 3540945e1b6e91abc4867bdda56593c09c863548) (cherry picked from commit d39b35be49eb68467ce1b2de528bb90732caac34) [51c216c82fe] STAR-121: Index group API changes (cherry picked from commit eec063120d7c617a5bc7215234cbaab60d8b78bc) (cherry picked from commit 8363f8e4696741f0a31e072921e20344e7e5686b) (cherry picked from commit 20b4628d0de65195f00d6752fb3a1244cb2411a8) (cherry picked from commit f84f69453936b6047c6125c5f46cfe8c7a15241b) (cherry picked from commit a5df8862facfc3f650a02c37560992b9b516335b) (cherry picked from commit 1041d46eb22ba936440b818aeebb61a8e68a734c) (cherry picked from commit defc28e1be12a9cb430d23103f6c86fb31135687) (cherry picked from commit c6ed9b144713af5fe82dfb850c16f6af97719604) commit d0a9b58a279ed12ed05cc682a1ba1eecb5abe835 Author: jacek-lewandowski Date: Wed Mar 3 13:15:04 2021 +0100 STAR-75: On-disk Trie support Co-authored-by: Branimir Lambov Co-authored-by: Jacek Lewandowski Co-authored-by: Mike Adamson (cherry picked from commit 0a442fefc891c87e2f0947433361345981055bc9) (cherry picked from commit d5d5fe3c179ab3dc82d8d157d3cc0f532ffa6b1f) (cherry picked from commit 02613e3831e4ee30ef7572bf822461d7478fcc3c) (cherry picked from commit 23945afa8247cc000109ec973f22648421ae534a) (cherry picked from commit c3972d9d6c9687b39a2e2d9acaa15daba220f1ee) (cherry picked from commit 786598d113e246607d0dd9ee56ed50e8ab5e2033) (cherry picked from commit c6b432f6a1d78a0290a5e98c154ce679e9158877) (cherry picked from commit ee76a124555b2dd0e82eabce1e96b93d462804c0) commit adf619a83f0fed76045d987a7f86cdf75775aeac Author: Jakub Zytka Date: Mon Mar 1 18:10:08 2021 +0100 STAR-172: Update artifacts to be called dse-db rather than cassandra (cherry picked from commit d8708d5b62db0e8eceb4744d0948557932e6a8b4) (cherry picked from commit a98a1053bbbc988bd462c62b129659de0d230a87) (cherry picked from commit d5b41b45747bf622700442476045e14eb7a62e06) (cherry picked from commit b6910eea1d8fa1e9bc5432ab144a7c96d8ca8fbc) (cherry picked from commit 30ef052c152a6dc3ac6ff1e8fd82ac25ff2b94c7) (cherry picked from commit 972372dd20956c1d06f907a062eac74ab0189da3) (cherry picked from commit 22fb77caa1e1a7970a7667eea3867866aca275cb) (cherry picked from commit b44af21ed394e4be25237ade92e0d4d17c49f857) commit 86ad7e7dabeecdaac6c0719da1119b9dc7af54c9 Author: jacek-lewandowski Date: Thu Nov 26 10:11:29 2020 +0100 STAR-1: Read path optimizations Add MetadataCollectorBench Extract AbstractType serialization code into separate serializer class and refactor SerializationHeader to use that Add a new SSTable version which supports partition deletion presence marker and improved min/max Replaced `min/maxClusteringValues` with `coveredClustering` of `Slice` type Added `clusteringTypes` and `hasPartitionLevelDeletions` fields Other refactorings are related to the required code changes due to those new fields in `StatsMetadata`. No semantic changes were made in this commit except that slices intersection will never return `true` if any of the slices is reversed (start > end). Apply read path optimizations when static columns are queried Extend the usage of lower bound opttimizations (cherry picked from commit 1544bb6f4fb2d14e254c607265e5f76e9647d0bf) (cherry picked from commit 5541d1acae04fe86d8bf5836f5355aeba6fe36e5) (cherry picked from commit 2f8fbf1b95b6d79c732a73d1351479778ad47346) (cherry picked from commit f18ea817db22df75ef628886b67544ff68e9849e) (cherry picked from commit 539b752620db39680fab44f5da16dad506e2b866) (cherry picked from commit c32b4d0e6992e2141676c731e2ef48d44a86b7fe) (cherry picked from commit c1f2ed137035d58e7b845c1d179efb3e92c101c5) (cherry picked from commit bbd569080737f7daf8d78ae05f83e7a221f10fa6) commit f17947ae4f994ab3fb4b746af777b107d659c95f Author: Branimir Lambov Date: Mon Jan 11 16:02:12 2021 +0200 STAR-55: Provides the Trie interface with MemtableTrie implementation also includes functionality to merge, intersect and iterate on tries. patch by Branimir Lambov; reviewed by Jason Rutherglen and Jacek Lewandowski (cherry picked from commit 78d30623ff9e174ba3599380443eb86846416530) (cherry picked from commit a7c1e38b7ad03eded53589b5ec6d57062cb5952e) (cherry picked from commit ce17849dc0690a22a6b6688ca1cd899e590edb77) (cherry picked from commit ec150641c60cb78a43971e5e7e42b10345497147) (cherry picked from commit 9c4c76d7de65b824278bba4698ee0748a4a34df3) (cherry picked from commit 5139a58c41c777bbebc47825acb684f0c3ae9b2c) (cherry picked from commit f2778718c8c0e24b004b79c02556d0efe062cb63) (cherry picked from commit a539c5cd7edd24517e87125f85c720682ceb8a88) commit d2bd4519d098221e0546e0c3ca42e9112e1929c7 Author: jacek-lewandowski Date: Tue Nov 10 13:54:32 2020 +0100 STAR-15: Allow for other implementations of SSTableFormat [288dd9e05f037674319b4891cae21c46afe2df2b] STAR-15: Add getType method to SSTableFormat iface [ea53c0104fa704c28640b18ad0a977153269ff32] STAR-15: Remove getIndexSerializer from SSTableFormat This includes introduction of PartitionIndexIterator. Pull down serializer creation to the locations where it is needed. In other locations, we use PartitionIndexIterator abstract [0b18884f7f547248c0fca9c9767d45e691695840] STAR-15: Just moved RowIndexEntry to o.a.c.io.sstable.format.big [bfe955fc702eaba1cf2914642e42cc9d1fb10aef] STAR-15: Extract IndexState to a separate class [044bb012176ccca7300643308e3181ea681c7d84] STAR-15: Move AbstractSSTableIterator to o.a.c.io.sstable.format [9e946e674c6b33aec08174a41c7c7c27b64d052d] STAR-15: SSTableIterator and SSTableReversedIterator moved to a.o.c.io.sstable.format.big [ff574f7e3681e3217125b28de04cf005875dcd77] STAR-15: Refactoring 1. Generic RowIndexEntry was extracted 2. AbstractSSTableIterator is now typed by row index entry 3. In SSTableReader some methods were squashed and some were pulled down 4. Test were adjusted to just work (for now) [a9493886af5d1b5c9d4e748b70b0dd3646713023] STAR-15: Moved IndexInfo to o.a.c.io.sstable.format.big [fce4eea79084645c3b9b40c6d915dd88d8a29072] STAR-15: Move ColumnIndex to o.a.c.io.sstable.format.big [91ab2a1b4b271094a8a069fa913401f38798d12c] STAR-15: Get rid of explicit BigTableIndexRowEntry usage in Verifier [0e2ba2923c24c0e5a49b04ce9963d0f626da8f9e] STAR-15: Refactor KeyIterator to use PartitionIndexIterator Also fixed a possible bug in ReducingKeyIterator - when an exception occurred during instantiation of some SSTables passed to the constructor, the index files were left unclosed In SASIIndexBuilder exception transformation from IOE -> FSRE has been moved to the enclosing block - it does not seem to change the semantics In SSTableExport opening the SSTable was moved to the upper level as sstable instance was needed to create PartitionIndexIterator [82f88f6dfb95c3168884d5dc77df5374fb4c97d0] STAR-15: Refactor PartitionIndexIterator :...skipping... [288dd9e05f037674319b4891cae21c46afe2df2b] STAR-15: Add getType method to SSTableFormat iface [ea53c0104fa704c28640b18ad0a977153269ff32] STAR-15: Remove getIndexSerializer from SSTableFormat This includes introduction of PartitionIndexIterator. Pull down serializer creation to the locations where it is needed. In other locations, we use PartitionIndexIterator abstract [0b18884f7f547248c0fca9c9767d45e691695840] STAR-15: Just moved RowIndexEntry to o.a.c.io.sstable.format.big [bfe955fc702eaba1cf2914642e42cc9d1fb10aef] STAR-15: Extract IndexState to a separate class [044bb012176ccca7300643308e3181ea681c7d84] STAR-15: Move AbstractSSTableIterator to o.a.c.io.sstable.format [9e946e674c6b33aec08174a41c7c7c27b64d052d] STAR-15: SSTableIterator and SSTableReversedIterator moved to a.o.c.io.sstable.format.big [ff574f7e3681e3217125b28de04cf005875dcd77] STAR-15: Refactoring 1. Generic RowIndexEntry was extracted 2. AbstractSSTableIterator is now typed by row index entry 3. In SSTableReader some methods were squashed and some were pulled down 4. Test were adjusted to just work (for now) [a9493886af5d1b5c9d4e748b70b0dd3646713023] STAR-15: Moved IndexInfo to o.a.c.io.sstable.format.big [fce4eea79084645c3b9b40c6d915dd88d8a29072] STAR-15: Move ColumnIndex to o.a.c.io.sstable.format.big [91ab2a1b4b271094a8a069fa913401f38798d12c] STAR-15: Get rid of explicit BigTableIndexRowEntry usage in Verifier [0e2ba2923c24c0e5a49b04ce9963d0f626da8f9e] STAR-15: Refactor KeyIterator to use PartitionIndexIterator Also fixed a possible bug in ReducingKeyIterator - when an exception occurred during instantiation of some SSTables passed to the constructor, the index files were left unclosed In SASIIndexBuilder exception transformation from IOE -> FSRE has been moved to the enclosing block - it does not seem to change the semantics In SSTableExport opening the SSTable was moved to the upper level as sstable instance was needed to create PartitionIndexIterator [82f88f6dfb95c3168884d5dc77df5374fb4c97d0] STAR-15: Refactor PartitionIndexIterator Also make it possible to obtain PartitionIndexIterator directly from :...skipping... [288dd9e05f037674319b4891cae21c46afe2df2b] STAR-15: Add getType method to SSTableFormat iface [ea53c0104fa704c28640b18ad0a977153269ff32] STAR-15: Remove getIndexSerializer from SSTableFormat This includes introduction of PartitionIndexIterator. Pull down serializer creation to the locations where it is needed. In other locations, we use PartitionIndexIterator abstract [0b18884f7f547248c0fca9c9767d45e691695840] STAR-15: Just moved RowIndexEntry to o.a.c.io.sstable.format.big [bfe955fc702eaba1cf2914642e42cc9d1fb10aef] STAR-15: Extract IndexState to a separate class [044bb012176ccca7300643308e3181ea681c7d84] STAR-15: Move AbstractSSTableIterator to o.a.c.io.sstable.format [9e946e674c6b33aec08174a41c7c7c27b64d052d] STAR-15: SSTableIterator and SSTableReversedIterator moved to a.o.c.io.sstable.format.big [ff574f7e3681e3217125b28de04cf005875dcd77] STAR-15: Refactoring 1. Generic RowIndexEntry was extracted 2. AbstractSSTableIterator is now typed by row index entry 3. In SSTableReader some methods were squashed and some were pulled down 4. Test were adjusted to just work (for now) [a9493886af5d1b5c9d4e748b70b0dd3646713023] STAR-15: Moved IndexInfo to o.a.c.io.sstable.format.big [fce4eea79084645c3b9b40c6d915dd88d8a29072] STAR-15: Move ColumnIndex to o.a.c.io.sstable.format.big [91ab2a1b4b271094a8a069fa913401f38798d12c] STAR-15: Get rid of explicit BigTableIndexRowEntry usage in Verifier [0e2ba2923c24c0e5a49b04ce9963d0f626da8f9e] STAR-15: Refactor KeyIterator to use PartitionIndexIterator Also fixed a possible bug in ReducingKeyIterator - when an exception occurred during instantiation of some SSTables passed to the constructor, the index files were left unclosed In SASIIndexBuilder exception transformation from IOE -> FSRE has been moved to the enclosing block - it does not seem to change the semantics In SSTableExport opening the SSTable was moved to the upper level as sstable instance was needed to create PartitionIndexIterator [82f88f6dfb95c3168884d5dc77df5374fb4c97d0] STAR-15: Refactor PartitionIndexIterator Also make it possible to obtain PartitionIndexIterator directly from the SSTableReader.Factory :...skipping... [288dd9e05f037674319b4891cae21c46afe2df2b] STAR-15: Add getType method to SSTableFormat iface [ea53c0104fa704c28640b18ad0a977153269ff32] STAR-15: Remove getIndexSerializer from SSTableFormat This includes introduction of PartitionIndexIterator. Pull down serializer creation to the locations where it is needed. In other locations, we use PartitionIndexIterator abstract [0b18884f7f547248c0fca9c9767d45e691695840] STAR-15: Just moved RowIndexEntry to o.a.c.io.sstable.format.big [bfe955fc702eaba1cf2914642e42cc9d1fb10aef] STAR-15: Extract IndexState to a separate class [044bb012176ccca7300643308e3181ea681c7d84] STAR-15: Move AbstractSSTableIterator to o.a.c.io.sstable.format [9e946e674c6b33aec08174a41c7c7c27b64d052d] STAR-15: SSTableIterator and SSTableReversedIterator moved to a.o.c.io.sstable.format.big [ff574f7e3681e3217125b28de04cf005875dcd77] STAR-15: Refactoring 1. Generic RowIndexEntry was extracted 2. AbstractSSTableIterator is now typed by row index entry 3. In SSTableReader some methods were squashed and some were pulled down 4. Test were adjusted to just work (for now) [a9493886af5d1b5c9d4e748b70b0dd3646713023] STAR-15: Moved IndexInfo to o.a.c.io.sstable.format.big [fce4eea79084645c3b9b40c6d915dd88d8a29072] STAR-15: Move ColumnIndex to o.a.c.io.sstable.format.big [91ab2a1b4b271094a8a069fa913401f38798d12c] STAR-15: Get rid of explicit BigTableIndexRowEntry usage in Verifier [0e2ba2923c24c0e5a49b04ce9963d0f626da8f9e] STAR-15: Refactor KeyIterator to use PartitionIndexIterator Also fixed a possible bug in ReducingKeyIterator - when an exception occurred during instantiation of some SSTables passed to the constructor, the index files were left unclosed In SASIIndexBuilder exception transformation from IOE -> FSRE has been moved to the enclosing block - it does not seem to change the semantics In SSTableExport opening the SSTable was moved to the upper level as sstable instance was needed to create PartitionIndexIterator [82f88f6dfb95c3168884d5dc77df5374fb4c97d0] STAR-15: Refactor PartitionIndexIterator Also make it possible to obtain PartitionIndexIterator directly from the SSTableReader.Factory :...skipping... [288dd9e05f037674319b4891cae21c46afe2df2b] STAR-15: Add getType method to SSTableFormat iface [ea53c0104fa704c28640b18ad0a977153269ff32] STAR-15: Remove getIndexSerializer from SSTableFormat This includes introduction of PartitionIndexIterator. Pull down serializer creation to the locations where it is needed. In other locations, we use PartitionIndexIterator abstract [0b18884f7f547248c0fca9c9767d45e691695840] STAR-15: Just moved RowIndexEntry to o.a.c.io.sstable.format.big [bfe955fc702eaba1cf2914642e42cc9d1fb10aef] STAR-15: Extract IndexState to a separate class [044bb012176ccca7300643308e3181ea681c7d84] STAR-15: Move AbstractSSTableIterator to o.a.c.io.sstable.format [9e946e674c6b33aec08174a41c7c7c27b64d052d] STAR-15: SSTableIterator and SSTableReversedIterator moved to a.o.c.io.sstable.format.big [ff574f7e3681e3217125b28de04cf005875dcd77] STAR-15: Refactoring 1. Generic RowIndexEntry was extracted 2. AbstractSSTableIterator is now typed by row index entry 3. In SSTableReader some methods were squashed and some were pulled down 4. Test were adjusted to just work (for now) [a9493886af5d1b5c9d4e748b70b0dd3646713023] STAR-15: Moved IndexInfo to o.a.c.io.sstable.format.big [fce4eea79084645c3b9b40c6d915dd88d8a29072] STAR-15: Move ColumnIndex to o.a.c.io.sstable.format.big [91ab2a1b4b271094a8a069fa913401f38798d12c] STAR-15: Get rid of explicit BigTableIndexRowEntry usage in Verifier [0e2ba2923c24c0e5a49b04ce9963d0f626da8f9e] STAR-15: Refactor KeyIterator to use PartitionIndexIterator Also fixed a possible bug in ReducingKeyIterator - when an exception occurred during instantiation of some SSTables passed to the constructor, the index files were left unclosed In SASIIndexBuilder exception transformation from IOE -> FSRE has been moved to the enclosing block - it does not seem to change the semantics In SSTableExport opening the SSTable was moved to the upper level as sstable instance was needed to create PartitionIndexIterator [82f88f6dfb95c3168884d5dc77df5374fb4c97d0] STAR-15: Refactor PartitionIndexIterator Also make it possible to obtain PartitionIndexIterator directly from the SSTableReader.Factory Added methods to reset the iterator position and set the position to exact value :...skipping... [288dd9e05f037674319b4891cae21c46afe2df2b] STAR-15: Add getType method to SSTableFormat iface [ea53c0104fa704c28640b18ad0a977153269ff32] STAR-15: Remove getIndexSerializer from SSTableFormat This includes introduction of PartitionIndexIterator. Pull down serializer creation to the locations where it is needed. In other locations, we use PartitionIndexIterator abstract [0b18884f7f547248c0fca9c9767d45e691695840] STAR-15: Just moved RowIndexEntry to o.a.c.io.sstable.format.big [bfe955fc702eaba1cf2914642e42cc9d1fb10aef] STAR-15: Extract IndexState to a separate class [044bb012176ccca7300643308e3181ea681c7d84] STAR-15: Move AbstractSSTableIterator to o.a.c.io.sstable.format [9e946e674c6b33aec08174a41c7c7c27b64d052d] STAR-15: SSTableIterator and SSTableReversedIterator moved to a.o.c.io.sstable.format.big [ff574f7e3681e3217125b28de04cf005875dcd77] STAR-15: Refactoring 1. Generic RowIndexEntry was extracted 2. AbstractSSTableIterator is now typed by row index entry 3. In SSTableReader some methods were squashed and some were pulled down 4. Test were adjusted to just work (for now) [a9493886af5d1b5c9d4e748b70b0dd3646713023] STAR-15: Moved IndexInfo to o.a.c.io.sstable.format.big [fce4eea79084645c3b9b40c6d915dd88d8a29072] STAR-15: Move ColumnIndex to o.a.c.io.sstable.format.big [91ab2a1b4b271094a8a069fa913401f38798d12c] STAR-15: Get rid of explicit BigTableIndexRowEntry usage in Verifier [0e2ba2923c24c0e5a49b04ce9963d0f626da8f9e] STAR-15: Refactor KeyIterator to use PartitionIndexIterator Also fixed a possible bug in ReducingKeyIterator - when an exception occurred during instantiation of some SSTables passed to the constructor, the index files were left unclosed In SASIIndexBuilder exception transformation from IOE -> FSRE has been moved to the enclosing block - it does not seem to change the semantics In SSTableExport opening the SSTable was moved to the upper level as sstable instance was needed to create PartitionIndexIterator [82f88f6dfb95c3168884d5dc77df5374fb4c97d0] STAR-15: Refactor PartitionIndexIterator Also make it possible to obtain PartitionIndexIterator directly from the SSTableReader.Factory Added methods to reset the iterator position and set the position to exact value [402ce9f31313d3c963a845d04bd5e7ee93598119] STAR-15: Refactor explicit usages of BigTableRowIndexEntry.Serializer Use generic PartitionIndexIterator or KeyIterator instead :...skipping... [288dd9e05f037674319b4891cae21c46afe2df2b] STAR-15: Add getType method to SSTableFormat iface [ea53c0104fa704c28640b18ad0a977153269ff32] STAR-15: Remove getIndexSerializer from SSTableFormat This includes introduction of PartitionIndexIterator. Pull down serializer creation to the locations where it is needed. In other locations, we use PartitionIndexIterator abstract [0b18884f7f547248c0fca9c9767d45e691695840] STAR-15: Just moved RowIndexEntry to o.a.c.io.sstable.format.big [bfe955fc702eaba1cf2914642e42cc9d1fb10aef] STAR-15: Extract IndexState to a separate class [044bb012176ccca7300643308e3181ea681c7d84] STAR-15: Move AbstractSSTableIterator to o.a.c.io.sstable.format [9e946e674c6b33aec08174a41c7c7c27b64d052d] STAR-15: SSTableIterator and SSTableReversedIterator moved to a.o.c.io.sstable.format.big [ff574f7e3681e3217125b28de04cf005875dcd77] STAR-15: Refactoring 1. Generic RowIndexEntry was extracted 2. AbstractSSTableIterator is now typed by row index entry 3. In SSTableReader some methods were squashed and some were pulled down 4. Test were adjusted to just work (for now) [a9493886af5d1b5c9d4e748b70b0dd3646713023] STAR-15: Moved IndexInfo to o.a.c.io.sstable.format.big [fce4eea79084645c3b9b40c6d915dd88d8a29072] STAR-15: Move ColumnIndex to o.a.c.io.sstable.format.big [91ab2a1b4b271094a8a069fa913401f38798d12c] STAR-15: Get rid of explicit BigTableIndexRowEntry usage in Verifier [0e2ba2923c24c0e5a49b04ce9963d0f626da8f9e] STAR-15: Refactor KeyIterator to use PartitionIndexIterator Also fixed a possible bug in ReducingKeyIterator - when an exception occurred during instantiation of some SSTables passed to the constructor, the index files were left unclosed In SASIIndexBuilder exception transformation from IOE -> FSRE has been moved to the enclosing block - it does not seem to change the semantics In SSTableExport opening the SSTable was moved to the upper level as sstable instance was needed to create PartitionIndexIterator [82f88f6dfb95c3168884d5dc77df5374fb4c97d0] STAR-15: Refactor PartitionIndexIterator Also make it possible to obtain PartitionIndexIterator directly from the SSTableReader.Factory Added methods to reset the iterator position and set the position to exact value [402ce9f31313d3c963a845d04bd5e7ee93598119] STAR-15: Refactor explicit usages of BigTableRowIndexEntry.Serializer Use generic PartitionIndexIterator or KeyIterator instead [ad4535715f117f349c99d98403b3f7c23454d9fd] Apply review comments (cherry picked from commit 4520f47729752b78b5a83e71e29cb6ff45ceb599) (cherry picked from commit 3aba2f97e3a64b125a840a1b244e1bd0fd81cd64) (cherry picked from commit 7bb0ad043e80acfa4494a08c7404e3c44d2c1bf4) (cherry picked from commit d60fcdb514b46ec0b9826b5747dabf7cecda20de) (cherry picked from commit 38f9341f767b6a780d79ef4c8d181d9fa8689118) (cherry picked from commit 1d0cf887564772301e2a00277f4b01db87452ecf) (cherry picked from commit 99a6a3dd093396c7607b6f095002ecaba964cda1) (cherry picked from commit 998a3d25dce4beb87c610f6a24352e0d9c6bc917) commit 5c450334d5a3c71469ca5b4d532fad5bb96ffbea Author: jacek-lewandowski Date: Fri Nov 6 14:52:05 2020 +0100 CORE-92: Remove component number argument from getComparatorSize That parameter was misleading, those methods do not use it at all Also refactored a bit DynamicCompositeTypeTest (cherry picked from commit b0dd03ebc8371f67a7098b46c5de60443e4db5fe) (cherry picked from commit 0779d149067647f6e57edf83bdd192caeba11254) (cherry picked from commit 06d7704df6182355dce4b53f1f390f74804f77b9) (cherry picked from commit 13cab14419d5888c34b79234a8bcb3162b7db5bc) (cherry picked from commit feccc45a252d5841e82bb9c0798bfe2a486823f1) (cherry picked from commit d7ecc1b6c41758cd5b072771dbd0a60e5a270b51) (cherry picked from commit 8c770af6554c5996936b16266717a20d6ee5ea89) CORE-92: Remove misleading method readCollectionSize from CollectionSerializer That method is misleading because it accepts a byte buffer as an input source but unlike any other byte buffer reading method it does not shift the buffer position (cherry picked from commit 1ffd9284de0925e753c5a6ac45c4348699cf5f01) (cherry picked from commit 2be1e6051d094cfc6ec307de5f6853ff70655a64) (cherry picked from commit 9b9775bcc46bc243728c874db6d84a0ff9764739) (cherry picked from commit 748d23e2da79e918c30cdd336dd6e48492edf489) (cherry picked from commit 32a0d8a54409afdd7c86ff049d725819d22303d7) (cherry picked from commit 34d7f2ae2d89a4a81d4ba4a20330f20cd33a5b88) (cherry picked from commit 308f99224334f0921c9347f7e7fbb3c34bc475e2) CORE-93: Add ByteComparable, ByteSource and related stuff (cherry picked from commit c76194ed765811b14c8e85bb577a9945d41cc1bd) (cherry picked from commit 3cdb4c0148db34587708f306e67bf9277054bd07) (cherry picked from commit 8b066116dc7591888859b436b2dd5b7621c05d71) (cherry picked from commit 05ca0cd455f6af4293546f5757347b8c119d6b95) (cherry picked from commit b6fde8c652f933039124f6d4ecf0c4b2390c88fa) (cherry picked from commit 09e34d8f8d1e67711d5389270469b8e9540b9352) (cherry picked from commit 354cebc0f229406f7dade5083557634d448bf725) CORE-93: Add isValueLengthFixed method to AbstractType (cherry picked from commit d4f1de75d0e65fd52323c70fa1f8e85fe4ff8e16) (cherry picked from commit 3ea8c7a1f34775639b169d117c5459f10feb295d) (cherry picked from commit 6b5704a982eeff7b5dde6ddfddc5b14dc184ead7) (cherry picked from commit 7f1b5392486fbca55739a352d435db3755be8ffe) (cherry picked from commit 71941b9ceeaec5fdf9f855df27eb90957921ba11) (cherry picked from commit 89e4a747eff54c289faef6c9af63bc996fb703f9) (cherry picked from commit 7240cea3189309f756713e517046a87f2ad318c6) CORE-93: Add implementation of asComparableBytes to types (cherry picked from commit 783f1c1f1f5fb6739d61f099c19f571543701d19) (cherry picked from commit 851bd6ac2abbd8c11353616b5027a8cd75c8687a) (cherry picked from commit cfd23c81f94d253fbffe880fd8c643fc6cdd9dcd) (cherry picked from commit a1b2e082dc046f46f542ab19f5786d466a0ef7e3) (cherry picked from commit f59454145084e2d31e0a445b43b53c4b437c6b71) (cherry picked from commit 1435d6d63a6d9cd593715c3c922dd744dbd32920) (cherry picked from commit 4b1bb10e39d23478b97a7b05b20bf11d589a0e65) CORE-93: Add implementation of asComparableBytes to tokens (cherry picked from commit 73c8be5ca94a5a02aa88f68b0ec03d753b9b89b0) (cherry picked from commit fa557fd092ec459bd2eee28c8d56dac49bd478ce) (cherry picked from commit 7640c0494c9a8568c9468b87dfea2e30327b2a65) (cherry picked from commit 7d6a8f8fac91d3a9cbd87911f276c20690c6e655) (cherry picked from commit 7960b1379e09be3b879e6faf0b8d1e1faff60e9d) (cherry picked from commit 9bd931eb98e63529e14b4cb7f699dc8a7756d231) (cherry picked from commit beb7c463b4f4467b1d1392b0044ace58143daef5) CORE-93: Add implementation of asComparableBytes to partition position (cherry picked from commit d29ec5b1198738e1a966b560acdc170dfef609bb) (cherry picked from commit 59a4b3b46a7c9eb4709c4a2c38198ef386a31173) (cherry picked from commit 7b67b6c0079408d55a205f89a654be4d847af9bc) (cherry picked from commit cb63c85c2bcb3aaf8005c2a4d0779aa0df86e867) (cherry picked from commit fb9bcb7601c990021237f43a40263d9e35f6ddd9) (cherry picked from commit 3ca83b02c1905eae3bff94307496291fd5e7b2fb) (cherry picked from commit 8409c1eafff472afa4201ac25b97240cba9af2ad) CORE-93: Add implementation of asComparableBytes to clustering (cherry picked from commit 4187defa75496f64e0e743b0091431cd261fa87c) (cherry picked from commit 19b701e06145976fbd254095ff5962bafcaf9014) (cherry picked from commit 1540142bd2414fb830fb3d2dafabebb488326ac7) (cherry picked from commit f050c472e7c029d46b63180ebb058f3aa2571c24) (cherry picked from commit ef281a9f40f2d734c1ee4397845e52cbd74f2580) (cherry picked from commit 6e5b0398b7ebe0735d50ee261d76803106099f1a) (cherry picked from commit ef492bcadfe4c91036df81ba0b3de3553461ef6e) CORE-93: Add some tests (cherry picked from commit 9f6983dd580f0ab346aa9df13d69b38c9ecccec8) (cherry picked from commit 819733f4304e16a69ffe85bc2f99c48b9244b628) (cherry picked from commit 49c3f29ad4e6b402c77903e62136cb6345c51daa) (cherry picked from commit e9bfa0644c3ecd775dc8d51a9f034595fb6c1054) (cherry picked from commit a88c6b377197d3950b01ed28c479f2d1b2278f12) (cherry picked from commit 1f32a657810210daa7ada81c992ef753d46ddcc6) (cherry picked from commit 99b90a65225c94d13ae86e1f52a768483385cccd) STAR-17: Implements inverse byte-comparable translation Also adds ValueAccessor support, fixes minor TupleType and DynamicCompositeType problems and adds documentation. patch by Branimir Lambov; reviewed by Dimitar Dimitrov and Jacek Lewandowski (cherry picked from commit b54130cec96961330ee83cc52b6e6f439f466fa4) (cherry picked from commit cc13730c2131b6be9b2b982b878f1bba31de2419) (cherry picked from commit 4b0e2387e03f1b8f6797f0f6c1a22fd1cb7373c5) (cherry picked from commit 03743221ce77676380a05e13e2e026659cb36431) (cherry picked from commit f351613321f94d8fdc3961cbbfc16fa8b1563224) (cherry picked from commit dfa4aae9f7f6b7d22d0f81dbc76ad711361fedee) (cherry picked from commit 563bc4e70e851170637378870a5fbe17fef61519) (cherry picked from commit f4637f6f9ccc8a06257eb2aea7d5f9272392a461) commit 3c9a38f971b387deeb811810b1c6a9e1143d118a Author: Ruslan Fomkin Date: Wed Jun 16 15:19:53 2021 +0200 STAR-593 test patched LogReplicaSet.append The ported bug fix patch changes impelmentation of LogReplicaSet.append in the error case, however no tests exist. This commit adds tests to cover the error path. It also changes the version of JUnit to use assertThrow. (cherry picked from commit ca66e38f764963d9bf4c1ea8018cb8e4cdad0cef) (cherry picked from commit 07666e00d0c9c76bae051a66c766592dc30f8d9b) (cherry picked from commit 0a0d2f256c2a522d1297a998430fa8f86f03a269) (cherry picked from commit e2e99b6a11d74aa3e2b28b4ea13ad7124aff1c86) (cherry picked from commit c2a48aa5f465481d125d99038334245908298842) (cherry picked from commit d43f4360d99754f6f40036c4a013c174407aef6b) (cherry picked from commit 85586d39eabb5a7ae8307cc0b24b4aac122304b7) commit b179e84d9037c559564bd8fc2119cc02712fe9b4 Author: Stefania Alborghetti Date: Tue Feb 20 11:41:22 2018 +0800 STAR-593 Harden txn log files against exceptions Harden txn log files against exceptions when adding records and improve log messages Port of riptano/apollo@83b93bc434dc2f07371e848a9b24854403bf740e (cherry picked from commit 8175eb82b5f7c31ee77dfe13ac57eb40a647c775) (cherry picked from commit 5d5368c719ed39b47d49ca3ef1ef0810e80a7e15) (cherry picked from commit 2dcc0e0643085325acb40f9c389774745af57539) (cherry picked from commit 99903a76bdda79e8e96c2ee6d65d7f4a83d9f0bf) (cherry picked from commit e42cafa84a3fa7898ae3e88665da8e821951db1c) (cherry picked from commit 9124d959c0387eb246a436fbb728dc1124d7c394) (cherry picked from commit 9ac76db9eba19d594dcbb6d2e6508faa51d48aeb) commit 78c8f6f3256d964b55ee97db68eabf68ecb2e029 Author: dan jatnieks Date: Thu Jun 17 03:03:36 2021 -0700 STAR-583 Fix infinite loop when replaying a truncated commit log file… (#194) STAR-583 Fix infinite loop when replaying a truncated commit log file and truncation is tolerated Co-authored-by: Massimiliano Tomassi (cherry picked from commit 2bf6dd4e31c48a3567cb3cd4eb515da8005d8e10) (cherry picked from commit 6e7c1287a80e2484f8c4b09dd9775ed39edd1dde) (cherry picked from commit 43534e168d6ebac8e2641c4b46cf51d1a043a3bc) (cherry picked from commit b49fa6e10e1103407f69f6526c1104f2ae42765c) (cherry picked from commit 481bc161c31f2c48c8c1dd8350d64b82ea692a35) (cherry picked from commit 5ba077345a0c0258c87120ec7f133cb8fb6ef744) (cherry picked from commit 7834e6e131e708e179b93cfde4f141fdd16a4f2a) commit 28dcb3c23d9f156fb72600504e8d6be7a075247c Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Wed Jun 16 10:56:25 2021 +0200 STAR-748: Fix Scrubber so that it can work in case of broken index (#193) Fix scrubber so that it can work with broken indexes. If the index is broken, but the data file is ok, we can continue scrubbing and the index will be rebuilt. With this patch it will work even if we cannot access the initial position in the index. Also, the corrupted file will be obsoleted before finishing the rewriter so that it will not be attempted to move starts (and thus fail at the end as move starts requires correct indexes). There are also some explanations added to how canonical set of sstables is constructed and why Co-authored-by: Branimir Lambov (cherry picked from commit c0be3ae5cdd89fae0afcbb277322a466ece4de00) (cherry picked from commit b777dbebfbdf3006141da5825acd0a0c1b542840) (cherry picked from commit 7bd43ecff6b50919a492db490f4821a7a34ed064) (cherry picked from commit d5321b700da844372d4a13bd959df74ce71534dd) (cherry picked from commit e5f81f93d96a1e4ac8b83fdd647bc864933cab43) (cherry picked from commit 334a8fbc51ccb3c4dabf9ddb3e432dba7bd62b28) (cherry picked from commit d5e92565433cf2f0942b545962b8c4e71b9ba525) commit 82e681662803a48d25367ff4d55491d247fb5958 Author: Jacek Lewandowski <6516951+jacek-lewandowski@users.noreply.github.com> Date: Tue Jun 15 19:31:44 2021 +0200 STAR-745: Recreate BF on fp chance change and limit the total memory usage (#177) * STAR-745: Add BloomFilter global memory limit Added a memory limiter which is a global instance (static field in BloomFilter) which monitors the total memory used by all created (and deserialized) Bloom filters. When the limit is reached we return a dummy always-true filter with a log error message. * STAR-745: Recreate Bloom filter on startup In certain situations BF will be recreated on startup (for example, if FP chance changed more than the defined tolerance). Reviewed by: Daniel Jatnieks Co-authored-by: Stefania Alborghetti (cherry picked from commit c1a2595e556cb325fe915febd6c57b77955a09d8) (cherry picked from commit a75b01a98120636d051f53301f82b4fff08c9fb1) (cherry picked from commit de52f5c03f011cce5db1a878289347d1eab24d0e) (cherry picked from commit 1a0ae2c651390788d967a8491241c5e1edae16c3) (cherry picked from commit 191305d43da0f0bb5314b1b22937d21a6ef5ef28) (cherry picked from commit 140d446587c142b38e86f740052733fcd4e202ce) (cherry picked from commit ed14633364dea4b413fadc55ec395527d05ebe08) commit a5f211f7d82d22f6c5d55d0a8fc5119d948a5e47 Author: Jaroslaw Grabowski Date: Mon Jun 14 11:12:38 2021 +0200 STAR-571 fix *SnichTests (#197) Migrate to commit log initialization implemented by ServerTestUtils that is race free. Files were removed by the cleanup that followed CL init. (cherry picked from commit 37e2e3d52aae414c521110efdf5bf1bb20c37fad) (cherry picked from commit 5d10d16c88458a1ae655e59575658b7cd74c3ea6) (cherry picked from commit bb4a904a5a8c9957f2a32b326b96b6039c66cdf0) (cherry picked from commit fa0293a51ebfafe173cc526d929dbdfd7b1c78b7) (cherry picked from commit 2f5313d865390839ee87fd37762d006c97442a38) (cherry picked from commit 8889b2ace12bf748c2dbf638d112aba2497abec9) (cherry picked from commit f32f43d9fc696ce843070d6ec7d2322225652320) commit f066fb34b16fed511b6358c1105296283b409f42 Author: Ruslan Fomkin Date: Fri Jun 11 11:10:57 2021 +0200 STAR-582 avoid assertion when repairing 1 node cluster (#185) Porting patch DB-1511, riptano/apollo#627 Co-authored-by: Zhao Yang (cherry picked from commit 998eca09654c6f571c6479333d8bbef3a74a7707) (cherry picked from commit 18a8ce994d87ec242639fc46d3750198bd94b46a) (cherry picked from commit 2b37f32029682b72d5fa9b29d7ecb6836bffb98a) (cherry picked from commit 2a29739737358a31b93f35ed30f9e40afadbd1ef) (cherry picked from commit 3d9e3a63dd283e33015a8c05adaa5c9c011bf0a9) (cherry picked from commit a21dd64a4a09af6a77b4ed4977c08e04aaa58a88) (cherry picked from commit 804457116876c394f8e7ac8bb08672bf4c35e7c1) commit 473a3a1da0ebc73aae683bdd6d4e9e756895d841 Author: dan jatnieks Date: Thu Jun 10 17:27:39 2021 -0700 STAR-578 avoid copying EMPTY_STATIC_ROW to heap with offheap memtable(#523) (#188) patch by Zhao Yang; reviewed by Robert Stupp for DB-1375 Co-authored-by: Zhao Yang (cherry picked from commit d09bc6624a657b3e0a37ce7f4c1b5e6a0fc82c91) (cherry picked from commit 90d17f0473d67a3d58fb558e0a731a656472770b) (cherry picked from commit ec57374f696c94f921514ef90eb414a0463790c6) (cherry picked from commit 2203b385e72ed2776e122a35eec0ba9fb33c3450) (cherry picked from commit aa9d44dee0d9230214d3fe2515bf0188d8dbd5e0) (cherry picked from commit 8920afa75a9f9d12d98011fa77507fae74eb061b) (cherry picked from commit fe423325842f4bd9c7ba2dfde427b75f40a48cb5) commit 99538c7de9f1d86b9d64403f136e3fc80f092fcf Author: Jakub Zytka Date: Wed Jun 2 17:42:24 2021 +0200 STAR-692: protect Date type from overflows regardless of whether it is constructed from date string or from millis since epoch Co-authored-by: Ulises Cervino Beresi (cherry picked from commit 7bcbf8444f853c812fc21fa9187867f17ddd5c4c) (cherry picked from commit 2a9a89b6e7e7da4a3653da78449daa14ef462e00) (cherry picked from commit 7088fdbe53952148ca5241f9348beeb2693adc45) (cherry picked from commit f6f511ad2fd4a69a7b703589b19a24856d53383e) (cherry picked from commit e5c88c85354be095d6d421bab644ee251ddd8cc2) (cherry picked from commit 4df1c57dd9d7f3cd04fcf56bd9a53a411ff1746f) (cherry picked from commit bdea752e49c4ffeeb01b2fea395138f2a40fa57f) commit 19b3ffed96fc23170b3f4711a90be6236cc92f99 Author: Jakub Zytka Date: Wed Jun 2 17:28:44 2021 +0200 STAR-692: test exposing Date type overflow when using functions (cherry picked from commit 3dcd047cb2f6cd0a39fbe547372a73142d80e71c) (cherry picked from commit b8eb62df7dbedb58ab4ef5624cb75e2cb507c373) (cherry picked from commit d60018f2f77695af0317eda364e00e99fda8f3ec) (cherry picked from commit c0b108816468d7e7077ee1e1e77f2c3e8d6b55c3) (cherry picked from commit c982429337ed30cb98bb38cab691bb7fe928e407) (cherry picked from commit dcfcbf658d1b094f45e05f44f841cc60671ec4e8) (cherry picked from commit 21561e69c7eba46d7e58ce7e7a4c4a8d25e41747) commit b90f34538d300640df5389de84cd46ead1a38e24 Author: Zhao Yang Date: Wed Oct 4 05:14:44 2017 -0500 STAR-566: Test for wrap-around in estimatedKeysForRanges This ports the up-to-date version of the test introduced by DB-1157 patch by Zhao Yang; reviewed by Branimir Lambov ported by Branimir Lambov (cherry picked from commit beec1105e96f5f025d61bbe35088e518820c6e00) (cherry picked from commit 12db5848f0477db8945c79ec23537dcdb9881180) (cherry picked from commit 5454619c6d3b502b1fcb2c863024d188a037d70a) (cherry picked from commit edd498cbdf0d41ae7fffe09cd2994ffd546149ff) (cherry picked from commit 78482b32e0184e2d3f5ba8f7c9a4a32e94234f0d) (cherry picked from commit a0e302528f36a4bda28d4f6cedc9e9530f46bbcc) (cherry picked from commit 67c1934ea5d63c6e79828da44f82ac8ad034b87f) commit 732b56fb12af868af97b35cdaa44e0dd644439a3 Author: Stefania Date: Tue Aug 22 17:41:39 2017 +0800 STAR-563: Fix SIGSEGVs on aborted flush If a flush is aborted, e.g. by exception thrown by flushAllNonCFSBackedIndexesBlocking(), this was done by closing the flush writer, potentially concurrently with operations on it. The latter is unsafe and may cause writes to released memory. Fixed by adding an abort mechanism to the flush runnables. Port of DB-962 with an earlier commit. patch by Stefania Alborghetti; reviewed by Alex Petrov ported by Branimir Lambov; reviewed by Ruslan Fomkin (cherry picked from commit 8df735dd3b558dddeb927ac1f8b917ba4cab164c) (cherry picked from commit f90677c7649ea7288fc5bc58e747513f09f815af) (cherry picked from commit 7145708ab6394f3cca923edc6cb92ff2d2755ba1) (cherry picked from commit 1569402509e21dc72bd4d4ff13accd15c066330b) (cherry picked from commit 66f8c107b72e9f82158ba6961773f5d5dd2deaba) (cherry picked from commit aa959baad1c2264e7f96115abd9d6285dc27e5c0) (cherry picked from commit 5b3e2029af0438afdcac6e58d58c88e2f3e8cbc3) commit 0815d7ebe4ac6b16cb52a9a518213df690ce30d0 Author: Jaroslaw Grabowski Date: Wed Jun 2 11:10:44 2021 +0200 STAR-561 selectors should contain elements with same type (#169) * selectors should contain elements with same type Expressions like "SELECT [(CAST(pk AS BIGINT), CAST(ck AS BIGINT)), (t, WRITETIME(t))] FROM %s" can't be deserialized. If we try to extract the result by calling getRows we get: org.apache.cassandra.serializers.MarshalException: Expected 8 or 0 byte long This commit makes sure that all collection elements have the same type. Co-authored-by: Zhao Yang * test for Maps.getExactMapTypeIfKnow Maps are a bit different than Lists and Sets. Maps can't be selected without a type hint, which makes it impossible to exercise query execution path with `getExactMapTypeIfKnow`. That is why tests for "map SELECTs" are missing TermSelectionTest.java. The previous fixed `Maps.getExactMapTypeIfKnow` but it can't be tested in a similar way as for Lists and Sets. That's why commit adds a separate unit tests for `Maps.getExactMapTypeIfKnow`. Co-authored-by: Zhao Yang (cherry picked from commit 78b0117ac1a233de739ed8d17f72d300a81a5fea) (cherry picked from commit 68984d8f6932c3d5ea56ba5e834213c9cd5dcf11) (cherry picked from commit 5c359bf1bc4de6c44f8e7c61f64bdc73a9018394) (cherry picked from commit 84cacddf8bd85ab1c7577c0029071141d643e4d1) (cherry picked from commit 28be2a403eb72d7f8914ba9c02952c30c4dd0873) (cherry picked from commit 8dd8b4b5f51ce155dcf379627c2836cfe9293f49) (cherry picked from commit 7c57b4e288e40790714eba389260727091b288ea) commit 568353e3171e222229075c36bfb551d22e5663c7 Author: Jaroslaw Grabowski Date: Tue Jun 1 14:55:09 2021 +0200 STAR-571 fix nanoseconds overflowing in CommitLogService (#174) * remove dead code * add awaitSyncAt test The test verifies two cases when nanoseconds overflowing causes awaitSyncAt to never return (as described in the original ticket). * fix nanoseconds overflowing in CommitLogService * Use clock consistently in all CommitLogService implementations * Compare nanoseconds as a difference as advised in the docs: https://docs.oracle.com/javase/8/docs/api/java/lang/System.html#nanoTime * add abs values where the overflowed value may cause a negative result Co-authored-by: Sergio Bossa (cherry picked from commit 91db2d4c8490e6beff706e1ca7e7339d52cbd155) (cherry picked from commit a484e25631d6d7ac2cadfdf2b1a6008d44794894) (cherry picked from commit 358298e2dff44ad8af8fc3275af9509e61d76445) (cherry picked from commit 948713ed8d68d05dfd65689a2195ffa52c3de0ec) (cherry picked from commit f6774560ef8247d5ec9db3cbbd47cf3f6418c159) (cherry picked from commit 9bc1dd419286e5fb3e75551fe827bab6347ede43) (cherry picked from commit 9a7d9abe54186f98cfb376cf58692aff07ceb08e) commit 2bab6e9dc74ba813288e699d97826ec7199c5abe Author: Jaroslaw Grabowski Date: Tue Jun 1 10:32:21 2021 +0200 STAR-564 Check only MODIFY on base when updating table with MV (#171) If a user has only MODIFY permission on a table and there is a materialized view built on the same table an insert will fail with the following error: Unauthorized: Error from server: code=2100 [Unauthorized] Only base MODIFY permission is required to update base with MV. Co-authored-by: Zhao Yang (cherry picked from commit 53c5aa68693cf084bb36d56389c7ede485abbbf6) (cherry picked from commit c1eefd1f5b8d2332fa14ec7a6c6db862dbcced88) (cherry picked from commit f206725637d35a6a7e88b575da8a3e4dd81d7d78) (cherry picked from commit 0103fa1796b51363f5d4658c124b87429bd5b16e) (cherry picked from commit d82876e884467237a46bc29f63328f0cf318e973) (cherry picked from commit a67722feff0c9999c218e4c99ae061287ad8e27c) (cherry picked from commit 7e5db99a4915ebd3f14c4ea506a9877f43a3e778) commit 126ef9205b3d861adf78b1ad142350eddb041c77 Author: Jaroslaw Grabowski Date: Tue Jun 1 09:39:25 2021 +0200 STAR-565 Use the indexed item type as backing table key validator (#172) ... of 2i on collections Co-authored-by: Andrés de la Peña andres.de_la_pena_garcia@datastax.com (cherry picked from commit 07edf0d25def34c1c8b2e7f06b93a817692784dd) (cherry picked from commit 03dc51051c1a2188edbb715128c08ad285aa8a0b) (cherry picked from commit f13eb4d89c1ef55071ef0e673478d5c5b82d688a) (cherry picked from commit 1b78b5113632ea7ed6403330d4ddd4a3721c76ee) (cherry picked from commit 0296241f37238d2bbe289ac417be938e44282b33) (cherry picked from commit e9c36337dc65981f693814c077ba069fc0988ada) (cherry picked from commit 492d4886fa991c207fdede6742bb1c0f8a796658) commit 03ea1fa1e4ccd2ab4de10a6583eaeed1744adee8 Author: Ruslan Fomkin Date: Tue Jun 1 09:37:12 2021 +0200 STAR-572 Improve error message when altering MV (#176) Improve error message when altering an MV with default ttl > 0. Co-authored-by: Brandon Williams (cherry picked from commit 8296fe1319524480ae1910fde5f23c266c1642ed) (cherry picked from commit 7abf8639965e317a73f1193fc5624ea968400a67) (cherry picked from commit 0c57f27b6a5347c32945fe9179c335a2580c53bb) (cherry picked from commit 3dd85b7af83e038eeb75e931777437c9b2e5b582) (cherry picked from commit 0e7c8d7ced953f3286bf0a92228459536819a986) (cherry picked from commit 8b2e960e223f2c67b1b4c71e7a29d88e8e83d098) (cherry picked from commit c7c8a72345193996acc8b1b09c722294171cc63e) commit cd3eb8aa12fd4da5ae124b8e6ff3898c3f394a09 Author: Sergio Bossa Date: Thu Oct 5 16:51:06 2017 +0100 STAR-567: Add test from DB-1208 (bug itself is not present) (cherry picked from commit f4cca35aaa78780dbc344217c2ac0a70ca617679) (cherry picked from commit e9bd39486b3ed0cfa7062be1798efae8316cda0d) (cherry picked from commit c093b1bf38abc7a85d004e684c725f43cb04c72f) (cherry picked from commit 3101b3ae865941d391a6c184ed1330d38eb7d48f) (cherry picked from commit 3339ac1f663148802facf2bcdfdae6d126e8be21) (cherry picked from commit c6390c7fcd0dd18d503a30909d88685dd57b5b65) (cherry picked from commit 5bd9a64dcddac38bef084ca92e0ce6dddc81155b) commit bd951b553a124217568c796272911395977da12b Author: Branimir Lambov Date: Fri Jul 16 12:24:54 2021 +0300 Initial dependency and build fixes for Converged Cassandra Fix JMH libraries (cherry picked from commit 75ba24f4f71b2a1b98a38b0885d270702b64dc56) (cherry picked from commit 6a4fb715b441cb1920d0e9ccfc7cf34c0a33249a) Fix dependecies to match OSS and support M1 (#361) Update version of common-codec to 1.15 (#306) CNDB uses methods from the later version of org.apache.common.codec.binary.Hex and the methods are not available in the current version 1.9. Thus updating the version, which is already in use in DataStax, i.e., 1.15. (cherry picked from commit af50d9e7cd25a2f4f131bd44fce1286cd72641b8) STAR-1133: Increase max heap size for javadoc generation (cherry picked from commit 7084129fd4e411f3a8cfaf01c86c1dcdddcb5481) (cherry picked from commit 12d599c4bcfe0be7c6ded544dc23249782878e76) (cherry picked from commit 7530b3fb8401c2a1b759bc64d514cdc046c3fde2) (cherry picked from commit bffb0868014f09ba8d767020db76ecdf352ffcd5) commit b59183cf868573e483cc115eb56250c6f4a5adbd Author: Jacek Lewandowski Date: Thu Apr 28 06:18:19 2022 +0200 Save sstable id as string in activity table patch by Jacek Lewandowski; reviewed by Andrés de la Peña and Ekaterina Dimitrova for CASSANDRA-17585 (cherry picked from commit abecf659d91961de11568a7b6b6654a12d57388b) (cherry picked from commit 428abeec1a6c3e8b60ce443f02a942c3ce4549ad) (cherry picked from commit 137ee387a3dd83465e8150d9be9b97b406509180) commit ce40672133ba305a4c41a1608d43adbaea16c88f Author: Jacek Lewandowski Date: Mon Jan 24 11:51:13 2022 +0100 Implement sstable generation identifier as uuid Patch by Jacek Lewandowski; reviewed by Andrés de la Peña, Benjamin Lerer and Dan Jatnieks for CASSANDRA-17048 (cherry picked from commit e4689148628408d2c3237d5dff71bb01d5077041) (cherry picked from commit 6fb7ef85d02a63f8932fcabf149f82e8c8207851) (cherry picked from commit 7b0546a8c589ce7f22f828cab64c5b3a08becb58) (cherry picked from commit 8dcf00553cf2e6bac5d0c82b155af999dc2374db) commit 479b44900b89aa6d638d0dab348621a7a163023f Author: Jacek Lewandowski Date: Fri Jun 10 11:43:53 2022 +0200 Fix a race condition where a keyspace can be opened while it is being removed patch by Jacek Lewandowski; reviewed by Andrés de la Peña and Ekaterina Dimitrova for CASSANDRA 17658 (cherry picked from commit e923f2ed62e7c5bf0f8b19290e850b2c93f410af) (cherry picked from commit cc124db7ac68da7ccc4959e098c8236567bf1b27) commit 862f481e93eda283cf92b4846c79510b9b925a3d Author: Jacek Lewandowski Date: Mon Oct 17 14:36:49 2022 +0200 Fix a race condition where a keyspace can be opened while it is being removed patch by Jacek Lewandowski; reviewed by Andrés de la Peña and Ekaterina Dimitrova for CASSANDRA 17658 (cherry picked from commit 799892f84e57c050abc3e34a6aafa66cea76b7ec) (cherry picked from commit 73b89eedea03b562d069d1a3bba4276b718754f4) commit 4cb9bfa8634ec5237460980f007de71644825062 Author: Jacek Lewandowski Date: Tue Apr 26 14:43:49 2022 +0200 Add information info whether sstables are dropped or not to SchemaChangeListener patch by Jacek Lewandowski; reviewed by Alex Petrov for CASSANDRA-17582 (cherry picked from commit 774432c4b5fc65c41fa912e945a6e5df99ebc26b) (cherry picked from commit ec4f28899c9b21befd7094e37110026f9b1db7bf) (cherry picked from commit 579907a58ae5e98655aafc91e5bba3ef4a64e9b0) commit a5fe9a0be01f17df7bd0e038f0bd071a8d74d5d3 Author: Jacek Lewandowski Date: Mon Mar 28 10:44:03 2022 +0200 Refactor schema management to allow for schema source pluggability Patch by Jacek Lewandowski, reviewed by Alex Petrov for CASSANDRA-17044. (cherry picked from commit 0eb4bea49fe391b93582b4c49b9fd5c854353085) (cherry picked from commit 001611c035c30570cd4d811ad50520a52d8cbb8a) (cherry picked from commit 3ca9f09df5f26b51837ccab315ed099da6b86010) (cherry picked from commit 77eb6abfa40eba07320ddab0f6a11ec5d5ffa7e3) commit 870e1444c781df7c05462b7ec1e8a9903d2e9042 Author: Jacek Lewandowski Date: Wed Nov 3 08:59:42 2021 +0100 Relax schema synchronization when opening a keyspace Patch by Jacek Lewandowski; Reviewed by Branimir Lambov and Alex Petrov for CASSANDRA-17071. (cherry picked from commit 66ee4349087c177e3bae1eb841c8e31dcccdcfea) (cherry picked from commit 20711386fc072623ad10cb7f723e730e940c0b97) (cherry picked from commit da3408b1f63ea0e984db6c17fc1a660b690e0057) (cherry picked from commit 3722ff6b08f3fe7c1087b95b64e36eb97f79bb5a) commit b4b9f319832a2dfa3c07acfba0b3dbeeb71cb26b Author: vgali7 Date: Mon Jul 10 09:27:49 2023 -0400 Migrate Python optparse to argparse patch by Vineet Gali; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17914 commit 2584f4f0709df2c1031171ac9bbec71376adaba8 Author: Stefan Miklosovic Date: Tue Jul 18 11:26:19 2023 +0200 increment version to 4.0.12 commit a7bf85c4f363b8f34eaed33d3f04c6939a55f531 Author: Marianne Lyne Manaog Date: Fri Jul 14 23:25:42 2023 +0100 Add jvm/test args to microbench Patch by Marianne Lyne Manaog; reviewed by brandonwilliams and edimitrova for CASSANDRA-18658 commit 4f0865f769af96a526a99b0b988e2e53048201ca Merge: f8584b943e 32afc93b0d Author: Jacek Lewandowski Date: Thu Jul 13 15:33:11 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Moved jflex from runtime to build dependencies commit 32afc93b0dbb3bbd2a2f4834a0301bb4f07f0c60 Author: Jacek Lewandowski Date: Wed Jul 12 09:18:40 2023 +0200 Moved jflex from runtime to build dependencies Patch by Jacek Lewandowski; reviewed by Berenguer Blasi, Mick Semb Wever and Stefan Miklosovic for CASSANDRA-18664 commit f8584b943e7cd62ed4cb66ead2c9b4a8f1c7f8b5 Author: Stefan Miklosovic Date: Wed Jul 12 21:47:08 2023 +0200 Prepare debian changelog for 4.0.11 commit 92c045919915e2ebd68138ac3902444e368f1a06 Merge: 6fa548f9e9 7ec5126870 Author: Stefan Miklosovic Date: Wed Jul 12 12:19:25 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 7ec5126870f1bee853e63bfd98a73769c63abf3b Author: Stefan Miklosovic Date: Thu Jul 6 10:44:39 2023 +0200 Fix CAST function for float to decimal patch by Stefan Miklosovic; reviewed by Andres de la Peña and Brandon Williams for CASSANDRA-18647 commit 6fa548f9e97ff317307ba23411e570f292a23793 Author: Brandon Williams Date: Mon Jul 10 10:04:32 2023 -0500 Revert "Allow internal address to change with reconnecting snitches" This reverts commit b791644fda91b343a679bb0c2c1e33e594524636. Patch by brandonwilliams; reviewed by bereng for CASSANDRa-18560 commit 309e9b1295dd0ef2ff436fc41055e1f5a8e618a1 Merge: c10c7375c1 3f09baf1c2 Author: Brandon Williams Date: Fri Jul 7 10:45:19 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 3f09baf1c2e833cc3028be08ab753f6c48788210 Merge: 5326a39a1c 493d15fffa Author: Brandon Williams Date: Fri Jul 7 10:41:42 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 493d15fffa21e57fcaef7cfb2099cbaa3ab6bb47 Author: Brandon Williams Date: Thu Jul 6 15:50:26 2023 -0500 Upgrade to OWASP 8.3.1 Patch by brandonwilliams; reviewed by edimitrova for CASSANDRA-18650 commit c10c7375c15026651aa09589488d127070019bc0 Merge: 822589e382 5326a39a1c Author: Brandon Williams Date: Thu Jul 6 10:07:12 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 5326a39a1ccfff9141f49246b57f59c504f8d2d3 Merge: d302b837b2 7150cc5b99 Author: Brandon Williams Date: Thu Jul 6 09:59:13 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 7150cc5b99aafa6194dbaa5b8c73dc82e33c5279 Author: Brandon Williams Date: Wed Jul 5 11:11:12 2023 -0500 Suppress CVE-2023-34462 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18649 commit 822589e382f907c16f9d38638b3d5946dfdedc5c Merge: 0c79b2857a d302b837b2 Author: Brandon Williams Date: Thu Jul 6 09:50:50 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit d302b837b2b9a6854a98f83ecf687b08a58562fc Author: Brandon Williams Date: Wed Jul 5 10:57:39 2023 -0500 Suppress CVE-2022-45688 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18643 commit 0c79b2857ac86e1e6fa40ddbff04a8bb5c603375 Merge: 0a53770ddc 00cf31882b Author: Brandon Williams Date: Wed Jul 5 10:07:47 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 00cf31882beaf5e28dc600489f9ea8b69d1803df Author: Brandon Williams Date: Wed Jul 5 10:07:34 2023 -0500 Ninja fix my bad merge commit 0a53770ddc5dcee48d175a158505a9b2b42cf89b Merge: b099aae3f0 028ea02a4a Author: Stefan Miklosovic Date: Tue Jul 4 13:44:50 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 028ea02a4a877d115c748d3e179c6afb2328e430 Author: Vincent White Date: Wed Jan 31 06:59:28 2018 +0000 Remove unrepaired SSTables from garbage collection when only_purge_repaired_tombstones is true patch by Vincent White; reviewed by Stefan Miklosovic and Branimir Lambov for CASSANDRA-14204 Co-authored-by: Stefan Miklosovic commit b099aae3f00b4c47afc2129811c7fbc2ca30c82c Merge: 0621658acd b6fdc80456 Author: Stefan Miklosovic Date: Wed Jun 28 16:23:52 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit b6fdc8045621e5d1367261a62226793cb56f6076 Merge: 293f14cf36 4ea7bb25b4 Author: Stefan Miklosovic Date: Wed Jun 28 16:16:46 2023 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 4ea7bb25b4079e951202762aeaabe1d23be5303c Author: Stefan Miklosovic Date: Mon Apr 25 15:02:03 2022 +0200 Add support for AWS Ec2 IMDSv2 patch by Stefan Miklosovic; reviewed by Jacek Lewandowski and Brandon Williams for CASSANDRA-16555 Co-authored-by: Jacek Lewandowski Co-authored-by: Paul Rütter commit 0621658acddd3fbd4f931be31c94cacebce88953 Merge: 04fd3747cf 293f14cf36 Author: Brandon Williams Date: Wed Jun 28 06:04:49 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 293f14cf36b3340ca301e17d625830a5c5ecbfc8 Merge: 55dc5b43f1 de7b1584f8 Author: Brandon Williams Date: Wed Jun 28 06:04:26 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit de7b1584f888eb6acc7d10602e302475169472bb Author: Brandon Williams Date: Tue Jun 27 13:44:42 2023 -0500 Suppress CVE-2023-35116 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18630 commit 04fd3747cfe8f3dad6ac29873b2638db58b98d1d Merge: 776b33753b 55dc5b43f1 Author: Stefan Miklosovic Date: Mon Jun 26 14:05:12 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 55dc5b43f1d5f270cd9136f68bcf17cf9e083c05 Merge: b15a7da208 a00d1de441 Author: Stefan Miklosovic Date: Mon Jun 26 13:59:20 2023 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 commit a00d1de4414bb5ab57a27cb463af0e33af6e36c8 Author: Stefan Miklosovic Date: Thu Jun 22 14:23:49 2023 +0200 Pass taskId from CompactionTask to system.compaction_history patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-12183 commit 776b33753b4779efc33a80acf3feb7da767c1bbf Author: Brandon Williams Date: Fri Jun 23 09:43:09 2023 -0500 Upgrade snappy to 1.1.10.1 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18608 commit 72b1949157ce15d0daaade27172633c08d6f5acc Merge: bc3700da05 b15a7da208 Author: Brandon Williams Date: Mon Jun 26 06:05:24 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit b15a7da20863efb85ffa892b1d19891563cc44dd Merge: 254991750c d99bccd649 Author: Brandon Williams Date: Mon Jun 26 06:04:45 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit d99bccd6493ceb7aed4e05b05f81913ea876d855 Author: Brandon Williams Date: Thu Jun 22 11:55:09 2023 -0500 Suppress CVE-2023-34455, CVE-2023-34454, CVE-2023-34453 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18608 commit bc3700da056f50de15050e5c9e9f3e2043292e0c Merge: 76d11fdf51 254991750c Author: Brandon Williams Date: Wed Jun 21 18:00:01 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 254991750c4fc38b38a78c895b8254f3baf19cfa Author: Brandon Williams Date: Wed Jun 21 17:59:51 2023 -0500 Add missing changes commit 76d11fdf517c70abb88b5a78c90190f0f99ae820 Merge: 92d0a40c96 ece247b6ee Author: Brandon Williams Date: Wed Jun 21 06:26:43 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit ece247b6eea44452ba90a02901f9e379e18938d1 Merge: 64d953fdca e67fa69114 Author: Brandon Williams Date: Wed Jun 21 06:26:30 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit e67fa69114ca8ebd79e31eaced7d91913f91a0c2 Author: Stefan Podkowinski Date: Wed Mar 16 20:36:44 2016 +0100 Remove hard-coded SSL cipher suites and protocols patch by Stefan Podkowinski; reviewed by Robert Stupp for CASSANDRA-10508 backported in CASSANDRA-18575 by German Eichberger; reviewed by brandonwilliams commit 92d0a40c96c49a5fa8178ee03be170fd26b44340 Author: Maciej Sokol Date: Wed Jun 14 08:10:50 2023 +0200 Fix assertion error when describing mv as table patch by Maciej Sokol; reviewed by Brandon Williams and Stefan Miklosovic for CASSANDRA-18596 commit 49dba85ec812e4a9b59b85e1dc77735160771d53 Merge: c37bcbf7e9 64d953fdca Author: Stefan Miklosovic Date: Thu Jun 15 13:16:18 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 64d953fdcac91715898454a3cb04d932910a70b7 Author: Cameron Zemek Date: Mon Jun 12 00:17:02 2023 +0200 Wait for live endpoints in gossip waiting to settle patch by Cameron Zemek; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-18543 commit c37bcbf7e9d2b3c8ec4e93aa661d358c9f382edf Author: Daniel Jatnieks Date: Thu Jun 8 14:04:20 2023 -0500 Include TLSv1.2 in negotiatedProtocolMustBeAcceptedProtocolTest Add a comment about the use of disabled TLSv1.1 with JDK 8 and higher to negotiatedProtocolMustBeAcceptedProtocolTest Patch by Dan Jatnieks; reviewed by Ekaterina Dimitrova and Brandon Williams for CASSANDRA-18540 commit 2fcdaa5b76b77106108579cad60226130492e37d Merge: 5143bd81e8 1eccb2bc1f Author: Brandon Williams Date: Wed Jun 14 11:34:51 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 1eccb2bc1ff69817b2fc8d16a4707b64d8b514e7 Author: ningzi.zhan Date: Thu Jun 8 14:59:17 2023 -0700 Fix error message handling when trying to use CLUSTERING ORDER with non-clustering column Patch by Ningzi Zhan and Tomasz Lasica; reviewed by brandonwilliams, edimitrova and Maxwell Guo for CASSANDRA-17818 Co-Authored-By: Tomek Lasica commit 5143bd81e82c35ce686dd40860ec2aebe30aaf22 Author: Jacek Lewandowski Date: Fri May 26 09:29:37 2023 +0200 Track the amount of read data per row If an sstable is corrupted in a nasty way, we may read invalid cell sizes and try to read much more data for a row than we should. In rare scenarios this can lead even to OOMs. This simple fix adds tracking and limiting the amount of data that is read per row. Row has its size stored in preamble which can be used as a limit. If the deserialization code tries to read more than that, it will simply fail with EOF which will prevent more serious problems later. Patch by Jacek Lewandowski; reviewed by Berenguer Blasi and Maxwell Guo for CASSANDRA-18513 # Conflicts: # src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java commit c94eb0e37b4e9d64303d7a8b7beca24270e39c05 Merge: 2393bb16f3 7de3e61d75 Author: Bereng Date: Mon Jun 12 10:01:25 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Generate.sh -s param to skip autodetection of tests commit 7de3e61d7533da30c7b43f2d8dac2c5b89f488cc Merge: 3855c8a194 c91e2714b9 Author: Bereng Date: Mon Jun 12 10:00:24 2023 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Generate.sh -s param to skip autodetection of tests commit c91e2714b9c43f7959da1b698f2c4e28c94b9b6d Author: Bereng Date: Fri Jun 9 09:47:30 2023 +0200 Generate.sh -s param to skip autodetection of tests patch by Berenguer Blasi; reviewed by Andres de la Peña for CASSANDRA-18553 commit 2393bb16f308d4d51335a97851646973aceb8066 Merge: f368b9dc1e 3855c8a194 Author: Andrés de la Peña Date: Wed Jun 7 11:04:52 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 3855c8a194d1d9ac904429407cb9586a6178e2f0 Merge: 03da864bab 8d3c30dde4 Author: Andrés de la Peña Date: Wed Jun 7 11:04:01 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 8d3c30dde4b615835073089594003ca40a730992 Author: Derek Chen-Becker Date: Mon Nov 7 13:32:01 2022 -0700 Validate the comparison git branch when generating CircleCI configs Allow the user to use a flag to specify a non-default comparison git branch when detecting changed unit tests to repeat. Patch by Derek Chen-Becker; reviewed by Andrés de la Peña and Berenguer Blasi for CASSANDRA-17997 commit f368b9dc1efce57be073cd618eb81b6fa18c4f11 Merge: 5655a33bc0 03da864bab Author: Stefan Miklosovic Date: Tue Jun 6 16:10:16 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 03da864bab9740c067363a8dcce13db2bfd47ce2 Author: mghildiy Date: Fri May 19 11:31:45 2023 +0530 Add keyspace and table name to exception message during ColumnSubselection deserialization patch by Manish Ghildiyal; reviewed by Brandon Williams and Stefan Miklosovic for CASSANDRA-18346 commit 5655a33bc0fe53db940471fc1cb5972f13cda0de Author: rgitt Date: Sun May 21 16:46:18 2023 +0530 Fix Down nodes counter in nodetool describecluster patch by Goel Ranju; reviewed by Brandon Williams and Stefan Miklosovic for CASSANDRA-18512 commit be8cd8ab6bcdc066ed08c3a4371984f10c6fa322 Merge: fb41fb46fc 9c970cc117 Author: Brandon Williams Date: Mon Jun 5 05:58:40 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 9c970cc1171255ec2deb0eab20095cd12f99bbfb Merge: 572db17350 e28ef4089e Author: Brandon Williams Date: Mon Jun 5 05:58:01 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit e28ef4089ecbc2f64182de0ba5c9e9dea113195b Author: Brandon Williams Date: Fri Jun 2 06:13:08 2023 -0500 Suppress CVE-2023-2976 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18562 commit fb41fb46fcbc65f42cef23f6d1be33b1eca317c0 Merge: eb6a45fe6d 572db17350 Author: Brandon Williams Date: Thu Jun 1 15:30:04 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 572db1735070806a5a2dc4e5f1b7e3a88dac6a26 Merge: 1923e6955a f72df5449a Author: Brandon Williams Date: Thu Jun 1 15:29:58 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit f72df5449a996776c14b21952e69648e4843d980 Merge: 527c0cb3a0 aa671e8f07 Author: Brandon Williams Date: Thu Jun 1 15:29:53 2023 -0500 Merge branch 'cassandra-2.2' into cassandra-3.0 commit aa671e8f075a61e5fda538c640d9d2bbce9d8ff2 Author: Brandon Williams Date: Thu Jun 1 15:27:25 2023 -0500 remove dh_python use from Debian packaging Patch by brandonwilliams; reviewed by mck for CASSANDRA-18558 commit eb6a45fe6d34f9dd35462eec6eba3e5904562f96 Merge: 57293e8281 1923e6955a Author: Brandon Williams Date: Thu Jun 1 15:02:22 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 1923e6955a6f17bfbcf3a0104d80886a702dcdcd Merge: 43ec184391 527c0cb3a0 Author: Brandon Williams Date: Thu Jun 1 15:00:33 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 527c0cb3a0ed842e08685d9cb9954ddb99f93465 Author: Brandon Williams Date: Wed May 31 17:07:30 2023 -0500 remove dh_python use from Debian packaging Patch by brandonwilliams; reviewed by mck for CASSANDRA-18558 commit 57293e8281774f26ad4092b97ae6e17eb3f5a7e9 Merge: f66adb0278 43ec184391 Author: Jon Meredith Date: Wed May 31 12:02:29 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 43ec1843918aba9e81d3c2dc1433a1ef4740a51f Author: Doug Rohrer Date: Tue May 30 15:01:59 2023 -0600 Add support for JMX in the in-jvm dtest framework patch by Doug Rohrer; reviewed by Alex Petrov, Jon Meredith, Francisco Guerrero Hernandez for CASSANDRA-18511 commit f66adb0278aa56e8c2696db6f3fe24d311bf7b44 Author: Stefan Miklosovic Date: Tue May 23 10:58:17 2023 +0200 Remove unnecessary shuffling of GossipDigests in Gossiper#makeRandomGossipDigest This patch also fixes the bug when creating GossipDigests in case epState in Gossiper#makeRandomGossipDigest was null. Previously, if it was null, it used generation and maxVersion of the last non-null epState instead of zeros. patch by Cameron Zemek; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-18546 commit 71b79cfbe39485230012dc0b793a91a54731eecb Author: Mick Semb Wever Date: Mon May 29 13:36:16 2023 +0200 Increment version to 4.0.11 and 4.1.3 commit 4555b94345aa7183902bf1b5a54ad40fce13d432 Author: Mick Semb Wever Date: Fri May 26 13:40:51 2023 +0200 ninja-fix FullQueryLoggerTest, requires allow_nodetool_archive_command false to begin with CASSANDRA-18550 commit f0699bca74f6f829bdaffc56830d88fc8400e4e0 Author: Brandon Williams Date: Thu May 25 09:24:53 2023 -0500 Ninja fix GetFullQueryLogTest commit da77d3f729160e84fbab37666de99550be794265 Author: Mick Semb Wever Date: Thu May 25 14:45:07 2023 +0200 Prepare debian changelog for 4.0.10 commit aafb4d19448f12ce600dc4e84a5b181308825b32 Author: Marcus Eriksson Date: Wed Apr 12 09:17:50 2023 +0200 Improve nodetool enable{audit,fullquery}log Patch by marcuse; reviewed by Dinesh Joshi and Mick Semb Wever for CASSANDRA-18550 commit 54528bff2027d6c8cab845a83f2c03ad9441edbd Author: ningzi.zhan Date: Thu May 18 13:51:14 2023 -0700 Report network cache info in nodetool Patch by Ningzi Zhan; reviewed by brandonwilliams and smiklosovic for CASSANDRA-18400 commit dc6ad3f6b14b501286313ee04b0ee2682a81cd9f Merge: ff820290dd e1e88e5bc4 Author: Stefan Miklosovic Date: Fri May 19 15:10:20 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit e1e88e5bc4d4d31e4ae8b8af300d565fe7635fdf Merge: 3ca94d65d3 b828f7ea1b Author: Stefan Miklosovic Date: Fri May 19 15:06:47 2023 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 commit b828f7ea1b735586da388ddfee17f26685e20cef Author: Stefan Miklosovic Date: Thu May 18 11:32:18 2023 +0200 Pass down all contact points to driver for cassandra-stress patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18025 commit ff820290ddc4173ddac176d8f1435a9206dffa54 Merge: 1053e3b475 3ca94d65d3 Author: Stefan Miklosovic Date: Thu May 18 22:36:50 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 3ca94d65d3fd8f3f010f91e196b37608b08e0828 Author: Ivan Senic Date: Wed Dec 15 15:31:49 2021 +0100 Remove unnecessary String.format invocation in QueryProcessor when getting a prepared statement from cache patch by Ivan Senic; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17202 commit 1053e3b475829c7f2d0dc4ab59322d5819d1496a Author: Tobias Lindaaker Date: Wed May 17 10:42:59 2023 -0700 Partial compaction can resurrect deleted data patch by Tobias Lindaaker, Marcus Eriksson; reviewed by David Capwell, Marcus Eriksson for CASSANDRA-18507 commit 2bb634a339e5f165550a9617ea53a77072089db7 Merge: b791644fda f5c42f428c Author: Stefan Miklosovic Date: Wed May 17 12:02:03 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit f5c42f428c8669225f60c8fad783244eed69c4e5 Merge: b11021ee68 b8e21fb80a Author: Stefan Miklosovic Date: Wed May 17 12:01:04 2023 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 commit b8e21fb80afa6696374292dc99a480fac9788e64 Author: Stefan Miklosovic Date: Wed May 17 12:00:19 2023 +0200 Validate the existence of a datacenter in nodetool rebuild patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-14319 commit b791644fda91b343a679bb0c2c1e33e594524636 Author: Brandon Williams Date: Thu May 4 05:46:11 2023 -0500 Allow internal address to change with reconnecting snitches Patch by brandonwilliams; reviewed by bereng for CASSANDRA-16718 commit 2d40ee90194c881ec1032f527d4e8b76d8c0c80a Merge: 75194201f1 b11021ee68 Author: Brandon Williams Date: Mon May 15 11:22:19 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit b11021ee68e2ac607f12b32ac84a1ee59ee533db Merge: 9ab3a82ffa 785e291900 Author: Brandon Williams Date: Mon May 15 11:22:05 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 785e29190005baa8626554d8259f060916626ce0 Author: Brandon Williams Date: Mon May 15 11:21:48 2023 -0500 Increment version to 3.0.30 commit 75194201f1f06d120f246f6fad025ca5f672943d Author: Bernardo Botella Corbi Date: Mon Oct 10 09:08:16 2022 -0700 Fix quoting in toCqlString methods of UDTs and aggregates patch by Bernardo Botella Corbi, reviewed by Stefan Miklosovic, Benjamin Lerer and Yifan Cai for CASSANDRA-17918 commit ae995eb3d3cc1c98f61db0d071522b6f09443927 Author: Jon Meredith Date: Tue May 9 08:54:40 2023 -0600 NPE when deserializing malformed collections from client patch by Jon Meredith; reviewed by Caleb Rackliffe for CASSANDRA-18505 commit 3be79b5f0f7e7c87228f0bab3068f5fbf1f766e2 Merge: f7de633ff8 9ab3a82ffa Author: Stefan Miklosovic Date: Fri May 5 13:39:00 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 9ab3a82ffabc1ecfb749899898b19c4f9b9d7067 Merge: ee5b601ce7 087cffce63 Author: Stefan Miklosovic Date: Fri May 5 13:38:43 2023 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 087cffce636b63c12e328994d52bdf8f4ccc9750 Author: Stefan Miklosovic Date: Fri May 5 13:22:55 2023 +0200 Prepare debian changelog for 3.0.29 commit f7de633ff84589657166cb73f7aff7906b5f5b0e Merge: 682ae0c64c ee5b601ce7 Author: Brandon Williams Date: Fri May 5 06:09:35 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit ee5b601ce7f205ef303e31c4240f69c21e583599 Author: Brandon Williams Date: Fri May 5 05:57:31 2023 -0500 Increment version to 3.11.16 commit 682ae0c64c9159c6c884252820ce1236fde39e17 Merge: dba4162666 b6b71c598e Author: Brandon Williams Date: Thu May 4 10:24:20 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit b6b71c598eef1a01c52cef0e7fa54c6e24d39095 Merge: 90d0857d34 4a62757624 Author: Brandon Williams Date: Thu May 4 10:22:25 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit dba41626669a66dcd78eb7b097988e1472b4a65c Merge: 65c99bfc42 aa73bc468a Author: Brandon Williams Date: Thu May 4 10:23:47 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit aa73bc468aa58805f406c756647384b8608e3c42 Merge: 90d0857d34 4a62757624 Author: Brandon Williams Date: Thu May 4 10:22:25 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 4a62757624d120a0e493d9d39bcd9bf8282b4f58 Author: Brandon Williams Date: Thu May 4 09:12:18 2023 -0500 Suppress CVE-2023-2251 Patch by brandonwilliams; reviewed by smiklosovic for CASSANDRA-18497 commit 65c99bfc42015ccfbcbc8963b3fb3c2a761ea053 Author: Henrik Ingo Date: Wed Apr 19 17:34:23 2023 +0300 Improve 'Not enough space for compaction' logging messages Adds separate WARN level messages to the check itself in Directories. patch by Henrik Ingo; reviewed by Mick Semb Wever, Brad Schoening, Claude Warren, Kan Maung, maxwellguo for CASSANDRA-18260 commit 1929550fc07ff938a7ddce36e55df16e46501801 Merge: b183e1f0de 90d0857d34 Author: Mick Semb Wever Date: Sat Apr 29 18:24:24 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 90d0857d341a1735edfa9450c4488372b3b44f87 Author: Steve Putala Date: Mon Apr 17 22:18:58 2023 -0400 fix typos in data modeling and getting started docs patch by Steve Putala; reviewed by Mick Semb Wever ref: https://github.com/apache/cassandra/pull/2280 commit b183e1f0ded43ba6f81760112da40c6e2568c3ff Merge: c4a305d17a 6cdcf5e56a Author: Stefan Miklosovic Date: Fri Apr 28 12:41:14 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 6cdcf5e56a77cf40c251125d68856a614eccbc53 Author: Stefan Miklosovic Date: Fri Apr 28 12:30:03 2023 +0200 Prepare debian changelog for 3.11.15 commit c4a305d17a81a0e04b7f9011e8b574f6919fda05 Merge: 45938e296e c74c9bb037 Author: Stefan Miklosovic Date: Thu Apr 27 11:17:47 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit c74c9bb037939882d4eb355b8ec9fe14212cbdb6 Merge: 0f3a990dd2 db78e746d7 Author: Stefan Miklosovic Date: Thu Apr 27 11:15:09 2023 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 commit db78e746d7e2fb9a2aa4c10e1322c6c5033f006b Author: Stefan Miklosovic Date: Mon Apr 17 12:20:21 2023 +0200 Do not remove SSTables when cause of FSReadError is OutOfMemoryError while using best_effort disk failure policy patch by Stefan Miklosovic; reviewed by Brandon Williams and Maxwell Guo for CASSANDRA-18336 commit 45938e296edada25c40037cad31d9a547397061f Merge: 5bcee5c06a 0f3a990dd2 Author: Stefan Miklosovic Date: Thu Apr 27 08:53:09 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 0f3a990dd2bc9532752a711a49a131c63d52e3c7 Author: Maxim Chanturiay Date: Sun Nov 6 17:27:29 2022 +0200 Fix the capital P usage in the CQL parser patch by Maxim Chanturiay; reviewed by Stefan Miklosovic and Benjamin Lerer for CASSANDRA-17919 commit 5bcee5c06a8c90b0236d3970fe24bdda4360c304 Merge: 09c1e67598 81e78192cb Author: Stefan Miklosovic Date: Wed Apr 26 13:37:51 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 81e78192cb229b0ddbf587c94701ca4d9670a51b Merge: 08b9471a1e 4f348786bd Author: Stefan Miklosovic Date: Wed Apr 26 13:34:43 2023 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 4f348786bd6d72f4402134a29ddb3355b1fade61 Author: Stefan Miklosovic Date: Thu Apr 20 11:09:06 2023 +0200 Do not remove truncated_at entry in system.local while dropping an index patch by Stefan Miklosovic; reviewed by Caleb Rackliffe for CASSANDRA-18105 commit 09c1e67598d7a0dc7da7bebee358fbc96d84680a Author: Jon Meredith Date: Fri Apr 21 15:24:58 2023 -0600 Incremental repairs fail on mixed IPv4/v6 addresses serializing SyncRequest patch by Jon Meredith; reviewed by Caleb Rackliffe for CASSANDRA-18474 commit cd9bed0aeadd94136a8a6c6ed284cc4684b0666c Author: Jon Meredith Date: Thu Apr 20 14:17:36 2023 -0600 Deadlock updating sstable metadata if disk boundaries need reloading patch by Jon Meredith; reviewed by Marcus Eriksson for CASSANDRA-18443 commit 3f67827387cd2fa790c5587d17105fcadeda451c Merge: 2fce3025c4 08b9471a1e Author: Andrés de la Peña Date: Mon Apr 17 14:48:27 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 08b9471a1ecfcb326ba4fbfdeed5d60aefcc671b Author: Andrés de la Peña Date: Mon Apr 17 11:23:12 2023 +0100 Fix sstable_count metric missing from tablestats json/yaml output patch by Andrés de la Peña; reviewed by Brandon Williams and Ekaterina Dimitrova for CASSANDRA-18448 commit 2fce3025c448316720d9cf7447fe7d1d34e17625 Author: Maciej Sokol Date: Mon Nov 7 10:39:26 2022 +0100 Fix nested selection of reversed collections patch by Maciej Sokol; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-17913 commit f2427a0c861aa9badab2b7532810e25b51812750 Author: Stefan Miklosovic Date: Fri Apr 14 11:20:04 2023 +0200 Increment version to 4.0.10 commit e9f8f2efa2ba75f223f31ca6801aff3fe2964745 Author: Stefan Miklosovic Date: Tue Apr 11 09:31:56 2023 +0200 Prepare debian changelog for 4.0.9 commit 7d84ad936622454afb01424d8a3c6db511654d60 Author: Brandon Williams Date: Fri Apr 7 13:42:18 2023 -0500 Revert "fix error in conflict edit" This reverts commit 21c3e9731ceb773ef6010dcfea9c8ede3204fda2. commit 5d315f869dcdda73f23f85e178a724391a51dccd Merge: 21c3e9731c 83dd275d5d Author: Brandon Williams Date: Fri Apr 7 13:41:56 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 21c3e9731ceb773ef6010dcfea9c8ede3204fda2 Author: Lorina Poland Date: Fri Apr 7 11:10:46 2023 -0700 fix error in conflict edit commit 9a3a37c5b3f4d92c10be3ce072c6b16251ad9eed Author: Lorina Poland Date: Tue Feb 14 10:53:29 2023 -0800 DOC - Added :stem: latexmath to Data Modeling page to fix display of equations patch by Lorina Poland; reviewed by Erick Ramirez for CASSANDRA-16906 commit 30639d8852825c14a98fb3a7ba46c89b559d56bf Author: Brandon Williams Date: Thu Mar 30 10:14:07 2023 -0500 Suppress CVE-2022-45688 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18389 commit 03be6d792679d8d4be560c9af4cba72e6b061f35 Author: Stefan Miklosovic Date: Thu Apr 6 10:53:29 2023 +0200 update zstd-jni library to version 1.5.5 patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18429 commit 83dd275d5d824abba4ced3cfbdada711f2e7bf63 Author: Lorina Poland Date: Tue Feb 14 10:53:29 2023 -0800 DOC - Added :stem: latexmath to Data Modeling page to fix display of equations patch by Lorina Poland; reviewed by Erick Ramirez for CASSANDRA-16906 commit 99f62b7338fc97a150e52e285f4eee3c636d6637 Author: Stefan Miklosovic Date: Tue Apr 4 09:29:07 2023 +0200 Prepare debian changelog for 4.0.9 commit f6fce7ab51e41dfd8d9584ce4c8dd35e0ed61598 Author: Josh McKenzie Date: Wed Mar 15 11:53:07 2023 -0400 Improve debuggability and correctness of ref detection This patch is a backport of CASSANDRA-17205 and also raises logging level to 'error' from 'warn' in StrongLeakDetector. Patch by jmckenzie; reviewed by smiklosovic for CASSANDRA-18332 commit b3256f5bf22c99a8f5410634dd41ca1643a34c1c Author: Maxim Muzafarov Date: Wed Mar 1 00:35:36 2023 +0100 Fix the number of running trees per thread for LongBTreeTest hanging test patch by Maxim Muzafarov; reviewed by Mick Semb Wever for CASSANDRA-18131 commit d41e69a6527fd42d0c17ac4f5a0ecb81d00837ca Author: Brandon Williams Date: Thu Mar 30 10:14:07 2023 -0500 Suppress CVE-2022-45688 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18389 commit ed904a72f41ed55acec9a26cff4f00a5fb1384c6 Merge: 853ae8c840 f9364e4f44 Author: Brandon Williams Date: Mon Apr 3 05:58:42 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit f9364e4f44b0d37905713a76d088218645ab72fb Author: Brandon Williams Date: Thu Mar 30 10:14:07 2023 -0500 Suppress CVE-2022-45688 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18389 commit 853ae8c84049be875921a40c9d5924724cc72792 Author: Caleb Rackliffe Date: Thu Mar 30 13:07:16 2023 -0500 Avoid loading the preferred IP for BulkLoader streaming patch by Caleb Rackliffe; reviewed by Jon Meredith for CASSANDRA-18370 Co-authored-by: Caleb Rackliffe Co-authored-by: Jon Meredith commit 4b9c18235ae10acc266d92ffe6c54936a789904a Merge: 5f61645778 ee586b53c2 Author: Jacek Lewandowski Date: Fri Mar 31 17:30:19 2023 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Save host id to system.local and flush immediately after startup commit ee586b53c2cb7178c10841dc02f4023c64dda10e Merge: 955cf156c8 f96659c530 Author: Jacek Lewandowski Date: Fri Mar 31 17:30:18 2023 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Save host id to system.local and flush immediately after startup Remove asf.yaml from non-trunk branches Notify the corresponding JIRA issue as soon as the PR is raised commit f96659c5306e62666e21c371c2ded646dd51672b Author: Jacek Lewandowski Date: Tue Mar 14 14:29:39 2023 +0100 Save host id to system.local and flush immediately after startup patch by Adriano Bonacin and Jacek Lewandowski; reviewed by Stefan Miklosovic and Sam Tunnicliffe for CASSANDRA-18153 commit 20bf2273a0b5024a78b9564d2f23536cf18650a5 Author: Brandon Williams Date: Wed Mar 29 06:32:05 2023 -0500 Remove asf.yaml from non-trunk branches commit 24f5b64f2616daa3a3dabe34deb7fd6ebfc43429 Author: Maxim Muzafarov Date: Wed Mar 29 06:24:34 2023 -0500 Notify the corresponding JIRA issue as soon as the PR is raised Patch by Maxim Muzafarov; reviewed by brandonwilliams and smiklosovic for CASSANDRA-18372 commit 5f61645778ed8d0eb8c275ee6447f6169b2bfb31 Merge: 5b877b9d2b c4535a2e2a Author: Brandon Williams Date: Wed Mar 29 06:32:13 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit c4535a2e2ac80f36ce56af72f5e93c0dad792441 Merge: 11b84d2ef2 2f8d11335e Author: Brandon Williams Date: Wed Mar 29 06:32:09 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 2f8d11335ee2e997c803554e3cd58274ccd462c7 Author: Brandon Williams Date: Wed Mar 29 06:32:05 2023 -0500 Remove asf.yaml from non-trunk branches commit 5b877b9d2b973444b63553f924d3e864c07a97b9 Merge: 2974810607 11b84d2ef2 Author: Brandon Williams Date: Wed Mar 29 06:26:15 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 11b84d2ef23177b46c969b158056a184bcb6f44f Merge: 955cf156c8 f352f2b637 Author: Brandon Williams Date: Wed Mar 29 06:25:59 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit f352f2b6372cc879c63ff535c684cbaea9e0f152 Author: Maxim Muzafarov Date: Wed Mar 29 06:24:34 2023 -0500 Notify the corresponding JIRA issue as soon as the PR is raised Patch by Maxim Muzafarov; reviewed by brandonwilliams and smiklosovic for CASSANDRA-18372 commit 2974810607b3612fdd50a30d02221c927be9beec Merge: 5104008db9 955cf156c8 Author: Jacek Lewandowski Date: Tue Mar 21 10:09:31 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Fix RepairJob unnecessarily reporting cancellation error commit 955cf156c898cfadcc4225b34052f5f35ff8463a Merge: 58d06d34ab f6a04ca81f Author: Jacek Lewandowski Date: Tue Mar 21 10:06:31 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Fix RepairJob unnecessarily reporting cancellation error commit f6a04ca81f2d0d8973fb7a99569c118122c500a2 Author: Jacek Lewandowski Date: Sat Mar 18 08:28:40 2023 +0100 Fix RepairJob unnecessarily reporting cancellation error patch by Jacek Lewandowski; reviewed by Brandon Williams for CASSANDRA-17701 commit 5104008db947f7ad6e6b65b474281939dba92829 Merge: 3cfb222122 58d06d34ab Author: Jacek Lewandowski Date: Tue Mar 14 15:15:15 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Fix the ordering of sstables when running sstableupgrade tool commit 58d06d34ab949fadf1db2309b5be8e35352d9766 Merge: 4b3cb0a772 8df69c8016 Author: Jacek Lewandowski Date: Tue Mar 14 15:13:19 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Fix the ordering of sstables when running sstableupgrade tool commit 8df69c8016fb84e59e558e74508a0767ad3a7cc0 Author: Claude Warren Date: Tue Mar 14 09:24:25 2023 +0100 Fix the ordering of sstables when running sstableupgrade tool patch by Claude Warren; reviewed by Jacek Lewandowski and Stefan Miklosovic for CASSANDRA-18143 commit 3cfb2221223a7631f116108aec0a45646e775705 Author: Jaroslaw Grabowski Date: Thu Mar 9 12:35:48 2023 +0100 Fix BufferPool.memoryInUse counter The counter was incorrectly decremented by the size of the unused portion of the provided buffer. It is now decremented by the number of bytes actually returned to the pool (that may be different than "size"). The number is calculated as a difference between original and resulting buffer capacity. patch by Jaroslaw Grabowski , reviewed by Jacek Lewandowski for CASSANDRA-18311 commit 2aedb58123e562ef6bdb7314033512d0128329ea Merge: a874e02eec 4b3cb0a772 Author: Andrés de la Peña Date: Fri Mar 10 11:51:25 2023 +0000 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 4b3cb0a772b7ab0de9dec19c148388d1aea729db Merge: 326b5aceb1 02048cffea Author: Andrés de la Peña Date: Fri Mar 10 11:50:04 2023 +0000 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 02048cffeae4267054541f96741b16b1d0d89a20 Author: Andrés de la Peña Date: Tue Feb 28 11:44:31 2023 +0000 Remove -l / -m / -h designation and have two options: free or paid tier circle config patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova and Josh McKenzie for CASSANDRA-18012 commit a874e02eecb282287929db6dfe76f1694c87a217 Merge: 40f9ca60f1 326b5aceb1 Author: Brandon Williams Date: Thu Mar 9 10:02:33 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 326b5aceb188912f377aac1e36485e0054605851 Merge: 4b84a8d7ac be574fc34b Author: Brandon Williams Date: Thu Mar 9 10:01:06 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit be574fc34ba9834929f1618ab63dd74446cd2683 Author: Brandon Williams Date: Thu Mar 9 09:53:58 2023 -0600 Fix default file system error handler for disk_failure_policy die Patch by Runtian Liu; reviewed by brandonwilliams and smiklosovic for CASSANDRA-18294 commit 40f9ca60f103783aa481bc9a91b92fd55b4ea625 Author: Benedict Elliott Smith Date: Wed Mar 1 19:08:20 2023 -0700 Improve memtable allocator accounting when updating AtomicBTreePartition patch by Benedict Elliott Smith; reviewed by Benjamin Lerer, Jon Meredith for CASSANDRA-18125 commit ccede05bf5fbf3d8e409960d6c93845f47ee6339 Merge: 1d532bce7b 4b84a8d7ac Author: Brandon Williams Date: Fri Feb 24 12:32:18 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 4b84a8d7ac5875bed35c81e1f0107c495452ba0b Merge: a5b05cea5e 5374f56a0a Author: Brandon Williams Date: Fri Feb 24 12:29:37 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 5374f56a0a8d95f30a1398368b86192bad935446 Author: Brandon Williams Date: Fri Feb 24 12:28:59 2023 -0600 Fix typo in suppressions commit 1d532bce7bb4cee8073a15a5b7572fc4d9ad8bcd Merge: 3a4fff38b5 a5b05cea5e Author: Brandon Williams Date: Wed Feb 22 10:25:08 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit a5b05cea5e2843d7ada11c03de823b8c07fb5455 Merge: 788b4eca2d eb36a86a72 Author: Brandon Williams Date: Wed Feb 22 10:24:22 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit eb36a86a72c0692327545c19f8eed23a6d165246 Author: Brandon Williams Date: Fri Feb 17 07:28:01 2023 -0600 Removing trailing year from copyright. Patch by brandonwilliams; reviewed by mck for CASSANDRA-18261 commit 3a4fff38b5d89650d6ac96df22d850a0935d7730 Author: Jon Meredith Date: Thu Feb 16 15:05:37 2023 -0700 Ninja fix CASSANDRA-18242 - was not generating IDEA workspace for J11 correctly commit fcc16f5f2bf80cdbb299cd81e12ea0bbb35cc0af Author: Stefan Miklosovic Date: Wed Feb 15 00:01:10 2023 +0100 Update zstd-jni to version 1.5.4-1 patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18259 commit 7b079503eba72aeb0025f0830c868efaa6c4f84c Author: Stefan Miklosovic Date: Wed Feb 15 10:12:15 2023 +0100 Increment version to 4.0.9 commit 4e0aad3319f90eaf7a5e5aed1fd3c5bb7a949fbd Author: Jon Meredith Date: Fri Feb 10 10:05:42 2023 -0700 Split and order IDEA workspace template VM_PARAMETERS patch by Jon Meredith; reviewed by David Capwell for CASSANDRA-18242 commit cb4f90542d005c0cacc7cbc464e5edb2fd748b60 Merge: 32c56df067 788b4eca2d Author: Ekaterina Dimitrova Date: Sat Feb 11 14:34:36 2023 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 788b4eca2d0b5f3d508daf9d1b08426f075b8fca Merge: 74b0b2086f 476d31bce2 Author: Ekaterina Dimitrova Date: Sat Feb 11 14:32:37 2023 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 476d31bce2597cf71cdb7dd446d5cac28ee69a82 Author: Ekaterina Dimitrova Date: Mon Oct 31 18:10:05 2022 -0400 Add Dtest-large and cqlshlib tests to CircleCI; Fix num tokens in CircleCI tests to match the default value; Fix CircleCI Python tests split bug; Fix CircleCI long unit tests patch by Ekaterina Dimitrova; reviewed by Andres de la Pena for CASSANDRA-18001 commit 32c56df067b72da8593c1ddaaf143fe8668459dd Author: Stefan Miklosovic Date: Tue Feb 7 22:35:13 2023 +0100 Prepare debian changelog for 4.0.8 commit b8494a82751ae2d31ab0f36412f7eacdcc1c3774 Author: Stefan Miklosovic Date: Tue Feb 7 10:21:16 2023 +0100 Log warning message on aggregation queries without key or on multiple keys patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18219 commit b74c86404a0d131677286206562bb4bfa3e8e1a9 Author: nvharikrishna Date: Sat Feb 4 00:02:08 2023 +0530 Fix the output of FQL dump tool to properly separate entries patch by N V Harikrishna; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-18215 commit 0c58fbb8dd25beab4b4a81650be1ed0ec888ff66 Author: yongj Date: Wed Feb 1 07:52:35 2023 +0000 Add cache type information for maximum memory usage warning message Patch by Yong Jiang; reviewed by brandonwilliams and smiklosovic for CASSANDRA-18184 commit ba434b8231360347984c9a9906e5469767bfbcec Author: nvharikrishna Date: Wed Dec 14 00:25:54 2022 +0530 Fix NPE in fqltool dump on null value If fullquerylog is enabled and a prepared query executed with null value binded, then fqltool dump is running into NPE. Checking value for null before wrapping it. patch by N V Harikrishna; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-18113 commit 9bf128aaa33c21427c826dab82414e3772d2ba24 Author: Jacek Lewandowski Date: Wed Jan 25 09:01:16 2023 +0100 Improve unit tests performance - Don't flush schema on every schema update in unit tests - Use unix command to delete test data - Shorten teardown - Stable processor count presented by JMX on Jenkins, CircleCI and local Patch by , reviewed by and for CASSANDRA-17427 commit ef2066fe6084e424ae7dd57f669d06b86a3924a1 Merge: f8ad33ed4e 74b0b2086f Author: Brandon Williams Date: Mon Jan 30 15:49:20 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 74b0b2086f3a7e838acde21c8020f10f0a47955a Author: Brandon Williams Date: Mon Jan 30 07:46:54 2023 -0600 remove walrus operator in gen-nodetool-docs.py Patch by brandonwilliams; reviewed by smiklosovic for CASSANDRA-18207 commit f8ad33ed4e363418a768551fb34464b111820453 Merge: 0d0c5695df 48d4214f6c Author: Brandon Williams Date: Mon Jan 30 07:30:32 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 48d4214f6cac5eaf9f9d922c63ddcd0b57082edf Merge: 89266aadde fa18ea1852 Author: Brandon Williams Date: Mon Jan 30 07:29:48 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 89266aadde0eccacc5551ae2dbd32d5720235d4c Author: Brandon Williams Date: Mon Jan 23 07:07:14 2023 -0600 Suppress CVE-2022-41915 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18147 commit fa18ea185233d97b9f2771b1a3a88f46dc1610d4 Author: Brandon Williams Date: Mon Jan 23 07:07:14 2023 -0600 Suppress CVE-2022-41915 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18147 commit 0d0c5695df574888a02bb457160aac9adc47c814 Merge: 0b7e3a8ee7 5ac75323cd Author: Brandon Williams Date: Mon Jan 30 07:26:16 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 5ac75323cd6cfef3e39a2fc476b174e9a70eff56 Merge: 8076f37c06 7ad693581b Author: Brandon Williams Date: Mon Jan 30 07:24:41 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 7ad693581bb87e75330c38daf266255dd78cbf6d Author: Brandon Williams Date: Mon Jan 23 07:07:14 2023 -0600 Suppress CVE-2022-41915 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18147 commit 0b7e3a8ee7f9359eaf63208e12f32b19e6874e74 Author: Maciej Sokol Date: Fri Jan 27 10:04:43 2023 -0600 Connect to listen address when own broadcast address is requested Patch by Maciej Sokol; reviewed by brandonwilliams and smiklosovic for CASSANDRA-18200 commit 2a8e1972fd5444336ea4285fa38fbcabe873dc60 Merge: c2e9e0d841 8076f37c06 Author: Stefan Miklosovic Date: Sat Jan 28 16:40:41 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 8076f37c06c3f11dfb571b9c9169090c6fa5ebf5 Merge: 03c86cfcb0 4c8e9097bd Author: Stefan Miklosovic Date: Sat Jan 28 16:21:39 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 4c8e9097bd616a1ae4c611a542cbdea20ba871dc Author: Stefan Miklosovic Date: Sat Jan 28 16:18:51 2023 +0100 Introduce check for names of test classes patch by Stefan Miklosovic; reviewed by Berenguer Blasi and David Capwell for CASSANDRA-17943 commit c2e9e0d8416866d6600444773f11f592711a5fe9 Merge: 299ead7534 03c86cfcb0 Author: Brandon Williams Date: Wed Jan 25 15:16:32 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 03c86cfcb0dd31a9c904e2561e3f8a1f26357a44 Author: Brandon Williams Date: Wed Jan 25 08:55:13 2023 -0600 Add concurrency to adoc generation in gen-nodetool-docs.py Patch by brandonwiliams; reviewed by mck for CASSANDRA-18197 commit 299ead7534a97af44035ab90002565f88f439144 Merge: 5f54d64c78 e9aa5ec514 Author: Brandon Williams Date: Tue Jan 24 12:26:12 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit e9aa5ec514c436a8fb7dede10fa97b8d8cca200c Merge: 92245b0f07 9a64be3234 Author: Brandon Williams Date: Tue Jan 24 11:59:53 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 9a64be32340829038e4895a667eda025d5219f1b Author: Brandon Williams Date: Thu Jan 12 09:55:33 2023 -0600 Suppress CVE-2021-1471, CVE-2021-3064, CVE-2021-4235 commit 5f54d64c78818125f47ecc509a757f4b55fcc893 Merge: 8bb9c72f58 92245b0f07 Author: Brandon Williams Date: Tue Jan 24 06:31:50 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 92245b0f07cda20d08a29bf5912018867ed16779 Merge: f52b316016 e7f55ab8c3 Author: Brandon Williams Date: Tue Jan 24 06:29:21 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit e7f55ab8c3bd6bac4c87354afec231d7237c35b8 Author: Brandon Williams Date: Thu Jan 12 06:50:40 2023 -0600 switch to SafeConstructor Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18150 commit 8bb9c72f582de6bcc39522ba9ade91fd5bc22f67 Author: lzurovchak1 Date: Wed Dec 21 13:34:18 2022 -0500 Add safeguard so cleanup fails when node has pending ranges Patch by Lindsey Zurovchak; Reviewed by Paulo Motta, Stefan Miklosovic for CASSANDRA-16418 Closes #2061 commit 669731b03379ab1d2c2b6b2acc7b705f6095c579 Merge: 7377e16121 f52b316016 Author: Stefan Miklosovic Date: Mon Jan 23 15:05:22 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit f52b3160168f1e53e2e99bd387be2c3fcd864c3f Merge: 6f719f811a 357c0d5a89 Author: Stefan Miklosovic Date: Mon Jan 23 14:58:54 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 357c0d5a896ca79065b0b3e18b4e6f33a87bd021 Author: Stefan Miklosovic Date: Fri Jan 20 13:25:26 2023 +0100 Expand build.dir property in rat targets patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18183 commit 7377e16121057cc736dfee5449b5ad2fef9756f3 Merge: 9a0af4112e 6f719f811a Author: Brandon Williams Date: Mon Jan 23 06:56:00 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 6f719f811acec652178ec7c660640e4fc3bd798b Merge: fc3ad6525f fe71dfd7a1 Author: Brandon Williams Date: Mon Jan 23 06:53:53 2023 -0600 Merge branch 'cassandra-3.0' into 'cassandra-3.11' commit fe71dfd7a15ded62b70eee4a2981d54baf35be8d Author: Brandon Williams Date: Fri Jan 20 10:40:52 2023 -0600 Suppress CVE-2022-41881 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18148 commit 9a0af4112e87f5b97056aa39e63c5ab461b60237 Author: Andrés de la Peña Date: Mon Jan 9 13:50:47 2023 +0000 Fix legacy clustering serialization for paging with compact storage patch by Andrés de la Peña; reviewed by Berenguer Blasi and Piotr Kołaczkowski for CASSANDRA-17507 commit b43293e200dd95c98bfb3ee436fe05bb18da114f Author: Brandon Williams Date: Tue Jan 17 14:18:56 2023 -0600 Upgrade cython Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18121 commit f6933a041901861e526b44af6fc974b29b33982e Author: Brandon Williams Date: Fri Dec 16 12:29:57 2022 -0600 convert cqlshlib from nose to pytest Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18088 commit 563a26c6b1493f46ebb597085ae589f1835acd50 Author: Brandon Williams Date: Fri Dec 16 10:09:39 2022 -0600 get newest pip in cassandra-cqlsh-tests.sh Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18121 commit d38427a67b1b9684ca85d27a19f321325032dafe Author: Brandon Williams Date: Thu Dec 8 10:35:54 2022 -0600 add py311 tests to circle Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18094 commit 0b5248d8c3ef743a8dda42f7be5dadb70e373b51 Author: Brandon Williams Date: Sun Dec 4 13:05:19 2022 -0600 Accommodate python 3.11 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18088 commit 815f7de34657865228e35e5f593fbe9d82941d0a Merge: 175272fff4 fc3ad6525f Author: Stefan Miklosovic Date: Wed Jan 18 15:00:43 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit fc3ad6525f93234132e888d5d8d454e7eedcadb3 Merge: d32c52aba7 8ecd7616fe Author: Stefan Miklosovic Date: Wed Jan 18 14:57:15 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 8ecd7616fe5d3ce0cfe8f4621eda1905a9110db1 Author: German Eichberger Date: Wed Jan 18 14:55:19 2023 +0100 Default role is created with zero timestamp patch by German Eichberger; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-12525 Co-authored-by: Stefan Miklosovic commit 175272fff46f8441890e481c031ad521df9c9983 Author: Maciej Sokol Date: Wed Jan 11 09:56:29 2023 +0100 Fix formatting of duration in cqlsh Patch by Maciej Sokol; reviewed by brandonwilliams and edimitrova for CASSANDRA-18141 commit ac8ddfb321551b8ffa75f75e5578713f9b083030 Merge: 6e6846892a d32c52aba7 Author: Brandon Williams Date: Tue Jan 17 09:57:51 2023 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit d32c52aba72c182fab0c30cf3d32ce723dde582a Merge: de8daabab6 e0e63f4415 Author: Brandon Williams Date: Tue Jan 17 09:56:50 2023 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit e0e63f44155a00a540407b402a6cae8480aa2c32 Author: Brandon Williams Date: Thu Jan 12 12:39:09 2023 -0600 Suppress CVE-2021-37533 Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18146 commit 6e6846892a5d7ec2feaf2c35d380ed5975d09517 Author: Stefan Miklosovic Date: Thu Nov 5 11:17:02 2020 +0100 Fix sstable loading of keyspaces named snapshots or backups Patch by Stefan Miklosovic; Reviewed by Paulo Motta for CASSANDRA-14013 Co-authored-by: Paulo Motta commit 2c03055715c874846a2970ec6ec13c2dcf8cdf8c Merge: a83de9bcd2 de8daabab6 Author: Bereng Date: Mon Jan 16 07:07:05 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit de8daabab662c23ee186461601226c236b8df5e0 Merge: b8a87abba4 f384e69482 Author: Bereng Date: Mon Jan 16 07:05:25 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit f384e694825a2cd0a95debf30accb153f00e80b0 Author: Bereng Date: Fri Jan 13 06:56:37 2023 +0100 When generate.sh fails its rc=0 - take 2 patch by Andres de la Peña and Berenguer Blasi; reviewed by Andres de la Peña for CASSANDRA-18032 Co-authored-by: Berenguer Blasi Co-authored-by: Andres de la Peña commit a83de9bcd21a032c1dbc6dc323c7e53f828b2cb6 Merge: 6c65c9e49e b8a87abba4 Author: Bereng Date: Thu Jan 12 08:17:48 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit b8a87abba47441d97b3c85a0473d02919e1c071b Author: Bereng Date: Fri Dec 9 11:13:52 2022 +0100 Do not leak 2015 synthetic memtable Epoch patch by Berenguer Blasi; reviewed by Caleb Rackliffe for CASSANDRA-18118 commit 6c65c9e49e656fc1fc4ae23adda391d0e3d4f2b4 Merge: f17e431f24 7fe71274d0 Author: Bereng Date: Thu Jan 12 07:07:30 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 7fe71274d009b18c16f71c7e223eb8f225ed5a8e Merge: 156cc6fedc 69dfed3671 Author: Bereng Date: Thu Jan 12 07:06:04 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 69dfed3671144c019fb69793e3152b5a71d76421 Author: Bereng Date: Fri Nov 11 09:56:49 2022 +0100 When generate.sh fails its rc=0 patch by Berenguer Blasi; reviewed by Andres de la Peña for CASSANDRA-18032 commit f17e431f24f831321092b651a9d3adf7b30b8fca Merge: 7877d59d19 156cc6fedc Author: Stefan Miklosovic Date: Tue Jan 10 09:03:51 2023 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 156cc6fedc2e1cbfd7ea6ca77224f7bc5c725cd2 Merge: fff313e447 877d6e6e95 Author: Stefan Miklosovic Date: Tue Jan 10 09:01:13 2023 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 877d6e6e956ffa387a7d4ec1c7f0ca8ccb23f33e Author: Maxim Muzafarov Date: Tue Dec 20 00:28:25 2022 +0100 Add to the IntelliJ Git Window issue navigation links to Cassandra's Jira patch by Maxim Muzafarov; reviewed by David Capwell, Aleksey Yeschenko, Stefan Miklosovic for CASSANDRA-18126 commit 7877d59d19244b7aa701ebe7b2192d6e191ee3fb Merge: ee266d283e fff313e447 Author: Ekaterina Dimitrova Date: Fri Dec 23 15:05:40 2022 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit fff313e4470f1f64fd90da2a6aff84f3981a0189 Merge: 0767c83416 df0b139701 Author: Ekaterina Dimitrova Date: Fri Dec 23 15:01:11 2022 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit df0b13970198fcf7079a1044c5911d8ba0b6dac3 Author: Ekaterina Dimitrova Date: Thu Dec 22 17:21:48 2022 -0500 Use CircleCI Large executor instead of XLarge executor in MIDRES config for Python upgrade tests patch by Ekaterina Dimitrova; reviewed by Michael Semb Wever for CASSANDRA-18127 commit ee266d283e740785817d7cb6749e6c135f4aacbb Author: Marcus Eriksson Date: Thu Oct 20 08:18:06 2022 +0200 Avoid ConcurrentModificationException in STCS/DTCS/TWCS.getSSTables Patch by marcuse; reviewed by Aleksey Yeschenko for CASSANDRA-17977 commit 95e26ad1ab29b52e720edf9792eba100817cce9d Merge: ace3920239 0767c83416 Author: Marcus Eriksson Date: Fri Dec 23 14:43:01 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 0767c834162d74bdc857f47615b9e8a5c7e76d5b Merge: baa9d0327f 730b898b74 Author: Marcus Eriksson Date: Fri Dec 23 14:41:50 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 730b898b7415e568138c6fc1e961f69b41ba1fa0 Author: Marcus Eriksson Date: Tue Oct 18 12:57:07 2022 +0200 Don't group TWCS sstables for anticompaction Patch by marcuse; reviewed by Aleksey Yeschenko for CASSANDRA-17970 commit ace3920239f01ffbabbc424299900aa6d847946d Merge: f01d2b4a3c baa9d0327f Author: Stefan Miklosovic Date: Tue Dec 20 10:07:00 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit baa9d0327fcb656524df604ad242004ef2887a3c Merge: eb91e2c354 f55b2fb1b3 Author: Stefan Miklosovic Date: Tue Dec 20 10:02:32 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit f55b2fb1b3a8af03a3424abfe7afa2ac60ac9dff Author: Stefan Miklosovic Date: Tue Dec 6 15:49:09 2022 +0100 Do not spam the logs with MigrationCoordinator not being able to pull schemas patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18096 commit f01d2b4a3ca114ac58d95befcab3dbbad9d960aa Author: Mick Semb Wever Date: Sat Nov 12 12:02:35 2022 +0100 Restore custom param types over messaging system patch by Mick Semb Wever; reviewed by Aleksey Yeschenko for CASSANDRA-17981 commit f22263cd8afe353e51ff6135e9bc6f2eeaafa2b2 Merge: c2bbee2020 eb91e2c354 Author: Yifan Cai Date: Mon Dec 5 18:50:39 2022 -0800 Merge branch 'cassandra-3.11' into cassandra-4.0 commit eb91e2c354d4436913066bf8c15594a64c564424 Merge: b7762e2aa2 473656c1d5 Author: Yifan Cai Date: Mon Dec 5 18:48:39 2022 -0800 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 473656c1d53edb998aa60d414221e397797de52b Author: Yifan Cai Date: Tue Nov 29 15:19:06 2022 -0800 Fix incorrect resource name in LIST PERMISSION output patch by Yifan Cai; reviewed by Berenguer Blasi, Sam Tunnicliffe for CASSANDRA-17848 commit c2bbee2020af7b07eb478c10df21a8d081ec6a7e Merge: bba7ab3eca b7762e2aa2 Author: Brandon Williams Date: Mon Dec 5 10:06:17 2022 -0600 Merge branch 'cassandra-3.11' into cassandra-4.0 commit b7762e2aa276ecf9cad2dd26ee52fe2463ae52db Merge: 5a53c36515 92019df4d8 Author: Brandon Williams Date: Mon Dec 5 10:02:35 2022 -0600 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 92019df4d8540b384d7fb8655f7c02293f7f7ec1 Author: Brandon Williams Date: Wed Nov 30 09:44:25 2022 -0600 Suppress CVE-2022-41854 and similar Patch by brandonwilliams; reviewed by bereng for CASSANDRA-18083 commit bba7ab3eca536744cb04cf92d19a9291eb1012f2 Merge: 1df8174d67 5a53c36515 Author: Jordan West Date: Tue Nov 29 18:27:45 2022 -0800 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 5a53c36515e5f29982458c9a09f3f900d4ac20a4 Merge: 7572725def 13d495aa7d Author: Jordan West Date: Tue Nov 29 18:26:40 2022 -0800 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 13d495aa7d5b7a7c121fcc9e105f79107c5c2a1c Author: Jordan West Date: Mon Jan 10 16:42:47 2022 -0800 fix intermittent failures in nodetool toppartitions caused by failure to validate invalid bytes Patch by Jordan West; Reviewed by Brandon Williams and Cheng Wang for CASSANDRA-17254 commit 1df8174d67f67a75b5b17e2dff7f9838dc3e3ff6 Author: Bereng Date: Mon Nov 14 13:41:26 2022 +0100 Cqlshlib tests are j11 greedy Patch by Berenguer Blasi; reviewed by Ekaterina Dimitrova and Michael Semb Wever for CASSANDRA-18039 commit 2638051af186eb7cee110c103ae011f853ae8bb3 Author: Stefan Miklosovic Date: Thu Nov 10 18:53:40 2022 +0100 Harden parsing of boolean values in CQL in PropertyDefinitions patch by Stefan Miklosovic; reviewed by Aleksey Yeschenko for CASSANDRA-17878 commit 0f2fb1eeb3e1616b80bae9bae32b824c17de6ae5 Author: Bereng Date: Fri Nov 11 07:13:58 2022 +0100 Cqlsh Cython failure on 4.0 Patch by Berenguer Blasi; reviewed by Michael Semb Wever for CASSANDRA-18036 commit 35826b7447b0c3233ca3dacad2e44de082e76896 Author: Natnael Adere Date: Wed Oct 19 14:34:05 2022 -0400 Fix error message about type hints (CASSANDRA-17915) Patch by Natnael Adere; reviewed by blerer and brandonwilliams for CASSANDRA-17915 commit 2b3d0d32a7da99949c0fa7f91af89c12deb20e3f Merge: 38806e795b 7572725def Author: Ekaterina Dimitrova Date: Thu Nov 10 12:31:07 2022 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 7572725defd4d5bba12097046b8338a955d8d032 Author: Derek Chen-Becker Date: Wed Oct 26 14:54:05 2022 -0600 [CASSANDRA-17950] Enable dtest-offheap in CircleCI The dtest-offheap test was only added to the Jenkins configuration, so this commit adds it to the CircleCI build for parity. Patch by Derek Chen-Becker; reviewed by Ekaterina Dimitrova, Derek Chen-Becker, Michael Semb Wever, and Andres de la Pena for CASSANDRA-17950 Co-authored-by: Ekaterina Dimitrova Co-authored-by: Derek Chen-Becker commit 38806e795b4f8745616f7c48c9bdae0760dcf7b5 Merge: 70478a1578 a44209cf43 Author: Mick Semb Wever Date: Thu Nov 10 14:41:04 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Upgrade Cython, minimum required now is 0.27.2 commit a44209cf4314a59f49884f5e35b201af5a200d37 Merge: 9213335f59 d9ef794ef6 Author: Mick Semb Wever Date: Thu Nov 10 14:40:10 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Upgrade Cython, minimum required now is 0.27.2 commit d9ef794ef6a03c2d7e0b1476cc0d29a008040095 Author: Mick Semb Wever Date: Thu Nov 10 12:08:04 2022 +0100 Upgrade Cython, minimum required now is 0.27.2 patch by Mick Semb Wever; reviewed by Berenguer Blasi for CASSANDRA-18028 commit 70478a1578fe4facb0f59fcb40529e0a4040ac81 Author: Brandon Williams Date: Mon Oct 31 11:51:41 2022 -0500 Fix testCommitLogInitWithException: signal condition instead of killing thread Patch by brandonwilliams; reviewed by adelapena and bereng for CASSANDRa-17928 commit 67c530171c7db1196c83783b346cc368494c6aa3 Merge: cea850d67d 9213335f59 Author: Jacek Lewandowski Date: Wed Nov 9 17:17:54 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 9213335f59293926b2d643fa8a156a882495dd42 Author: Jacek Lewandowski Date: Thu Nov 3 17:29:18 2022 +0100 Fix Splitter sometimes creating more splits than requested Spliter.splitOwnedRanges for some inputs creates an extra split. For example, when we request 7 ranges from 0..31 range, it will return 8 ranges. There is an assertion in that method which verifies whether it returns the requested number of splits. Since those numbers differs, when Cassandra is be started with assertions enabled, it would fail. patch by Jacek Lewandowski; reviewed by Marcus Eriksson for CASSANDRA-18013 commit cea850d67d5b726fe7ca9fef39c8578b46210155 Merge: 7dbbe6907b 7b7762826e Author: Andrés de la Peña Date: Tue Nov 8 17:49:06 2022 +0000 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 7b7762826e367b2a82ba1e99b12f8f19c583b920 Merge: f9b2cd6f3f 955231cacf Author: Andrés de la Peña Date: Tue Nov 8 17:48:29 2022 +0000 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 955231cacfc2732dd1fd4275049e224ab220d107 Author: Andrés de la Peña Date: Tue Nov 8 13:27:49 2022 +0000 CircleCI: Remove repeated jobs from default LOWRES, MIDRES and HIGHRES files patch by Andrés de la Peña; reviewed by Berenguer Blasi and Ekaterina Dimitrova for CASSANDRA-18024 commit 7dbbe6907be58138f2c88d0adaab5c99a038e2b1 Author: Andrés de la Peña Date: Mon Nov 7 15:41:47 2022 +0000 CircleCI: Fix j11_utests_fqltool executor patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova for CASSANDRA-18020 commit 17e694277597404aadfec664dfe4c11da7ed24a4 Merge: 5e705a8492 f9b2cd6f3f Author: Mick Semb Wever Date: Sat Nov 5 15:51:47 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: ninja-fix antcall maybe-build-test needs inheritRefs (CASSANDRA-18000) commit f9b2cd6f3f1bdfa775b6e8bde7e5b22dc6e60d0b Merge: bdd6606486 30a3266780 Author: Mick Semb Wever Date: Sat Nov 5 15:50:56 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: ninja-fix antcall maybe-build-test needs inheritRefs (CASSANDRA-18000) commit 30a3266780c841f52a6accfece48d138d839e32d Author: Mick Semb Wever Date: Sat Nov 5 15:50:28 2022 +0100 ninja-fix antcall maybe-build-test needs inheritRefs (CASSANDRA-18000) commit 5e705a84927bd4de1020adec94b4dcc86bd23354 Merge: 6885bf9724 bdd6606486 Author: Andrés de la Peña Date: Fri Nov 4 16:26:08 2022 +0000 Merge branch 'cassandra-3.11' into cassandra-4.0 commit bdd6606486e91024e123afc4e56465a95b102d1a Merge: 43dcb5cd14 95400649d8 Author: Andrés de la Peña Date: Fri Nov 4 16:25:25 2022 +0000 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 95400649d83f5759a452a381eb6034f4c62bda94 Author: Andrés de la Peña Date: Thu Nov 3 18:57:34 2022 +0000 Improve behaviour of the Ant flag to skip test building, no-build-test patch by Andrés de la Peña; reviewed by Berenguer Blasi for CASSANDRA-18000 commit 6885bf9724167416a89ea5ba4e5cb4a495ebecd2 Merge: 8f06ec372e 43dcb5cd14 Author: Ekaterina Dimitrova Date: Fri Nov 4 10:41:47 2022 -0400 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 43dcb5cd14d06ad1708d884280b80f00345ec599 Merge: b82dca6f81 90fe8608a4 Author: Ekaterina Dimitrova Date: Fri Nov 4 10:38:12 2022 -0400 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 90fe8608a401dd086c8918a045d01ff288849f49 Author: Ekaterina Dimitrova Date: Wed Sep 21 12:17:33 2022 -0400 CirleCI: fix Python upgrade DTests config patch by Ekaterina Dimitrova; reviewed by Michael Semb Wever for CASSANDRA-17912 commit 8f06ec372e20c55b85a083e43aa3b8e0993b648d Merge: 6152cb08db b82dca6f81 Author: Brandon Williams Date: Thu Nov 3 08:08:27 2022 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit b82dca6f81f67994af2e62f4cfd74a74ddbb45ca Merge: 7df905a429 e0331f631e Author: Brandon Williams Date: Thu Nov 3 08:07:57 2022 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit e0331f631e419d82977d94b561cac2411860f279 Merge: 116ce3bc05 bba519501c Author: Brandon Williams Date: Thu Nov 3 08:07:29 2022 -0500 Merge branch 'cassandra-2.2' into cassandra-3.0 commit bba519501cbc7f47577fd7ba4d08969ff8e1d90c Author: Brandon Williams Date: Wed Aug 31 13:30:31 2022 -0500 add python2-dev to build-deps Patch by brandonwilliams; reviewed by edimitrova for CASSANDRA-17871 commit 6152cb08dba770f46496d84cde067c6776c66fa0 Merge: 0db0b8f781 7df905a429 Author: Andrés de la Peña Date: Thu Nov 3 11:41:39 2022 +0000 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 7df905a4293af692b793b9364f2011f1c0a3a0d9 Merge: deede44af5 116ce3bc05 Author: Andrés de la Peña Date: Thu Nov 3 11:36:22 2022 +0000 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 116ce3bc0562f69e698f569db0904debd9f87cd0 Author: Andrés de la Peña Date: Thu Nov 3 11:30:38 2022 +0000 CircleCI: Add jobs for missing specialized unit tests Fix run_repeated_utests for non-testsome targets patch by Andrés de la Peña; reviewed by Ekaterina Dimitrova and Berenguer Blasi for CASSANDRA-17987 commit 0db0b8f781ee90921b0d60d2fc41a289e6d4d389 Author: Mick Semb Wever Date: Tue Nov 1 20:25:01 2022 +0100 Update NetBeans project file for dependency changes since 7th July 2021 patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova for CASSANDRA-18002 commit 842cbe28a6d1cee00fefcf73fcdd7330f07e78ea Merge: 35ef5b9957 deede44af5 Author: Andrés de la Peña Date: Fri Oct 28 13:54:40 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit deede44af5b0d8d6ec19ecf1e613544262d85144 Merge: ed07f7d6bc ccc0b14d56 Author: Andrés de la Peña Date: Fri Oct 28 13:54:23 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit ccc0b14d56d52221c64de8d029950fa265791eb3 Author: Andrés de la Peña Date: Fri Oct 28 13:54:01 2022 +0100 CircleCI: Fix generate.sh without options modifying config patch by Andrés de la Peña; reviewed by Brandon Williams and Berenguer Blasi for CASSANDRA-17995 commit 35ef5b99577ef8b04b8d4b326154775f510ade42 Author: Stefan Miklosovic Date: Mon Oct 24 22:11:37 2022 +0200 Fix possible race condition on repair snapshots patch by Stefan Miklosovic; reviewed by David Capwell for CASSANDRA-17955 commit ac165325bc281364833013dd3bf5e1916fe2ac62 Merge: 8429229b7d ed07f7d6bc Author: Mick Semb Wever Date: Sun Oct 23 23:25:42 2022 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Increment versions to 3.0.29, 3.11.15, 4.0.8 commit ed07f7d6bc3e2185997f8e0180b9643bcbf57543 Merge: 761d9f9516 a3b4119b5a Author: Mick Semb Wever Date: Sun Oct 23 23:19:59 2022 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Increment versions to 3.0.29, 3.11.15, 4.0.8 commit a3b4119b5a8a90e53d4075371e98671e69783495 Author: Mick Semb Wever Date: Sun Oct 23 23:18:45 2022 +0200 Increment versions to 3.0.29, 3.11.15, 4.0.8 commit 8429229b7d55182f2a6450dad2a48b3bfadd74f8 Merge: ce1584ccc8 761d9f9516 Author: Stefan Miklosovic Date: Fri Oct 21 10:05:47 2022 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 761d9f9516799a3aba1a42da7f14ab98dc7db66f Merge: c7d665dcc7 2b31675fbd Author: Stefan Miklosovic Date: Fri Oct 21 10:04:28 2022 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 2b31675fbd2fa217d864f314536dc15eea082638 Author: Stefan Miklosovic Date: Wed Oct 19 13:53:36 2022 +0200 Fix running Ant rat targets without git patch by Stefan Miklosovic; reviewed by Michael Semb Wever for CASSANDRA-17974 commit ce1584ccc8ff65dc7e600f13f31b553732a1f12e Author: Ekaterina Dimitrova Date: Thu Sep 1 17:11:40 2022 -0400 Fix ASM bytecode version inconsistency patch by Ekaterina Dimitrova; reviewed by Berenguer Blasi and Benedict Elliot Smith for CASSANDRA-17873 commit a9df34c62a3596a9fc40de200adffe10b960146a Merge: 3251b85def c7d665dcc7 Author: Andrés de la Peña Date: Wed Oct 19 12:49:45 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit c7d665dcc7c06120c6c149a7b9fda27b8ea13b67 Merge: d465ac2dc9 2b511c5714 Author: Andrés de la Peña Date: Wed Oct 19 12:48:49 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 2b511c57143063553fe1256679d5ce8aefc50129 Author: Andrés de la Peña Date: Wed Oct 19 12:47:59 2022 +0100 ninja-fix: Fix bashisms on .circleci/generate.sh patch by Andrés de la Peña; reviewed by Brandon Williams for CASSANDRA-17939 commit 3251b85def05d963342e9799e046dd4a833ba37c Merge: 277fa4fca4 d465ac2dc9 Author: Mick Semb Wever Date: Wed Oct 19 11:32:42 2022 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Prepare debian changelog for 3.11.14 Prepare debian changelog for 3.0.28 commit d465ac2dc93f494b9585d7ef6789278c6c987243 Merge: 9d3327ef13 96c5332ee1 Author: Mick Semb Wever Date: Wed Oct 19 11:32:23 2022 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Prepare debian changelog for 3.0.28 commit 277fa4fca4a80eb327be6559f993c91e42dd4009 Author: Mick Semb Wever Date: Wed Oct 19 10:57:20 2022 +0200 Prepare debian changelog for 4.0.7 commit 9d3327ef1321fe1bf4e7fc73ed6111da7c994553 Author: Mick Semb Wever Date: Wed Oct 19 10:27:41 2022 +0200 Prepare debian changelog for 3.11.14 commit 96c5332ee15f45ca5410caaa787cc88d6947b3c9 Author: Mick Semb Wever Date: Wed Oct 19 09:59:09 2022 +0200 Prepare debian changelog for 3.0.28 commit 9a24fa81e5f44476c9994b9d9a7aabf9b89f3f8b Merge: a124f099e0 95d0a936f9 Author: Mick Semb Wever Date: Tue Oct 18 22:53:05 2022 +0200 Merge branch 'cassandra-3.11' into cassandra-4.0 * cassandra-3.11: Fix JMX security vulnerabilities commit 95d0a936f9611e6c43b79f6e4ba470c27d384567 Merge: 6b0cd7db04 b2660bcf78 Author: Mick Semb Wever Date: Tue Oct 18 22:49:07 2022 +0200 Merge branch 'cassandra-3.0' into cassandra-3.11 * cassandra-3.0: Fix JMX security vulnerabilities commit b2660bcf78ccf08a3a0ae39a8c9ffb397efef9ff Author: Abe Ratnofsky Date: Tue Aug 30 15:23:26 2022 -0700 Fix JMX security vulnerabilities patch by Abe Ratnofsky; reviewed by Jon Meredith, Mick Semb Wever, Sam Tunnicliffe for CASSANDRA-17921 commit a124f099e036a83b7ef2b69975b122dc83ba3e5f Merge: cc622a20e1 6b0cd7db04 Author: Brandon Williams Date: Tue Oct 18 11:16:20 2022 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 6b0cd7db043095904c500660a36ae27826f693a7 Merge: f13cdddcad 3d4e6944ef Author: Brandon Williams Date: Tue Oct 18 11:08:02 2022 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 3d4e6944ef11d96858507779b9280f5b9c198ffa Author: Brandon Williams Date: Mon Oct 17 10:49:22 2022 -0500 Suppress CVE-2019-2684 Patch by brandonwilliams; reviewed by smiklosovic for CASSANDRA-17965 commit cc622a20e1908d94626217b85737948b1df9b96a Merge: 4157e7a8e0 f13cdddcad Author: Andrés de la Peña Date: Tue Oct 18 16:51:29 2022 +0100 Merge branch 'cassandra-3.11' into cassandra-4.0 commit f13cdddcad17a4c674c3011e7fff714d4f89f683 Merge: 2e6528542b 9aa28a81ec Author: Andrés de la Peña Date: Tue Oct 18 16:47:08 2022 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 commit 9aa28a81ecb65d8e586039f419ca3272c1de64c2 Author: Andrés de la Peña Date: Sat Oct 15 15:38:37 2022 +0100 CircleCI: Automatically detect and repeat new or modified JUnit tests patch by Andrés de la Peña; reviewed by Josh McKenzie for CASSANDRA-17939 commit 4157e7a8e04655af4553d9003b9cb46897dddc2c Merge: 488c0c75a8 2e6528542b Author: Brandon Williams Date: Tue Oct 18 10:31:37 2022 -0500 Merge branch 'cassandra-3.11' into cassandra-4.0 commit 2e6528542b21a5d79eeba6d22ddc2a289805f98c Author: Brandon Williams Date: Mon Oct 17 10:12:08 2022 -0500 Suppress CVE-2022-42003 and CVE-2022-42004 Patch by brandonwilliams; reviewed by smiklosovic for CASSANDRA-17966 commit 488c0c75a8f632f2db4e3db39f2ebcf8a489971e Author: Stefan Miklosovic Date: Tue Oct 11 23:13:13 2022 +0200 Remove empty cq4 log files to prevent BinLog from failing to start This patch also backports CASSANDRA-17595. patch by Stefan Miklosovic; reviewed by Caleb Rackliffe for CASSANDRA-17933 commit e13356d75d2d3c200f1636337cf15329bd1b829b Author: Piotr Kołaczkowski Date: Fri May 6 09:39:21 2022 +0200 Fix multiple BufferPool bugs patch by Piotr Kołaczkowski; reviewed by Aleksey Yeschenko for CASSANDRA-16681 --- .build/build-owasp.xml | 2 +- .build/build-rat.xml | 13 +- .build/build-resolver.xml | 2 + .build/dependency-check-suppressions.xml | 42 + .circleci/config-2_1.yml | 1759 --- .circleci/config-2_1.yml.high_res.patch | 106 - .circleci/config.yml | 10872 ++++++++++++--- .circleci/config.yml.FREE | 11630 ++++++++++++++++ .circleci/config.yml.HIGHRES | 4330 ------ .circleci/config.yml.LOWRES | 4330 ------ .circleci/config.yml.MIDRES | 4330 ------ .circleci/config.yml.PAID | 11630 ++++++++++++++++ .circleci/config_template.yml | 3338 +++++ ...s.patch => config_template.yml.PAID.patch} | 258 +- .circleci/generate.sh | 317 +- .circleci/readme.md | 186 +- .gitignore | 1 + CHANGES.txt | 136 +- NEWS.txt | 24 + NOTICE.txt | 2 +- bin/cqlsh.py | 170 +- build.xml | 139 +- conf/cassandra-rackdc.properties | 6 + conf/cassandra.yaml | 2 + debian/changelog | 28 +- debian/rules | 1 - .../data_modeling/data_modeling_queries.adoc | 2 +- .../data_modeling/data_modeling_rdbms.adoc | 2 +- .../data_modeling/data_modeling_refining.adoc | 1 + .../pages/getting_started/production.adoc | 2 +- doc/scripts/gen-nodetool-docs.py | 28 +- ide/idea/vcs.xml | 10 + ide/idea/workspace.xml | 48 +- ide/nbproject/project.xml | 2 +- pylib/cassandra-cqlsh-tests.sh | 7 +- pylib/cqlshlib/copyutil.py | 1 + pylib/cqlshlib/formatting.py | 2 +- pylib/cqlshlib/saferscanner.py | 26 +- pylib/cqlshlib/test/cassconnect.py | 6 +- pylib/cqlshlib/test/test_cqlsh_output.py | 29 +- pylib/cqlshlib/test/test_keyspace_init.cql | 17 +- pylib/cqlshlib/test/test_sslhandling.py | 8 +- pylib/pytest.ini | 2 + pylib/requirements.txt | 3 +- relocate-dependencies.pom | 2 +- src/antlr/Lexer.g | 21 +- src/antlr/Parser.g | 12 +- .../cassandra/auth/CassandraRoleManager.java | 5 +- .../cassandra/auth/FunctionResource.java | 25 +- .../auth/jmx/AuthorizationProxy.java | 70 + .../config/CassandraRelevantProperties.java | 74 +- .../cassandra/config/DatabaseDescriptor.java | 5 + .../config/YamlConfigurationLoader.java | 6 +- .../cassandra/cql3/functions/CastFcts.java | 5 +- .../cql3/functions/FunctionName.java | 26 + .../cql3/functions/FunctionResolver.java | 2 +- .../cassandra/cql3/functions/UDAggregate.java | 4 +- .../cql3/functions/UDFByteCodeVerifier.java | 4 +- .../cql3/selection/ElementsSelector.java | 17 +- .../cassandra/cql3/selection/Selectable.java | 30 +- .../cql3/statements/DescribeStatement.java | 2 +- .../cql3/statements/PropertyDefinitions.java | 23 +- .../cql3/statements/SelectStatement.java | 30 + .../schema/CreateAggregateStatement.java | 3 + .../schema/CreateFunctionStatement.java | 3 + .../schema/CreateTableStatement.java | 10 +- .../cassandra/db/ClusteringComparator.java | 2 +- .../cassandra/db/ColumnFamilyStore.java | 17 +- .../org/apache/cassandra/db/Directories.java | 36 +- .../cassandra/db/SerializationHeader.java | 36 +- .../db/SinglePartitionReadCommand.java | 3 +- .../apache/cassandra/db/SystemKeyspace.java | 4 +- .../org/apache/cassandra/db/TypeSizes.java | 1 + .../cassandra/db/commitlog/CommitLog.java | 32 +- .../db/compaction/CompactionController.java | 20 +- .../db/compaction/CompactionManager.java | 50 +- .../db/compaction/CompactionRealm.java | 2 + .../db/compaction/CompactionTask.java | 50 +- .../TimeWindowCompactionStrategy.java | 14 + .../compaction/UnifiedCompactionStrategy.java | 1 + .../db/filter/ColumnSubselection.java | 8 +- .../db/lifecycle/LogTransaction.java | 16 +- .../cassandra/db/lifecycle/Tracker.java | 12 +- .../apache/cassandra/db/lifecycle/View.java | 15 + .../apache/cassandra/db/marshal/UserType.java | 2 +- .../memtable/AbstractAllocatorMemtable.java | 40 +- .../db/memtable/AbstractMemtable.java | 19 +- .../cassandra/db/memtable/Memtable.java | 2 + .../cassandra/db/memtable/TrieMemtable.java | 10 +- .../db/partitions/BTreePartitionUpdater.java | 16 +- .../repair/CassandraTableRepairManager.java | 27 +- .../apache/cassandra/db/rows/ArrayCell.java | 2 +- .../apache/cassandra/db/rows/BTreeRow.java | 7 +- .../apache/cassandra/db/rows/BufferCell.java | 2 +- .../apache/cassandra/db/rows/ColumnData.java | 49 +- .../cassandra/db/rows/ComplexColumnData.java | 2 +- .../apache/cassandra/db/rows/NativeCell.java | 18 +- .../org/apache/cassandra/db/rows/Row.java | 8 + .../db/rows/UnfilteredSerializer.java | 10 +- .../db/virtual/SystemViewsKeyspace.java | 18 +- .../tokenallocator/ReplicationStrategy.java | 2 +- .../dht/tokenallocator/TokenAllocation.java | 125 +- .../org/apache/cassandra/gms/Gossiper.java | 97 +- .../hints/HintsDispatchExecutor.java | 48 +- .../hints/HintsRateLimiterFactory.java | 76 + .../apache/cassandra/hints/HintsService.java | 18 + .../org/apache/cassandra/index/Index.java | 8 +- .../cassandra/index/IndexBuildDecider.java | 66 + .../index/SecondaryIndexManager.java | 249 +- .../index/sai/StorageAttachedIndex.java | 35 +- .../sai/StorageAttachedIndexBuilder.java | 1 + .../index/sai/StorageAttachedIndexGroup.java | 2 +- .../disk/v1/kdtree/TraversingBKDReader.java | 2 +- .../index/sai/virtual/IndexesSystemView.java | 4 + .../cassandra/io/sstable/Descriptor.java | 11 +- .../io/sstable/SSTableHeaderFix.java | 8 +- .../cassandra/io/sstable/SSTableId.java | 2 +- .../io/sstable/SequenceBasedSSTableId.java | 2 +- .../cassandra/io/sstable/StorageHandler.java | 4 +- .../io/sstable/ULIDBasedSSTableId.java | 2 +- .../io/sstable/UUIDBasedSSTableId.java | 2 +- .../io/util/TrackedDataInputPlus.java | 62 +- ...AbstractCloudMetadataServiceConnector.java | 99 + .../locator/Ec2MetadataServiceConnector.java | 214 + .../locator/Ec2MultiRegionSnitch.java | 26 +- .../apache/cassandra/locator/Ec2Snitch.java | 65 +- .../cassandra/locator/SnitchProperties.java | 8 + .../cassandra/metrics/KeyspaceMetrics.java | 3 + .../cassandra/metrics/TableMetrics.java | 9 +- .../cassandra/net/CustomParamsSerializer.java | 73 + .../org/apache/cassandra/net/Message.java | 18 + .../cassandra/net/MessagingService.java | 30 +- .../net/OutboundConnectionSettings.java | 2 + .../org/apache/cassandra/net/ParamType.java | 5 +- .../notifications/MetricsNotification.java | 58 - .../repair/messages/SyncRequest.java | 8 +- .../cassandra/schema/KeyspaceMetadata.java | 39 + .../schema/MigrationCoordinator.java | 6 +- .../cassandra/schema/SchemaKeyspace.java | 3 +- .../cassandra/schema/TableMetadata.java | 27 +- .../org/apache/cassandra/schema/Tables.java | 27 +- .../security/ThreadAwareSecurityManager.java | 3 +- .../serializers/CollectionSerializer.java | 8 + .../cassandra/serializers/ListSerializer.java | 9 +- .../cassandra/serializers/MapSerializer.java | 8 +- .../cassandra/serializers/SetSerializer.java | 4 +- .../service/ActiveRepairService.java | 15 +- .../service/DefaultFSErrorHandler.java | 31 +- .../apache/cassandra/service/GCInspector.java | 8 +- .../cassandra/service/StorageService.java | 53 +- .../cassandra/service/pager/PagingState.java | 8 + .../streaming/StreamConnectionFactory.java | 14 + .../async/NettyStreamingMessageSender.java | 2 +- .../tools/BulkLoadConnectionFactory.java | 6 + .../org/apache/cassandra/tools/NodeProbe.java | 38 +- .../tools/SSTableMetadataViewer.java | 1 + .../cassandra/tools/StandaloneUpgrader.java | 4 +- .../tools/nodetool/DescribeCluster.java | 2 +- .../tools/nodetool/EnableFullQueryLog.java | 3 +- .../apache/cassandra/tools/nodetool/Info.java | 16 + .../nodetool/stats/TableStatsHolder.java | 2 + .../apache/cassandra/utils/ByteArrayUtil.java | 24 + .../org/apache/cassandra/utils}/Isolated.java | 4 +- .../cassandra/utils/JMXServerUtils.java | 28 +- .../utils/JVMStabilityInspector.java | 14 +- .../apache/cassandra/utils/MBeanWrapper.java | 212 +- .../utils/RMIClientSocketFactoryImpl.java | 62 + .../cassandra/utils/ReflectionUtils.java | 55 + .../org/apache/cassandra/utils}/Shared.java | 2 +- .../apache/cassandra/utils/binlog/BinLog.java | 79 +- .../cassandra/utils/binlog/BinLogOptions.java | 7 + .../apache/cassandra/utils/btree/BTree.java | 15 +- .../cassandra/utils/concurrent/Ref.java | 2 +- .../cassandra/utils/memory/BufferPool.java | 4 +- .../cassandra/utils/memory/MemtablePool.java | 2 +- .../cassandra/anttasks/TestNameCheckTask.java | 85 + .../apache/cassandra/utils/LongBTreeTest.java | 17 +- test/conf/cassandra.yaml | 2 + .../distributed/action/GossipHelper.java | 12 + .../distributed/impl/AbstractCluster.java | 24 +- .../CollectingRMIServerSocketFactoryImpl.java | 87 + .../impl/INodeProvisionStrategy.java | 11 + .../cassandra/distributed/impl/Instance.java | 41 +- .../distributed/impl/InstanceConfig.java | 24 +- .../distributed/impl/IsolatedExecutor.java | 2 +- .../distributed/impl/IsolatedJmx.java | 230 + .../distributed/shared/ClusterUtils.java | 1 + .../cassandra/distributed/test/AuthTest.java | 163 +- .../test/BootstrapBinaryDisabledTest.java | 2 +- ...amples.java => ByteBuddyExamplesTest.java} | 2 +- .../test/CompactionControllerConfigTest.java | 24 - .../CompactionOverlappingSSTableTest.java | 114 + .../distributed/test/FailingRepairTest.java | 10 +- .../distributed/test/IPMembershipTest.java | 5 +- .../distributed/test/IndexDroppingTest.java | 122 + .../test/InternodeEncryptionOptionsTest.java | 12 +- ...> JVMStabilityInspectorThrowableTest.java} | 69 +- .../NativeTransportEncryptionOptionsTest.java | 4 + ...ur.java => ReprepareOldBehaviourTest.java} | 2 +- .../distributed/test/ResourceLeakTest.java | 60 + .../distributed/test/jmx/JMXFeatureTest.java | 113 + .../test/jmx/JMXGetterCheckTest.java | 136 + .../test/metric/TableMetricTest.java | 29 +- .../test/ring/CleanupFailureTest.java | 111 + .../test/sai/AbstractQueryTester.java | 2 +- .../test/sai/IndexConsistencyTest.java | 2 +- ...ompactStoragePagingWithProtocolTester.java | 179 + ...mpactStoragePagingWithProtocolV30Test.java | 33 + ...mpactStoragePagingWithProtocolV3XTest.java | 33 + .../upgrade/MixedModeMessageForwardTest.java | 2 +- .../test/microbench/CachingBenchTest.java} | 7 +- .../microbench/GcCompactionBenchTest.java} | 7 +- .../audit/AuditLoggerCleanupTest.java | 145 + .../cassandra/auth/FunctionResourceTest.java | 45 +- .../org/apache/cassandra/cql3/BatchTest.java | 5 +- .../org/apache/cassandra/cql3/CQLTester.java | 17 +- .../cassandra/cql3/GcCompactionTest.java | 58 +- ...TestSplit1.java => KeywordSplit1Test.java} | 4 +- ...TestSplit2.java => KeywordSplit2Test.java} | 4 +- .../cassandra/cql3/KeywordTestBase.java | 2 +- .../apache/cassandra/cql3/ViewSchemaTest.java | 2 +- .../cql3/functions/CastFctsTest.java | 2 +- .../cql3/functions/OperationFctsTest.java | 10 +- .../CreateTableStatementGraphTest.java | 4 + .../statements/DescribeStatementTest.java | 147 + .../statements/PropertyDefinitionsTest.java | 79 +- .../validation/entities/CollectionsTest.java | 44 +- .../cql3/validation/entities/UFTest.java | 20 + .../operations/AggregationTest.java | 17 + .../operations/CompactStorageTest.java | 17 +- .../validation/operations/CreateTest.java | 17 + .../cql3/validation/operations/DropTest.java | 9 + .../validation/operations/InsertTest.java | 38 + .../org/apache/cassandra/db/CleanupTest.java | 6 +- .../cassandra/db/ColumnFamilyMetricTest.java | 11 +- .../cassandra/db/ColumnFamilyStoreTest.java | 145 +- .../apache/cassandra/db/DirectoriesTest.java | 179 +- .../org/apache/cassandra/db/ImportTest.java | 44 +- .../org/apache/cassandra/db/KeyCacheTest.java | 8 +- .../CommitLogInitWithExceptionTest.java | 33 +- .../compaction/BackgroundCompactionsTest.java | 37 +- .../BaseCompactionStrategyTest.java | 8 +- .../db/compaction/NeverPurgeTest.java | 9 +- .../db/compaction/PartialCompactionsTest.java | 208 + .../TimeWindowCompactionStrategyTest.java | 26 + .../cassandra/db/lifecycle/TrackerTest.java | 13 - .../db/memtable/MemtableSizeTestBase.java | 2 +- ...cBTreePartitionMemtableAccountingTest.java | 422 + .../db/rows/UnfilteredSerializerTest.java | 157 + .../apache/cassandra/dht/SplitterTest.java | 37 +- .../cassandra/fql/FullQueryLoggerTest.java | 33 +- .../index/SecondaryIndexManagerTest.java | 71 + .../index/sai/cql/IndexParallelBuildTest.java | 123 + .../cassandra/io/sstable/DescriptorTest.java | 152 + .../io/sstable/SSTableHeaderFixTest.java | 13 +- .../io/sstable/SSTableLoaderTest.java | 52 +- .../cassandra/io/sstable/UDTMoveTest.java | 38 +- .../cassandra/locator/Ec2ConnectorTest.java | 106 + ...{EC2SnitchTest.java => Ec2SnitchTest.java} | 155 +- .../org/apache/cassandra/net/MessageTest.java | 36 + .../schema/CreateTableValidationTest.java | 69 +- .../schema/KeyspaceMetadataTest.java | 45 + .../apache/cassandra/schema/MockSchema.java | 20 +- .../service/DefaultFSErrorHandlerTest.java | 121 + .../service/DiskFailurePolicyTest.java | 164 + .../cassandra/service/StorageServiceTest.java | 56 + .../cassandra/tools/NodeToolTPStatsTest.java | 1 + .../stats/NodetoolTableStatsTest.java | 31 + .../nodetool/stats/TableStatsPrinterTest.java | 129 +- .../cassandra/transport/SerDeserTest.java | 36 + .../utils/JVMStabilityInspectorTest.java | 14 +- .../cassandra/utils/KillerForTests.java | 5 + .../org/apache/cassandra/utils/UUIDTest.java | 1 - .../cassandra/fqltool/commands/Dump.java | 28 +- .../cassandra/fqltool/commands/DumpTest.java | 147 + .../stress/settings/StressSettings.java | 3 +- .../stress/util/JavaDriverClient.java | 27 +- ...stence and virtual tables for Nodes (#445) | 17 + ...ageProvider for clearing file system cache | 28 + ...R-1260: Improve testing performance (#424) | 80 + ... CL API to update to remote storage (#404) | 33 + .../STAR-2000/237-49646a9e27 STAR-1150 (#362) | 15 + ... STAR-782: Improves compaction performance | 23 + ...06: End-to-end Row Awareness for SAI (#97) | 34 + ... STAR-894 Port Mockable FS from OSS (#269) | 159 + ...STable and CFS abstractions for compaction | 64 + update-history/STAR-2000/294-2ab888d2ad | 47 + ...dd the Unified Compaction Strategy. (#132) | 41 + ...e them easily accessible on Fallout (#101) | 126 + ...7: TrieIndex SSTable format implementation | 43 + .../372-319fdb7440 STAR-14: Memtable API | 17 + ... to be called dse-db rather than cassandra | 21 + ...-5791: Add Cross-region LWT support (#587) | 18 + ...cy and build fixes for Converged Cassandra | 15 + ...ment sstable generation identifier as uuid | 35 + ...ce can be opened while it is being removed | 15 + ...eviewed by Alex Petrov for CASSANDRA-17582 | 21 + ...nt to allow for schema source pluggability | 72 + ...litter (invalid number of splits produced) | 22 + ...atchTest and UUIDTests to run fully (#563) | 13 + .../STAR-2000/97-0f8abbcf7c STAR-1489 (#524) | 44 + ... consistency level in query options (#525) | 21 + ...: Fix NativeTransportEncryptionOptionsTest | 89 + ...be defined per segment manager path (#505) | 68 + ...STAR-1497 Fixes for CNDB SchemaTest (#497) | 14 + ...stence and virtual tables for Nodes (#445) | 441 + ... running time of testRandomRangeAndBatches | 14 + ...TAR-1320: Upgrade dtest-api to 0.13 (#442) | 13 + ...ml to avoid spurious failures in CI (#431) | 17 + ...R-1260: Improve testing performance (#424) | 67 + ...1248: Report leaks as test failures (#418) | 44 + ...y to establish TLSv1.1 connections in test | 77 + ... CL API to update to remote storage (#404) | 16 + ...a0 STAR-871: Fix TopologyChangeTest (#377) | 51 + ...cache invalidation after compaction (#371) | 14 + .../239-a0f3766847 STAR-1150 (#362) | 40 + ... STAR-782: Improves compaction performance | 61 + ...pace name in statements and results (#323) | 29 + ...table and keyspace thresholds in CC (#646) | 115 + ...tore Raw CQL query inside statement (#333) | 17 + ... store File instead of String value (#320) | 16 + ...formance improvements to Nodes impl (#292) | 31 + ... to discard flushed remote sstables (#310) | 16 + ...-996: Implement node information component | 152 + ...le system abstractions to Stargazer (#273) | 135 + ... STAR-894 Port Mockable FS from OSS (#269) | 523 + ...STable and CFS abstractions for compaction | 73 + ...mbership from cndb (token metadata) (#242) | 35 + ...po during CNDB FailureDetector port (#245) | 57 + ...ory for vector similarity functions (#630) | 94 + ...alyzer and some language unit tests (#192) | 27 + ...STAR-762: Implement paging in bytes (#223) | 16 + ...dd the Unified Compaction Strategy. (#132) | 82 + ...ure-connect-bundle option to cqlsh. (#190) | 88 + ...-level and serial-consistency-level (#173) | 73 + ...cqlsh disable cqlsh_history logging (#165) | 60 + ...sitive rate includes true negatives (#164) | 16 + ...: Port guardrails from astra branch (#124) | 46 + ...e them easily accessible on Fallout (#101) | 168 + ...-501: Fixed number of CPUs = 2 for testing | 14 + update-history/oss-4.0.11/364-74b96a8f69 | 136 + ...action task ID in compaction history (#93) | 65 + .../370-1f9e56a62f STAR-14: Memtable API | 137 + ...ded to mark on postings reader close (#87) | 16 + ... to be called dse-db rather than cassandra | 33 + ...for other implementations of SSTableFormat | 20 + ...cy and build fixes for Converged Cassandra | 14 + ...ment sstable generation identifier as uuid | 127 + ...eviewed by Alex Petrov for CASSANDRA-17582 | 22 + ...nt to allow for schema source pluggability | 209 + ...saging metrics impl is configurable (#594) | 14 + ...afd76d9 Add Mutation Serialization Caching | 20 + ...to add Micrometer BufferPoolMetrics (#562) | 54 + ... of iterations etc in LongBTreeTest (#554) | 93 + update-history/oss-4.0.11/78-3c68b149bc | 19 + 355 files changed, 49386 insertions(+), 18083 deletions(-) delete mode 100644 .circleci/config-2_1.yml delete mode 100644 .circleci/config-2_1.yml.high_res.patch create mode 100644 .circleci/config.yml.FREE delete mode 100644 .circleci/config.yml.HIGHRES delete mode 100644 .circleci/config.yml.LOWRES delete mode 100644 .circleci/config.yml.MIDRES create mode 100644 .circleci/config.yml.PAID create mode 100644 .circleci/config_template.yml rename .circleci/{config-2_1.yml.mid_res.patch => config_template.yml.PAID.patch} (51%) create mode 100644 pylib/pytest.ini create mode 100644 src/java/org/apache/cassandra/hints/HintsRateLimiterFactory.java create mode 100644 src/java/org/apache/cassandra/index/IndexBuildDecider.java create mode 100644 src/java/org/apache/cassandra/locator/AbstractCloudMetadataServiceConnector.java create mode 100644 src/java/org/apache/cassandra/locator/Ec2MetadataServiceConnector.java create mode 100644 src/java/org/apache/cassandra/net/CustomParamsSerializer.java delete mode 100644 src/java/org/apache/cassandra/notifications/MetricsNotification.java rename {test/distributed/org/apache/cassandra/distributed/shared => src/java/org/apache/cassandra/utils}/Isolated.java (95%) create mode 100644 src/java/org/apache/cassandra/utils/RMIClientSocketFactoryImpl.java create mode 100644 src/java/org/apache/cassandra/utils/ReflectionUtils.java rename {test/distributed/org/apache/cassandra/distributed/shared => src/java/org/apache/cassandra/utils}/Shared.java (96%) create mode 100644 test/anttasks/org/apache/cassandra/anttasks/TestNameCheckTask.java create mode 100644 test/distributed/org/apache/cassandra/distributed/impl/CollectingRMIServerSocketFactoryImpl.java create mode 100644 test/distributed/org/apache/cassandra/distributed/impl/IsolatedJmx.java rename test/distributed/org/apache/cassandra/distributed/test/{ByteBuddyExamples.java => ByteBuddyExamplesTest.java} (98%) create mode 100644 test/distributed/org/apache/cassandra/distributed/test/CompactionOverlappingSSTableTest.java create mode 100644 test/distributed/org/apache/cassandra/distributed/test/IndexDroppingTest.java rename test/distributed/org/apache/cassandra/distributed/test/{JVMStabilityInspectorCorruptSSTableExceptionTest.java => JVMStabilityInspectorThrowableTest.java} (70%) rename test/distributed/org/apache/cassandra/distributed/test/{ReprepareTestOldBehaviour.java => ReprepareOldBehaviourTest.java} (98%) create mode 100644 test/distributed/org/apache/cassandra/distributed/test/jmx/JMXFeatureTest.java create mode 100644 test/distributed/org/apache/cassandra/distributed/test/jmx/JMXGetterCheckTest.java create mode 100644 test/distributed/org/apache/cassandra/distributed/test/ring/CleanupFailureTest.java create mode 100644 test/distributed/org/apache/cassandra/distributed/upgrade/CompactStoragePagingWithProtocolTester.java create mode 100644 test/distributed/org/apache/cassandra/distributed/upgrade/CompactStoragePagingWithProtocolV30Test.java create mode 100644 test/distributed/org/apache/cassandra/distributed/upgrade/CompactStoragePagingWithProtocolV3XTest.java rename test/{long/org/apache/cassandra/cql3/CachingBench.java => microbench/org/apache/cassandra/test/microbench/CachingBenchTest.java} (98%) rename test/{long/org/apache/cassandra/cql3/GcCompactionBench.java => microbench/org/apache/cassandra/test/microbench/GcCompactionBenchTest.java} (98%) create mode 100644 test/unit/org/apache/cassandra/audit/AuditLoggerCleanupTest.java rename test/unit/org/apache/cassandra/cql3/{KeywordTestSplit1.java => KeywordSplit1Test.java} (92%) rename test/unit/org/apache/cassandra/cql3/{KeywordTestSplit2.java => KeywordSplit2Test.java} (92%) create mode 100644 test/unit/org/apache/cassandra/db/compaction/PartialCompactionsTest.java create mode 100644 test/unit/org/apache/cassandra/db/partitions/AtomicBTreePartitionMemtableAccountingTest.java create mode 100644 test/unit/org/apache/cassandra/db/rows/UnfilteredSerializerTest.java create mode 100644 test/unit/org/apache/cassandra/index/sai/cql/IndexParallelBuildTest.java create mode 100644 test/unit/org/apache/cassandra/locator/Ec2ConnectorTest.java rename test/unit/org/apache/cassandra/locator/{EC2SnitchTest.java => Ec2SnitchTest.java} (60%) create mode 100644 test/unit/org/apache/cassandra/service/DefaultFSErrorHandlerTest.java create mode 100644 test/unit/org/apache/cassandra/service/DiskFailurePolicyTest.java create mode 100644 tools/fqltool/test/unit/org/apache/cassandra/fqltool/commands/DumpTest.java create mode 100644 update-history/STAR-2000/133-01e3c1b3f3 STAR-1329 Implement storage persistence and virtual tables for Nodes (#445) create mode 100644 update-history/STAR-2000/152-2c271f06c1 STAR-1335: port CNDB-4508 - add abstraction to StorageProvider for clearing file system cache create mode 100644 update-history/STAR-2000/175-2b9a184d12 STAR-1260: Improve testing performance (#424) create mode 100644 update-history/STAR-2000/195-33cb1c6cae STAR-1230 Provide CL API to update to remote storage (#404) create mode 100644 update-history/STAR-2000/237-49646a9e27 STAR-1150 (#362) create mode 100644 update-history/STAR-2000/239-702dc7b6ea STAR-782: Improves compaction performance create mode 100644 update-history/STAR-2000/273-3395487679 STAR-406: End-to-end Row Awareness for SAI (#97) create mode 100644 update-history/STAR-2000/285-7ae967f2f0 STAR-894 Port Mockable FS from OSS (#269) create mode 100644 update-history/STAR-2000/289-48e1cec32e STAR-791: Introduce SSTable and CFS abstractions for compaction create mode 100644 update-history/STAR-2000/294-2ab888d2ad create mode 100644 update-history/STAR-2000/314-fb05a5a4a3 STAR-13 Add the Unified Compaction Strategy. (#132) create mode 100644 update-history/STAR-2000/353-3d134e119b STAR-410: Define compaction metrics and make them easily accessible on Fallout (#101) create mode 100644 update-history/STAR-2000/370-1f9fc75f80 STAR-247: TrieIndex SSTable format implementation create mode 100644 update-history/STAR-2000/372-319fdb7440 STAR-14: Memtable API create mode 100644 update-history/STAR-2000/387-22fb77caa1 STAR-172: Update artifacts to be called dse-db rather than cassandra create mode 100644 update-history/STAR-2000/39-351ea78f66 CNDB-5791: Add Cross-region LWT support (#587) create mode 100644 update-history/STAR-2000/421-7530b3fb84 Initial dependency and build fixes for Converged Cassandra create mode 100644 update-history/STAR-2000/423-7b0546a8c5 Implement sstable generation identifier as uuid create mode 100644 update-history/STAR-2000/424-e923f2ed62 Fix a race condition where a keyspace can be opened while it is being removed create mode 100644 update-history/STAR-2000/426-ec4f28899c Add information info whether sstables are dropped or not to SchemaChangeListener patch by Jacek Lewandowski; reviewed by Alex Petrov for CASSANDRA-17582 create mode 100644 update-history/STAR-2000/427-3ca9f09df5 Refactor schema management to allow for schema source pluggability create mode 100644 update-history/STAR-2000/48-d4bb82428b STAR-1590: Fix Splitter (invalid number of splits produced) create mode 100644 update-history/STAR-2000/58-4d537da980 STAR-1626 Enable BatchTest and UUIDTests to run fully (#563) create mode 100644 update-history/STAR-2000/97-0f8abbcf7c STAR-1489 (#524) create mode 100644 update-history/STAR-2000/98-95e69bd916 Add method to update consistency level in query options (#525) create mode 100644 update-history/oss-4.0.11/102-56310209ca STAR-1191: Fix NativeTransportEncryptionOptionsTest create mode 100644 update-history/oss-4.0.11/119-bb6d9422ed STAR-1493 replayId should be defined per segment manager path (#505) create mode 100644 update-history/oss-4.0.11/130-0f144bb256 STAR-1497 Fixes for CNDB SchemaTest (#497) create mode 100644 update-history/oss-4.0.11/142-83fd2915bb STAR-1329 Implement storage persistence and virtual tables for Nodes (#445) create mode 100644 update-history/oss-4.0.11/146-bbdb810e51 STAR-261: Reduce the running time of testRandomRangeAndBatches create mode 100644 update-history/oss-4.0.11/170-48fe81b217 STAR-1320: Upgrade dtest-api to 0.13 (#442) create mode 100644 update-history/oss-4.0.11/175-5a2d16d4ed STAR-1291: bump server side request timeouts in test cassandra.yaml to avoid spurious failures in CI (#431) create mode 100644 update-history/oss-4.0.11/179-b007c540b0 STAR-1260: Improve testing performance (#424) create mode 100644 update-history/oss-4.0.11/183-18755c8ac8 STAR-1248: Report leaks as test failures (#418) create mode 100644 update-history/oss-4.0.11/185-817e0a16a0 STAR-1261: Don't try to establish TLSv1.1 connections in test create mode 100644 update-history/oss-4.0.11/199-c47530b3cf STAR-1230 Provide CL API to update to remote storage (#404) create mode 100644 update-history/oss-4.0.11/225-5959c4d1a0 STAR-871: Fix TopologyChangeTest (#377) create mode 100644 update-history/oss-4.0.11/230-ad2d31da42 STAR-1177: Fix accidentally removed keycache invalidation after compaction (#371) create mode 100644 update-history/oss-4.0.11/239-a0f3766847 STAR-1150 (#362) create mode 100644 update-history/oss-4.0.11/241-41ee610f8d STAR-782: Improves compaction performance create mode 100644 update-history/oss-4.0.11/249-d1c489d052 STAR-1061: Make it possible to override keyspace name in statements and results (#323) create mode 100644 update-history/oss-4.0.11/25-36dc86ab76 CNDB-6974 Remove table and keyspace thresholds in CC (#646) create mode 100644 update-history/oss-4.0.11/250-d21078359e STAR-1097 Store Raw CQL query inside statement (#333) create mode 100644 update-history/oss-4.0.11/257-02a92fdff0 STAR-1056 Update FSError to store File instead of String value (#320) create mode 100644 update-history/oss-4.0.11/264-61fe029e69 STAR-893: apply some performance improvements to Nodes impl (#292) create mode 100644 update-history/oss-4.0.11/265-a822bcb26a STAR-992: add APIs for replayer to discard flushed remote sstables (#310) create mode 100644 update-history/oss-4.0.11/266-a7a8296417 STAR-996: Implement node information component create mode 100644 update-history/oss-4.0.11/283-da98fddb2e CNDB-1707: Port remote file system abstractions to Stargazer (#273) create mode 100644 update-history/oss-4.0.11/287-3528cd9b6a STAR-894 Port Mockable FS from OSS (#269) create mode 100644 update-history/oss-4.0.11/291-6e4b5bd638 STAR-791: Introduce SSTable and CFS abstractions for compaction create mode 100644 update-history/oss-4.0.11/298-19e533726f STAR-789: Refactor and port pluggable cluster membership from cndb (token metadata) (#242) create mode 100644 update-history/oss-4.0.11/299-294407b027 STAR-868: Fix test failure caused by typo during CNDB FailureDetector port (#245) create mode 100644 update-history/oss-4.0.11/30-bf490bcb2d backport FunctionFactory for vector similarity functions (#630) create mode 100644 update-history/oss-4.0.11/300-d6c1172cd8 STAR-123 custom lucene analyzer and some language unit tests (#192) create mode 100644 update-history/oss-4.0.11/303-8dd49b0b9d STAR-762: Implement paging in bytes (#223) create mode 100644 update-history/oss-4.0.11/314-dc58b0b06f STAR-13 Add the Unified Compaction Strategy. (#132) create mode 100644 update-history/oss-4.0.11/321-01c45e5da1 STAR-765. Add --secure-connect-bundle option to cqlsh. (#190) create mode 100644 update-history/oss-4.0.11/328-abe4d5ee24 STAR-432. Add options for consistency-level and serial-consistency-level (#173) create mode 100644 update-history/oss-4.0.11/329-1b586e9381 STAR-429 cqlsh disable cqlsh_history logging (#165) create mode 100644 update-history/oss-4.0.11/331-d4ecbd21a1 STAR-725 Bloom false positive rate includes true negatives (#164) create mode 100644 update-history/oss-4.0.11/350-29c90ac629 STAR-409: Port guardrails from astra branch (#124) create mode 100644 update-history/oss-4.0.11/351-41b49c90fe STAR-410: Define compaction metrics and make them easily accessible on Fallout (#101) create mode 100644 update-history/oss-4.0.11/356-798c68121d STAR-501: Fixed number of CPUs = 2 for testing create mode 100644 update-history/oss-4.0.11/364-74b96a8f69 create mode 100644 update-history/oss-4.0.11/365-a4858b1bcd STAR-392: Reuse compaction task ID in compaction history (#93) create mode 100644 update-history/oss-4.0.11/370-1f9e56a62f STAR-14: Memtable API create mode 100644 update-history/oss-4.0.11/372-c6ed9b1447 STAR-348 moved postings decoded to mark on postings reader close (#87) create mode 100644 update-history/oss-4.0.11/374-b44af21ed3 STAR-172: Update artifacts to be called dse-db rather than cassandra create mode 100644 update-history/oss-4.0.11/377-998a3d25dc STAR-15: Allow for other implementations of SSTableFormat create mode 100644 update-history/oss-4.0.11/397-bffb086801 Initial dependency and build fixes for Converged Cassandra create mode 100644 update-history/oss-4.0.11/399-8dcf00553c Implement sstable generation identifier as uuid create mode 100644 update-history/oss-4.0.11/402-579907a58a Add information info whether sstables are dropped or not to SchemaChangeListener patch by Jacek Lewandowski; reviewed by Alex Petrov for CASSANDRA-17582 create mode 100644 update-history/oss-4.0.11/403-77eb6abfa4 Refactor schema management to allow for schema source pluggability create mode 100644 update-history/oss-4.0.11/48-35fe294ce1 CNDB-6235 messaging metrics impl is configurable (#594) create mode 100644 update-history/oss-4.0.11/50-12bafd76d9 Add Mutation Serialization Caching create mode 100644 update-history/oss-4.0.11/60-aeb8226d4e STAR-1705 Port CNDB-4974 changes to add Micrometer BufferPoolMetrics (#562) create mode 100644 update-history/oss-4.0.11/65-d816d10b01 reduce number of iterations etc in LongBTreeTest (#554) create mode 100644 update-history/oss-4.0.11/78-3c68b149bc diff --git a/.build/build-owasp.xml b/.build/build-owasp.xml index f3174999e8d4..a792730fb993 100644 --- a/.build/build-owasp.xml +++ b/.build/build-owasp.xml @@ -17,7 +17,7 @@ ~ limitations under the License. --> - + diff --git a/.build/build-rat.xml b/.build/build-rat.xml index ce8476415852..8488df31bdf4 100644 --- a/.build/build-rat.xml +++ b/.build/build-rat.xml @@ -27,7 +27,7 @@ - + @@ -35,7 +35,10 @@ - + + + + @@ -44,9 +47,9 @@ - + - + @@ -98,7 +101,7 @@ - + diff --git a/.build/build-resolver.xml b/.build/build-resolver.xml index c42a5fbd12cc..890b3aaf44f3 100644 --- a/.build/build-resolver.xml +++ b/.build/build-resolver.xml @@ -57,6 +57,8 @@ + + diff --git a/.build/dependency-check-suppressions.xml b/.build/dependency-check-suppressions.xml index 9a84700c64af..ae7ff368a7ac 100644 --- a/.build/dependency-check-suppressions.xml +++ b/.build/dependency-check-suppressions.xml @@ -24,6 +24,7 @@ ^pkg:maven/com\.datastax\.cassandra/cassandra\-driver\-core@.*$ CVE-2018-8016 + CVE-2019-2684 CVE-2020-13946 CVE-2020-17516 CVE-2021-44521 @@ -31,11 +32,17 @@ ^pkg:maven/org\.yaml/snakeyaml@.*$ + CVE-2023-2251 CVE-2022-25857 CVE-2022-38749 CVE-2022-38750 CVE-2022-38751 CVE-2022-38752 + CVE-2022-41854 + CVE-2022-1471 + CVE-2022-3064 + CVE-2021-4235 + CVE-2017-18640 @@ -46,6 +53,24 @@ ^pkg:maven/com\.google\.guava/guava@.*$ CVE-2020-8908 + CVE-2023-2976 + + + + ^pkg:maven/org\.apache\.commons.*$ + CVE-2021-37533 + + + ^pkg:maven/commons-io/.*$ + CVE-2021-37533 + + + ^pkg:maven/commons-cli/.*$ + CVE-2021-37533 + + + ^pkg:maven/commons-codec/.*$ + CVE-2021-37533 @@ -57,5 +82,22 @@ CVE-2021-37137 CVE-2021-43797 CVE-2022-24823 + CVE-2022-41881 + CVE-2022-41915 + CVE-2023-34462 + + + + ^pkg:maven/com\.fasterxml\.jackson\.core/jackson\-databind@.*$ + CVE-2022-42003 + CVE-2022-42004 + CVE-2023-35116 + + + + + ^pkg:maven/com\.fasterxml\.jackson\.core/jackson\-core@.*$ + CVE-2022-45688 + diff --git a/.circleci/config-2_1.yml b/.circleci/config-2_1.yml deleted file mode 100644 index 6a2aa5a50b17..000000000000 --- a/.circleci/config-2_1.yml +++ /dev/null @@ -1,1759 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -version: 2.1 - -default_env_vars: &default_env_vars - - # The values of some of these environment variables are meant to be frequently changed by developers. - # The generate.sh script contains a list of accepted environment variables that should contain some of - # these variables. Also, some variables are mentioned in the documentation, at least in - # .circleci/readme.md and in doc/source/development/testing.rst. - # If you modify these variables, or if you add new variables whose values are meant to be changed frequently, - # please remember to modify the generate.sh script and the documentation accordingly. - - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - LANG: en_US.UTF-8 - KEEP_TEST_DIR: true - DEFAULT_DIR: /home/cassandra/cassandra-dtest - PYTHONIOENCODING: utf-8 - PYTHONUNBUFFERED: true - CASS_DRIVER_NO_EXTENSIONS: true - CASS_DRIVER_NO_CYTHON: true - #Skip all syncing to disk to avoid performance issues in flaky CI environments - CASSANDRA_SKIP_SYNC: true - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - # The Ant test target to run, for example: - # REPEATED_UTEST_TARGET: testsome - # REPEATED_UTEST_TARGET: test-jvm-dtest-some - # REPEATED_UTEST_TARGET: test-cdc - # REPEATED_UTEST_TARGET: test-compression - # REPEATED_UTEST_TARGET: test-system-keyspace-directory - REPEATED_UTEST_TARGET: testsome - # The name of JUnit class to be run multiple times, for example: - # REPEATED_UTEST_CLASS: org.apache.cassandra.cql3.ViewTest - # REPEATED_UTEST_CLASS: org.apache.cassandra.distributed.test.PagingTest - REPEATED_UTEST_CLASS: - # The optional specific methods within REPEATED_UTEST_CLASS to be run, for example: - # REPEATED_UTEST_METHODS: testCompoundPartitionKey - # REPEATED_UTEST_METHODS: testCompoundPartitionKey,testStaticTable - # Please note that some Ant targets will ignore the -Dtest.methods argument produced by this. - REPEATED_UTEST_METHODS: - # The number of times that the repeated JUnit test should be run - REPEATED_UTEST_COUNT: 100 - # Whether the test iteration should stop on the first failure - REPEATED_UTEST_STOP_ON_FAILURE: false - - # A Python dtest to be run multiple times, for example: - # REPEATED_DTEST_NAME: cqlsh_tests/test_cqlsh.py - # REPEATED_DTEST_NAME: cqlsh_tests/test_cqlsh.py::TestCqlshSmoke - # REPEATED_DTEST_NAME: cqlsh_tests/test_cqlsh.py::TestCqlshSmoke::test_create_index - REPEATED_DTEST_NAME: - # Whether the repeated Python dtest should use vnodes - REPEATED_DTEST_VNODES: false - # The number of times that the repeated Python dtest should be run - REPEATED_DTEST_COUNT: 100 - # Whether the test iteration should stop on the first failure - REPEATED_DTEST_STOP_ON_FAILURE: false - - # A Python upgrade dtest to be run multiple times, for example: - # REPEATED_UPGRADE_DTEST_NAME: upgrade_tests/cql_tests.py - # REPEATED_UPGRADE_DTEST_NAME: upgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_current_4_0_x_To_indev_4_0_x - REPEATED_UPGRADE_DTEST_NAME: - # The number of times that the repeated Python upgrade dtest should be run - REPEATED_UPGRADE_DTEST_COUNT: 100 - # Whether the test iteration should stop on the first failure - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - # The name of JVM upgrade dtest class to be run multiple times, for example: - # REPEATED_JVM_UPGRADE_DTEST_CLASS: org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityV30Test - REPEATED_JVM_UPGRADE_DTEST_CLASS: - # The optional specific methods within REPEATED_JVM_UPGRADE_DTEST_CLASS to be run, for example: - # REPEATED_JVM_UPGRADE_DTEST_METHODS: testAvailabilityV30ToV3X - # REPEATED_JVM_UPGRADE_DTEST_METHODS: testAvailabilityV30ToV3X,testAvailabilityV30ToV4 - REPEATED_JVM_UPGRADE_DTEST_METHODS: - # The number of times that the repeated JVM upgrade dtest should be run - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - # Whether the JVM upgrade dtest iteration should stop on the first failure - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - -j8_par_executor: &j8_par_executor - executor: - name: java8-executor - #exec_resource_class: xlarge - parallelism: 4 - -j8_small_par_executor: &j8_small_par_executor - executor: - name: java8-executor - #exec_resource_class: xlarge - parallelism: 1 - -j8_small_executor: &j8_small_executor - executor: - name: java8-executor - exec_resource_class: medium - parallelism: 1 - -j8_medium_par_executor: &j8_medium_par_executor - executor: - name: java8-executor - #exec_resource_class: xlarge - parallelism: 1 - -j8_seq_executor: &j8_seq_executor - executor: - name: java8-executor - #exec_resource_class: xlarge - parallelism: 1 # sequential, single container tests: no parallelism benefits - -j11_par_executor: &j11_par_executor - executor: - name: java11-executor - #exec_resource_class: xlarge - parallelism: 4 - -j11_small_par_executor: &j11_small_par_executor - executor: - name: java11-executor - #exec_resource_class: xlarge - parallelism: 1 - -j11_small_executor: &j11_small_executor - executor: - name: java11-executor - #exec_resource_class: medium - parallelism: 1 - -j8_repeated_utest_executor: &j8_repeated_utest_executor - executor: - name: java8-executor - parallelism: 4 - -j8_repeated_dtest_executor: &j8_repeated_dtest_executor - executor: - name: java8-executor - parallelism: 4 - -j8_repeated_upgrade_dtest_executor: &j8_repeated_upgrade_dtest_executor - executor: - name: java8-executor - parallelism: 4 - -j8_repeated_jvm_upgrade_dtest_executor: &j8_repeated_jvm_upgrade_dtest_executor - executor: - name: java8-executor - parallelism: 4 - -j11_repeated_utest_executor: &j11_repeated_utest_executor - executor: - name: java11-executor - parallelism: 4 - -j11_repeated_dtest_executor: &j11_repeated_dtest_executor - executor: - name: java11-executor - parallelism: 4 - -j8_with_dtests_jobs: &j8_with_dtests_jobs - jobs: - - start_j8_build: - type: approval - - j8_build: - requires: - - start_j8_build - # Java 8 unit tests - - start_j8_unit_tests: - type: approval - - j8_unit_tests: - requires: - - start_j8_unit_tests - - j8_build - - start_j8_jvm_dtests: - type: approval - - j8_jvm_dtests: - requires: - - start_j8_jvm_dtests - - j8_build - - start_j8_cqlshlib_tests: - type: approval - - j8_cqlshlib_tests: - requires: - - start_j8_cqlshlib_tests - - j8_build - # Java 11 unit tests - - start_j11_unit_tests: - type: approval - - j11_unit_tests: - requires: - - start_j11_unit_tests - - j8_build - # specialized unit tests (all run on request using Java 8) - - start_utests_long: - type: approval - - utests_long: - requires: - - start_utests_long - - j8_build - - start_utests_compression: - type: approval - - utests_compression: - requires: - - start_utests_compression - - j8_build - - start_utests_stress: - type: approval - - utests_stress: - requires: - - start_utests_stress - - j8_build - - start_utests_fqltool: - type: approval - - utests_fqltool: - requires: - - start_utests_fqltool - - j8_build - - start_utests_system_keyspace_directory: - type: approval - - utests_system_keyspace_directory: - requires: - - start_utests_system_keyspace_directory - - j8_build - - start_j8_dtest_jars_build: - type: approval - - j8_dtest_jars_build: - requires: - - j8_build - - start_j8_dtest_jars_build - - start_jvm_upgrade_dtest: - type: approval - - j8_jvm_upgrade_dtests: - requires: - - start_jvm_upgrade_dtest - - j8_dtest_jars_build - # Java 8 dtests - - start_j8_dtests: - type: approval - - j8_dtests-with-vnodes: - requires: - - start_j8_dtests - - j8_build - - j8_dtests-no-vnodes: - requires: - - start_j8_dtests - - j8_build - # Java 11 dtests - - start_j11_dtests: - type: approval - - j11_dtests-with-vnodes: - requires: - - start_j11_dtests - - j8_build - - j11_dtests-no-vnodes: - requires: - - start_j11_dtests - - j8_build - # Java 8 upgrade tests - - start_upgrade_tests: - type: approval - - j8_upgradetests-no-vnodes: - requires: - - start_upgrade_tests - - j8_build - # Java 8 cqlsh dtests - - start_j8_cqlsh_tests: - type: approval - - j8_cqlsh-dtests-py2-with-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py3-with-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py38-with-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py2-no-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py3-no-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py38-no-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - # Java 11 cqlsh dtests - - start_j11_cqlsh_tests: - type: approval - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - # Java 8 repeated utest - - start_j8_repeated_utest: - type: approval - - j8_repeated_utest: - requires: - - start_j8_repeated_utest - - j8_build - # Java 11 repeated utest - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j8_build - # Java 8 repeated dtest - - start_j8_repeated_dtest: - type: approval - - j8_repeated_dtest: - requires: - - start_j8_repeated_dtest - - j8_build - # Java 11 repeated dtest - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j8_build - # Repeated Python upgrade dtest - - start_repeated_upgrade_dtest: - type: approval - - repeated_upgrade_dtest: - requires: - - start_repeated_upgrade_dtest - - j8_build - # Repeated JVM upgrade dtest - - start_repeated_jvm_upgrade_dtest: - type: approval - - repeated_jvm_upgrade_dtest: - requires: - - start_repeated_jvm_upgrade_dtest - - j8_dtest_jars_build - -j8_pre-commit_jobs: &j8_pre-commit_jobs - jobs: - - start_pre-commit_tests: - type: approval - - j8_build: - requires: - - start_pre-commit_tests - # Java 8 unit tests - - j8_unit_tests: - requires: - - j8_build - - j8_jvm_dtests: - requires: - - j8_build - - j8_cqlshlib_tests: - requires: - - j8_build - # Java 11 unit tests - - j11_unit_tests: - requires: - - j8_build - # specialized unit tests (all run on request using Java 8) - - start_utests_long: - type: approval - - utests_long: - requires: - - start_utests_long - - j8_build - - start_utests_compression: - type: approval - - utests_compression: - requires: - - start_utests_compression - - j8_build - - start_utests_stress: - type: approval - - utests_stress: - requires: - - start_utests_stress - - j8_build - - start_utests_fqltool: - type: approval - - utests_fqltool: - requires: - - start_utests_fqltool - - j8_build - - utests_system_keyspace_directory: - requires: - - j8_build - - start_j8_dtest_jars_build: - type: approval - - j8_dtest_jars_build: - requires: - - j8_build - - start_j8_dtest_jars_build - - start_jvm_upgrade_dtest: - type: approval - - j8_jvm_upgrade_dtests: - requires: - - start_jvm_upgrade_dtest - - j8_dtest_jars_build - # Java 8 dtests - - j8_dtests-with-vnodes: - requires: - - j8_build - - j8_dtests-no-vnodes: - requires: - - j8_build - # Java 11 dtests - - j11_dtests-with-vnodes: - requires: - - j8_build - - j11_dtests-no-vnodes: - requires: - - j8_build - # Java 8 upgrade tests (on request) - - start_upgrade_tests: - type: approval - - j8_upgradetests-no-vnodes: - requires: - - j8_build - - start_upgrade_tests - # Java 8 cqlsh dtests - - j8_cqlsh-dtests-py2-with-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py3-with-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py38-with-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py2-no-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py3-no-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py38-no-vnodes: - requires: - - j8_build - # Java 11 cqlsh dtests - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - j8_build - # Java 8 repeated utest (on request) - - start_j8_repeated_utest: - type: approval - - j8_repeated_utest: - requires: - - start_j8_repeated_utest - - j8_build - # Java 11 repeated utest (on request) - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j8_build - # Java 8 repeated dtest (on request) - - start_j8_repeated_dtest: - type: approval - - j8_repeated_dtest: - requires: - - start_j8_repeated_dtest - - j8_build - # Java 11 repeated dtest (on request) - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j8_build - # Repeated Python upgrade dtest (on request) - - start_repeated_upgrade_dtest: - type: approval - - repeated_upgrade_dtest: - requires: - - start_repeated_upgrade_dtest - - j8_build - # Repeated JVM upgrade dtest (on request) - - start_repeated_jvm_upgrade_dtest: - type: approval - - repeated_jvm_upgrade_dtest: - requires: - - start_repeated_jvm_upgrade_dtest - - j8_dtest_jars_build - -j11_with_dtests_jobs: &j11_with_dtests_jobs - jobs: - - start_j11_build: - type: approval - - j11_build: - requires: - - start_j11_build - # Java 11 unit tests - - start_j11_unit_tests: - type: approval - - j11_unit_tests: - requires: - - start_j11_unit_tests - - j11_build - - start_j11_jvm_dtests: - type: approval - - j11_jvm_dtests: - requires: - - start_j11_jvm_dtests - - j11_build - - start_j11_cqlshlib_tests: - type: approval - - j11_cqlshlib_tests: - requires: - - start_j11_cqlshlib_tests - - j11_build - # Java 11 dtests - - start_j11_dtests: - type: approval - - j11_dtests-with-vnodes: - requires: - - start_j11_dtests - - j11_build - - j11_dtests-no-vnodes: - requires: - - start_j11_dtests - - j11_build - - start_j11_cqlsh_tests: - type: approval - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - # Java 11 repeated utest - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j11_build - # Java 11 repeated dtest - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j11_build - -j11_pre-commit_jobs: &j11_pre-commit_jobs - jobs: - - start_pre-commit_tests: - type: approval - - j11_build: - requires: - - start_pre-commit_tests - - j11_unit_tests: - requires: - - j11_build - - j11_jvm_dtests: - requires: - - j11_build - - j11_cqlshlib_tests: - requires: - - j11_build - - j11_jvm_dtests: - requires: - - j11_build - - j11_cqlshlib_tests: - requires: - - j11_build - - j11_dtests-with-vnodes: - requires: - - j11_build - - j11_dtests-no-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - j11_build - # Java 11 repeated utest (on request) - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j11_build - # Java 11 repeated dtest (on request) - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j11_build - -workflows: - version: 2 - java8_separate_tests: *j8_with_dtests_jobs - java8_pre-commit_tests: *j8_pre-commit_jobs - java11_separate_tests: *j11_with_dtests_jobs - java11_pre-commit_tests: *j11_pre-commit_jobs - -executors: - java8-executor: - parameters: - exec_resource_class: - type: string - default: medium - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: << parameters.exec_resource_class >> - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - environment: - <<: *default_env_vars - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - java11-executor: - parameters: - exec_resource_class: - type: string - default: medium - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: << parameters.exec_resource_class >> - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - environment: - <<: *default_env_vars - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - CASSANDRA_USE_JDK11: true - -build_common: &build_common - parallelism: 1 # This job doesn't benefit from parallelism - steps: - - log_environment - - clone_cassandra - - build_cassandra - - run_eclipse_warnings - - persist_to_workspace: - root: /home/cassandra - paths: - - cassandra - - .m2 - -jobs: - j8_build: - executor: java8-executor - <<: *build_common - - j11_build: - executor: java11-executor - <<: *build_common - - j8_dtest_jars_build: - executor: java8-executor - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - build_cassandra_dtest_jars - - persist_to_workspace: - root: /home/cassandra - paths: - - dtest_jars - - j8_unit_tests: - <<: *j8_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - create_junit_containers - - log_environment - - run_parallel_junit_tests - - j8_jvm_dtests: - <<: *j8_small_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - create_junit_containers: - classlistprefix: distributed - extra_filters: "| grep -v upgrade" - - log_environment - - run_parallel_junit_tests: - classlistprefix: distributed - target: "testclasslist" - - j11_jvm_dtests: - <<: *j11_small_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - create_junit_containers: - classlistprefix: distributed - extra_filters: "| grep -v upgrade" - - log_environment - - run_parallel_junit_tests: - classlistprefix: distributed - target: "testclasslist" - - j8_jvm_upgrade_dtests: - <<: *j8_medium_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - create_junit_containers: - classlistprefix: distributed - extra_filters: "| grep upgrade" - - log_environment - - run_parallel_junit_tests: - classlistprefix: distributed - target: "testclasslist" - - j11_unit_tests: - <<: *j11_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - create_junit_containers - - log_environment - - run_parallel_junit_tests - - j8_cqlshlib_tests: - <<: *j8_small_executor - steps: - - attach_workspace: - at: /home/cassandra - - run_cqlshlib_tests - - j11_cqlshlib_tests: - <<: *j11_small_executor - steps: - - attach_workspace: - at: /home/cassandra - - run_cqlshlib_tests - - utests_long: - <<: *j8_seq_executor - steps: - - attach_workspace: - at: /home/cassandra - - run_junit_tests: - target: long-test - - utests_compression: - <<: *j8_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - create_junit_containers - - log_environment - - run_parallel_junit_tests: - target: testclasslist-compression - - utests_stress: - <<: *j8_seq_executor - steps: - - attach_workspace: - at: /home/cassandra - - run_junit_tests: - target: stress-test - - utests_fqltool: - <<: *j8_seq_executor - steps: - - attach_workspace: - at: /home/cassandra - - run_junit_tests: - target: fqltool-test - - utests_system_keyspace_directory: - <<: *j8_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - create_junit_containers - - log_environment - - run_parallel_junit_tests: - target: testclasslist-system-keyspace-directory - - j8_dtests-with-vnodes: - <<: *j8_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j8_with_vnodes - run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql'" - - run_dtests: - file_tag: j8_with_vnodes - pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' - - j11_dtests-with-vnodes: - <<: *j11_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - log_environment - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j11_with_vnodes - run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql'" - - run_dtests: - file_tag: j11_with_vnodes - pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' - - j8_dtests-no-vnodes: - <<: *j8_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j8_without_vnodes - run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k not cql'" - - run_dtests: - file_tag: j8_without_vnodes - pytest_extra_args: '--skip-resource-intensive-tests' - - j11_dtests-no-vnodes: - <<: *j11_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - log_environment - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j11_without_vnodes - run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k not cql'" - - run_dtests: - file_tag: j11_without_vnodes - pytest_extra_args: '--skip-resource-intensive-tests' - - j8_upgradetests-no-vnodes: - <<: *j8_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j8_upgradetests_without_vnodes - run_dtests_extra_args: '--execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all' - - run_dtests: - file_tag: j8_upgradetests_without_vnodes - pytest_extra_args: '--execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all' - - j8_cqlsh-dtests-py2-with-vnodes: - <<: *j8_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j8_with_vnodes - run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" - - run_dtests: - file_tag: j8_with_vnodes - pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' - extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' - - j8_cqlsh-dtests-py3-with-vnodes: - <<: *j8_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j8_with_vnodes - run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" - - run_dtests: - file_tag: j8_with_vnodes - pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' - extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' - - j8_cqlsh-dtests-py38-with-vnodes: - <<: *j8_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv: - python_version: '3.8' - - create_dtest_containers: - file_tag: j8_with_vnodes - run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" - python_version: '3.8' - - run_dtests: - file_tag: j8_with_vnodes - pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' - extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8' - python_version: '3.8' - - j8_cqlsh-dtests-py2-no-vnodes: - <<: *j8_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j8_without_vnodes - run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" - - run_dtests: - file_tag: j8_without_vnodes - pytest_extra_args: '--skip-resource-intensive-tests' - extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' - - j8_cqlsh-dtests-py3-no-vnodes: - <<: *j8_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j8_without_vnodes - run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" - - run_dtests: - file_tag: j8_without_vnodes - pytest_extra_args: '--skip-resource-intensive-tests' - extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' - - j8_cqlsh-dtests-py38-no-vnodes: - <<: *j8_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv: - python_version: '3.8' - - create_dtest_containers: - file_tag: j8_without_vnodes - run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" - python_version: '3.8' - - run_dtests: - file_tag: j8_without_vnodes - pytest_extra_args: '--skip-resource-intensive-tests' - extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8' - python_version: '3.8' - - j11_cqlsh-dtests-py2-with-vnodes: - <<: *j11_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j11_with_vnodes - run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" - - run_dtests: - file_tag: j11_with_vnodes - pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' - extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' - - j11_cqlsh-dtests-py3-with-vnodes: - <<: *j11_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j11_with_vnodes - run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" - - run_dtests: - file_tag: j11_with_vnodes - pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' - extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' - - j11_cqlsh-dtests-py38-with-vnodes: - <<: *j11_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv: - python_version: '3.8' - - create_dtest_containers: - file_tag: j11_with_vnodes - run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" - python_version: '3.8' - - run_dtests: - file_tag: j11_with_vnodes - pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' - extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8' - python_version: '3.8' - - j11_cqlsh-dtests-py2-no-vnodes: - <<: *j11_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j11_without_vnodes - run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" - - run_dtests: - file_tag: j11_without_vnodes - pytest_extra_args: '--skip-resource-intensive-tests' - extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' - - j11_cqlsh-dtests-py3-no-vnodes: - <<: *j11_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - create_dtest_containers: - file_tag: j11_without_vnodes - run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" - - run_dtests: - file_tag: j11_without_vnodes - pytest_extra_args: '--skip-resource-intensive-tests' - extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' - - j11_cqlsh-dtests-py38-no-vnodes: - <<: *j11_par_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv: - python_version: '3.8' - - create_dtest_containers: - file_tag: j11_without_vnodes - run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" - python_version: '3.8' - - run_dtests: - file_tag: j11_without_vnodes - pytest_extra_args: '--skip-resource-intensive-tests' - extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8' - python_version: '3.8' - - j8_repeated_utest: - <<: *j8_repeated_utest_executor - steps: - - attach_workspace: - at: /home/cassandra - - log_environment - - run_repeated_utest: - target: ${REPEATED_UTEST_TARGET} - class: ${REPEATED_UTEST_CLASS} - methods: ${REPEATED_UTEST_METHODS} - count: ${REPEATED_UTEST_COUNT} - stop_on_failure: ${REPEATED_UTEST_STOP_ON_FAILURE} - - j11_repeated_utest: - <<: *j11_repeated_utest_executor - steps: - - attach_workspace: - at: /home/cassandra - - log_environment - - run_repeated_utest: - target: ${REPEATED_UTEST_TARGET} - class: ${REPEATED_UTEST_CLASS} - methods: ${REPEATED_UTEST_METHODS} - count: ${REPEATED_UTEST_COUNT} - stop_on_failure: ${REPEATED_UTEST_STOP_ON_FAILURE} - - j8_repeated_dtest: - <<: *j8_repeated_dtest_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - run_repeated_dtest: - tests: ${REPEATED_DTEST_NAME} - vnodes: ${REPEATED_DTEST_VNODES} - upgrade: "false" - count: ${REPEATED_DTEST_COUNT} - stop_on_failure: ${REPEATED_DTEST_STOP_ON_FAILURE} - - j11_repeated_dtest: - <<: *j11_repeated_dtest_executor - steps: - - attach_workspace: - at: /home/cassandra - - log_environment - - clone_dtest - - create_venv - - run_repeated_dtest: - tests: ${REPEATED_DTEST_NAME} - vnodes: ${REPEATED_DTEST_VNODES} - upgrade: "false" - count: ${REPEATED_DTEST_COUNT} - stop_on_failure: ${REPEATED_DTEST_STOP_ON_FAILURE} - - repeated_jvm_upgrade_dtest: - <<: *j8_repeated_jvm_upgrade_dtest_executor - steps: - - attach_workspace: - at: /home/cassandra - - log_environment - - run_repeated_utest: - target: test-jvm-dtest-some - class: ${REPEATED_JVM_UPGRADE_DTEST_CLASS} - methods: ${REPEATED_JVM_UPGRADE_DTEST_METHODS} - count: ${REPEATED_JVM_UPGRADE_DTEST_COUNT} - stop_on_failure: ${REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE} - - repeated_upgrade_dtest: - <<: *j8_repeated_upgrade_dtest_executor - steps: - - attach_workspace: - at: /home/cassandra - - clone_dtest - - create_venv - - run_repeated_dtest: - tests: ${REPEATED_UPGRADE_DTEST_NAME} - vnodes: "false" - upgrade: "true" - stop_on_failure: ${REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE} - count: ${REPEATED_UPGRADE_DTEST_COUNT} - -commands: - log_environment: - steps: - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - clone_cassandra: - steps: - - run: - name: Clone Cassandra Repository (via git) - command: | - git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra - - clone_dtest: - steps: - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - build_cassandra: - steps: - - run: - name: Build Cassandra - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant clean realclean jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - no_output_timeout: 15m - - build_cassandra_dtest_jars: - steps: - - run: - name: Build Cassandra DTest jars - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - mkdir ~/dtest_jars - git remote add apache https://github.com/apache/cassandra.git - for branch in cassandra-2.2 cassandra-3.0 cassandra-3.11 cassandra-4.0 trunk; do - # check out the correct cassandra version: - git remote set-branches --add apache '$branch' - git fetch --depth 1 apache $branch - git checkout $branch - git clean -fd - # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - cp build/dtest*.jar ~/dtest_jars - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - done - # and build the dtest-jar for the branch under test - ${ANT_HOME}/bin/ant realclean - git checkout origin/$CIRCLE_BRANCH - git clean -fd - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - cp build/dtest*.jar ~/dtest_jars - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - ls -l ~/dtest_jars - no_output_timeout: 15m - - run_eclipse_warnings: - steps: - - run: - name: Run eclipse-warnings - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - ant eclipse-warnings - - create_junit_containers: - parameters: - classlistprefix: - type: string - default: unit - extra_filters: - type: string - default: "" - steps: - - run: - name: Determine <> Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/<>/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/<>/;;g" | grep "Test\.java$" <> > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - - no_output_timeout: 15m - - run_junit_tests: - parameters: - target: - type: string - no_output_timeout: - type: string - default: 15m - steps: - - run: - name: Run Unit Tests (<>) - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - ant <> - no_output_timeout: <> - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - - run_cqlshlib_tests: - parameters: - no_output_timeout: - type: string - default: 15m - steps: - - run: - name: Run cqlshlib Unit Tests - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra/pylib - ./cassandra-cqlsh-tests.sh .. - no_output_timeout: <> - - store_test_results: - path: /tmp/cassandra/pylib - - run_parallel_junit_tests: - parameters: - target: - type: string - default: testclasslist - no_output_timeout: - type: string - default: 15m - classlistprefix: - type: string - default: unit - steps: - - run: - name: Run Unit Tests (<>) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.<>.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant <> -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=<> - no_output_timeout: <> - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - - create_venv: - parameters: - python_version: - type: enum - default: "3.6" - enum: ["3.6", "3.7", "3.8"] - steps: - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env<>/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - create_dtest_containers: - parameters: - file_tag: - type: string - run_dtests_extra_args: - type: string - default: '' - extra_env_args: - type: string - default: '' - tests_filter_pattern: - type: string - default: '' - python_version: - type: enum - default: "3.6" - enum: ["3.6", "3.7", "3.8"] - steps: - - run: - name: Determine Tests to Run (<>) - no_output_timeout: 5m - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - cd cassandra-dtest - source ~/env<>/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - - if [ -n '<>' ]; then - export <> - fi - - echo "***Collected DTests (<>)***" - set -eo pipefail && ./run_dtests.py <> --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_<>_raw --cassandra-dir=../cassandra - if [ -z '<>' ]; then - mv /tmp/all_dtest_tests_<>_raw /tmp/all_dtest_tests_<> - else - grep -e '<>' /tmp/all_dtest_tests_<>_raw > /tmp/all_dtest_tests_<> || { echo "Filter did not match any tests! Exiting build."; exit 0; } - fi - set -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_<> > /tmp/split_dtest_tests_<>.txt - cat /tmp/split_dtest_tests_<>.txt | tr '\n' ' ' > /tmp/split_dtest_tests_<>_final.txt - cat /tmp/split_dtest_tests_<>_final.txt - - run_dtests: - parameters: - file_tag: - type: string - pytest_extra_args: - type: string - default: '' - extra_env_args: - type: string - default: '' - python_version: - type: enum - default: "3.6" - enum: ["3.6", "3.7", "3.8"] - steps: - - run: - name: Run dtests (<>) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_<>_final.txt" - cat /tmp/split_dtest_tests_<>_final.txt - - source ~/env<>/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n '<>' ]; then - export <> - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_<>_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest <> --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_<>.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_<> - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_<>_logs - - run_repeated_utest: - parameters: - target: - type: string - class: - type: string - methods: - type: string - count: - type: string - stop_on_failure: - type: string - steps: - - run: - name: Run repeated JUnit test - no_output_timeout: 15m - command: | - if [ "<>" == "" ]; then - echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "<>" == "" ]; then - echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "<>" -le 0 ]; then - echo "Repeated utest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((<> / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (<> % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running <> <> <> <> times" - - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - - target=<> - class_path=<> - class_name="${class_path##*.}" - - # Prepare the -Dtest.name argument. - # It can be the fully qualified class name or the short class name, depending on the target. - if [[ $target == "test" || \ - $target == "test-cdc" || \ - $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then - name="-Dtest.name=$class_name" - else - name="-Dtest.name=$class_path" - fi - - # Prepare the -Dtest.methods argument, which is optional - if [ "<>" == "" ]; then - methods="" - else - methods="-Dtest.methods=<>" - fi - - # Run the test target as many times as requested collecting the exit code, - # stopping the iteration only if stop_on_failure is set. - exit_code="$?" - for i in $(seq -w 1 $count); do - - echo "Running test iteration $i of $count" - - # run the test - status="passes" - if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then - status="fails" - exit_code=1 - fi - - # move the stdout output file - dest=/tmp/results/repeated_utest/stdout/${status}/${i} - mkdir -p $dest - mv stdout.txt $dest/<>-<>.txt - - # move the XML output files - source=build/test/output - dest=/tmp/results/repeated_utest/output/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # move the log files - source=build/test/logs - dest=/tmp/results/repeated_utest/logs/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # maybe stop iterations on test failure - if [[ <> = true ]] && (( $exit_code > 0 )); then - break - fi - done - - (exit ${exit_code}) - fi - fi - - store_test_results: - path: /tmp/results/repeated_utest/output - - store_artifacts: - path: /tmp/results/repeated_utest/stdout - destination: stdout - - store_artifacts: - path: /tmp/results/repeated_utest/output - destination: junitxml - - store_artifacts: - path: /tmp/results/repeated_utest/logs - destination: logs - - run_repeated_dtest: - parameters: - tests: - type: string - vnodes: - type: string - upgrade: - type: string - count: - type: string - stop_on_failure: - type: string - steps: - - run: - name: Run repeated Python dtest - no_output_timeout: 15m - command: | - if [ "<>" == "" ]; then - echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "<>" == "" ]; then - echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "<>" -le 0 ]; then - echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((<> / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (<> % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running <> $count times" - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - - stop_on_failure_arg="" - if <>; then - stop_on_failure_arg="-x" - fi - - vnodes_args="" - if <>; then - vnodes_args="--use-vnodes --num-tokens=16" - fi - - upgrade_arg="" - if <>; then - upgrade_arg="--execute-upgrade-tests" - fi - - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir <> | tee /tmp/dtest/stdout.txt - fi - fi - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_logs diff --git a/.circleci/config-2_1.yml.high_res.patch b/.circleci/config-2_1.yml.high_res.patch deleted file mode 100644 index 3e92def52eec..000000000000 --- a/.circleci/config-2_1.yml.high_res.patch +++ /dev/null @@ -1,106 +0,0 @@ ---- config-2_1.yml 2022-05-30 12:01:09.000000000 -0400 -+++ config-2_1.yml.HIGHRES 2022-05-30 12:03:00.000000000 -0400 -@@ -101,14 +101,14 @@ - j8_par_executor: &j8_par_executor - executor: - name: java8-executor -- #exec_resource_class: xlarge -- parallelism: 4 -+ exec_resource_class: xlarge -+ parallelism: 100 - - j8_small_par_executor: &j8_small_par_executor - executor: - name: java8-executor -- #exec_resource_class: xlarge -- parallelism: 1 -+ exec_resource_class: xlarge -+ parallelism: 5 - - j8_small_executor: &j8_small_executor - executor: -@@ -119,62 +119,68 @@ - j8_medium_par_executor: &j8_medium_par_executor - executor: - name: java8-executor -- #exec_resource_class: xlarge -- parallelism: 1 -+ exec_resource_class: xlarge -+ parallelism: 2 - - j8_seq_executor: &j8_seq_executor - executor: - name: java8-executor -- #exec_resource_class: xlarge -+ exec_resource_class: xlarge - parallelism: 1 # sequential, single container tests: no parallelism benefits - - j11_par_executor: &j11_par_executor - executor: - name: java11-executor -- #exec_resource_class: xlarge -- parallelism: 4 -+ exec_resource_class: xlarge -+ parallelism: 100 - --j11_small_par_executor: &j11_small_par_executor -+j11_small_executor: &j11_small_executor - executor: - name: java11-executor -- #exec_resource_class: xlarge -+ exec_resource_class: medium - parallelism: 1 - --j11_small_executor: &j11_small_executor -+j11_small_par_executor: &j11_small_par_executor - executor: - name: java11-executor -- #exec_resource_class: medium -- parallelism: 1 -+ exec_resource_class: xlarge -+ parallelism: 5 - - j8_repeated_utest_executor: &j8_repeated_utest_executor - executor: - name: java8-executor -- parallelism: 4 -+ exec_resource_class: xlarge -+ parallelism: 100 - - j8_repeated_dtest_executor: &j8_repeated_dtest_executor - executor: - name: java8-executor -- parallelism: 4 -+ exec_resource_class: xlarge -+ parallelism: 100 - - j8_repeated_upgrade_dtest_executor: &j8_repeated_upgrade_dtest_executor - executor: - name: java8-executor -- parallelism: 4 -+ exec_resource_class: xlarge -+ parallelism: 100 - - j8_repeated_jvm_upgrade_dtest_executor: &j8_repeated_jvm_upgrade_dtest_executor - executor: - name: java8-executor -- parallelism: 4 -+ exec_resource_class: xlarge -+ parallelism: 100 - - j11_repeated_utest_executor: &j11_repeated_utest_executor - executor: - name: java11-executor -- parallelism: 4 -+ exec_resource_class: xlarge -+ parallelism: 100 - - j11_repeated_dtest_executor: &j11_repeated_dtest_executor - executor: - name: java11-executor -- parallelism: 4 -+ exec_resource_class: xlarge -+ parallelism: 100 - - j8_with_dtests_jobs: &j8_with_dtests_jobs - jobs: diff --git a/.circleci/config.yml b/.circleci/config.yml index 6e941ea5ea52..1bae80b66cc7 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -83,7 +83,7 @@ jobs: if [ -z "$test_timeout" ]; then test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed -Dno-build-test=true no_output_timeout: 15m - store_test_results: path: /tmp/cassandra/build/test/output/ @@ -109,27 +109,34 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - repeated_jvm_upgrade_dtest: + j11_utests_fqltool_repeat: docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l @@ -160,113 +167,123 @@ jobs: which java java -version - run: - name: Run repeated JUnit test + name: Repeatedly run new or modifed JUnit tests no_output_timeout: 15m command: | - if [ "${REPEATED_JVM_UPGRADE_DTEST_CLASS}" == "" ]; then - echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" == "" ]; then - echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" -le 0 ]; then - echo "Repeated utest count is lesser or equals than zero, exiting without running any test" - else + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_JVM_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_FQLTOOL_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_FQLTOOL_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running test-jvm-dtest-some ${REPEATED_JVM_UPGRADE_DTEST_CLASS} ${REPEATED_JVM_UPGRADE_DTEST_METHODS} ${REPEATED_JVM_UPGRADE_DTEST_COUNT} times" + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_FQLTOOL} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=fqltool-test + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi - target=test-jvm-dtest-some - class_path=${REPEATED_JVM_UPGRADE_DTEST_CLASS} - class_name="${class_path##*.}" + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi # Prepare the -Dtest.name argument. # It can be the fully qualified class name or the short class name, depending on the target. if [[ $target == "test" || \ $target == "test-cdc" || \ $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then - name="-Dtest.name=$class_name" + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" else - name="-Dtest.name=$class_path" + name_arg="-Dtest.name=$class" fi # Prepare the -Dtest.methods argument, which is optional - if [ "${REPEATED_JVM_UPGRADE_DTEST_METHODS}" == "" ]; then - methods="" + if [[ $method == "" ]]; then + methods_arg="" else - methods="-Dtest.methods=${REPEATED_JVM_UPGRADE_DTEST_METHODS}" + methods_arg="-Dtest.methods=$method" fi - # Run the test target as many times as requested collecting the exit code, - # stopping the iteration only if stop_on_failure is set. - exit_code="$?" for i in $(seq -w 1 $count); do - - echo "Running test iteration $i of $count" + echo "Running test $test, iteration $i of $count" # run the test status="passes" - if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then + if !( set -o pipefail && \ + ant fqltool-test $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then status="fails" exit_code=1 fi # move the stdout output file - dest=/tmp/results/repeated_utest/stdout/${status}/${i} + dest=/tmp/results/repeated_utests/stdout/${status}/${i} mkdir -p $dest - mv stdout.txt $dest/test-jvm-dtest-some-${REPEATED_JVM_UPGRADE_DTEST_CLASS}.txt + mv stdout.txt $dest/${test}.txt # move the XML output files - source=build/test/output - dest=/tmp/results/repeated_utest/output/${status}/${i} + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} mkdir -p $dest if [[ -d $source && -n "$(ls $source)" ]]; then mv $source/* $dest/ fi # move the log files - source=build/test/logs - dest=/tmp/results/repeated_utest/logs/${status}/${i} + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} mkdir -p $dest if [[ -d $source && -n "$(ls $source)" ]]; then mv $source/* $dest/ fi # maybe stop iterations on test failure - if [[ ${REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then break fi done - - (exit ${exit_code}) - fi - fi + done + (exit ${exit_code}) - store_test_results: - path: /tmp/results/repeated_utest/output + path: /tmp/results/repeated_utests/output - store_artifacts: - path: /tmp/results/repeated_utest/stdout + path: /tmp/results/repeated_utests/stdout destination: stdout - store_artifacts: - path: /tmp/results/repeated_utest/output + path: /tmp/results/repeated_utests/output destination: junitxml - store_artifacts: - path: /tmp/results/repeated_utest/logs + path: /tmp/results/repeated_utests/logs destination: logs environment: - ANT_HOME: /usr/share/ant @@ -284,24 +301,32 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true j8_cqlsh-dtests-py2-with-vnodes: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest @@ -354,7 +379,12 @@ jobs: mkdir -p /tmp/results/dtests # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi - store_test_results: path: /tmp/results - store_artifacts: @@ -379,27 +409,34 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_unit_tests: + j8_cqlsh_dtests_py311_vnode: docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l @@ -408,70 +445,61 @@ jobs: - attach_workspace: at: /home/cassandra - run: - name: Determine unit Tests to Run + name: Clone Cassandra dtest Repository (via git) command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - run: - name: Log Environment Information + name: Configure virtualenv and python Dependencies command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze - run: - name: Run Unit Tests (testclasslist) + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) + no_output_timeout: 15m command: | - set -x + echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt + + source ~/env3.11/bin/activate export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - store_test_results: - path: /tmp/cassandra/build/test/output/ + path: /tmp/results - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml + path: /tmp/dtest + destination: dtest_j8_with_vnodes - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -488,28 +516,34 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - repeated_upgrade_dtest: + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_vnode_repeat: docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l @@ -517,6 +551,28 @@ jobs: steps: - attach_workspace: at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version - run: name: Clone Cassandra dtest Repository (via git) command: | @@ -537,25 +593,25 @@ jobs: name: Run repeated Python dtest no_output_timeout: 15m command: | - if [ "${REPEATED_UPGRADE_DTEST_NAME}" == "" ]; then + if [ "${REPEATED_DTESTS}" == "" ]; then echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" == "" ]; then + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" -le 0 ]; then + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" else # Calculate the number of test iterations to be run by the current parallel runner. # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then count=$((count+1)) fi if (($count <= 0)); then echo "No tests to run in this runner" else - echo "Running ${REPEATED_UPGRADE_DTEST_NAME} $count times" + echo "Running ${REPEATED_DTESTS} $count times" source ~/env3.6/bin/activate export PATH=$JAVA_HOME/bin:$PATH @@ -568,23 +624,25 @@ jobs: echo "** done env" mkdir -p /tmp/results/dtests + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + stop_on_failure_arg="" - if ${REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE}; then + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then stop_on_failure_arg="-x" fi vnodes_args="" - if false; then + if true; then vnodes_args="--use-vnodes --num-tokens=16" fi upgrade_arg="" - if true; then - upgrade_arg="--execute-upgrade-tests" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" fi # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_UPGRADE_DTEST_NAME} | tee /tmp/dtest/stdout.txt + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt fi fi - store_test_results: @@ -611,25 +669,33 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py38-no-vnodes: + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtests_large_vnode: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium @@ -650,46 +716,27 @@ jobs: # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate + source ~/env3.6/bin/activate export PATH=$JAVA_HOME/bin:$PATH pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt pip3 uninstall -y cqlsh pip3 freeze - run: - name: Determine Tests to Run (j8_without_vnodes) + name: Determine Tests to Run (j8_large_with_vnodes) no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_large_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --only-resource-intensive-tests --force-resource-intensive-tests --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_large_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_large_with_vnodes_raw /tmp/all_dtest_tests_j8_large_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_large_with_vnodes_raw > /tmp/all_dtest_tests_j8_large_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_large_with_vnodes > /tmp/split_dtest_tests_j8_large_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_large_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt\n" - run: - name: Run dtests (j8_without_vnodes) + name: Run dtests (j8_large_with_vnodes) no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + command: "echo \"cat /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_large_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" - store_test_results: path: /tmp/results - store_artifacts: path: /tmp/dtest - destination: dtest_j8_without_vnodes + destination: dtest_j8_large_with_vnodes - store_artifacts: path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs + destination: dtest_j8_large_with_vnodes_logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -706,25 +753,32 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_cqlsh-dtests-py3-with-vnodes: + j11_utests_system_keyspace_directory: docker: - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium @@ -735,56 +789,70 @@ jobs: - attach_workspace: at: /home/cassandra - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m + name: Determine unit Tests to Run command: | - echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-system-keyspace-directory) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-system-keyspace-directory -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m - store_test_results: - path: /tmp/results + path: /tmp/cassandra/build/test/output/ - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes + path: /tmp/cassandra/build/test/output + destination: junitxml - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs + path: /tmp/cassandra/build/test/logs + destination: logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -801,86 +869,253 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - CASSANDRA_USE_JDK11: true - j11_cqlsh-dtests-py3-no-vnodes: + j8_utests_stress: docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l - parallelism: 4 + parallelism: 1 steps: - attach_workspace: at: /home/cassandra - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies + name: Run Unit Tests (stress-test) command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant stress-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_stress_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version - run: - name: Run dtests (j11_without_vnodes) + name: Repeatedly run new or modifed JUnit tests no_output_timeout: 15m command: | - echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - - source ~/env3.6/bin/activate + set -x export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ fi - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_STRESS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_STRESS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_STRESS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=stress-test-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant stress-test-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) - store_test_results: - path: /tmp/results + path: /tmp/results/repeated_utests/output - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes + path: /tmp/results/repeated_utests/stdout + destination: stdout - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -897,26 +1132,33 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - CASSANDRA_USE_JDK11: true - j11_cqlsh-dtests-py38-with-vnodes: + j11_cqlsh_dtests_py311: docker: - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium @@ -937,26 +1179,26 @@ jobs: # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate + source ~/env3.11/bin/activate export PATH=$JAVA_HOME/bin:$PATH pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt pip3 uninstall -y cqlsh pip3 freeze - run: - name: Determine Tests to Run (j11_with_vnodes) + name: Determine Tests to Run (j11_without_vnodes) no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - run: - name: Run dtests (j11_with_vnodes) + name: Run dtests (j11_without_vnodes) no_output_timeout: 15m command: | - echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - source ~/env3.8/bin/activate + source ~/env3.11/bin/activate export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 fi java -version @@ -967,16 +1209,21 @@ jobs: echo "** done env" mkdir -p /tmp/results/dtests # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi - store_test_results: path: /tmp/results - store_artifacts: path: /tmp/dtest - destination: dtest_j11_with_vnodes + destination: dtest_j11_without_vnodes - store_artifacts: path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs + destination: dtest_j11_without_vnodes_logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -993,26 +1240,33 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - CASSANDRA_USE_JDK11: true - j8_cqlsh-dtests-py3-with-vnodes: + j8_utests_compression_repeat: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium @@ -1023,56 +1277,146 @@ jobs: - attach_workspace: at: /home/cassandra - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies + name: Log Environment Information command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version - run: - name: Run dtests (j8_with_vnodes) + name: Repeatedly run new or modifed JUnit tests no_output_timeout: 15m command: | - echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt - - source ~/env3.6/bin/activate + set -x export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ fi - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-compression + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-compression $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) - store_test_results: - path: /tmp/results + path: /tmp/results/repeated_utests/output - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes + path: /tmp/results/repeated_utests/stdout + destination: stdout - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -1089,25 +1433,149 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py2-no-vnodes: + j11_unit_tests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py3: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium @@ -1146,8 +1614,8 @@ jobs: source ~/env3.6/bin/activate export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 fi java -version @@ -1159,7 +1627,12 @@ jobs: mkdir -p /tmp/results/dtests # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi - store_test_results: path: /tmp/results - store_artifacts: @@ -1184,25 +1657,32 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_repeated_dtest: + j11_cqlsh_dtests_py38: docker: - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium @@ -1212,28 +1692,6 @@ jobs: steps: - attach_workspace: at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - run: name: Clone Cassandra dtest Repository (via git) command: | @@ -1245,73 +1703,244 @@ jobs: # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate + source ~/env3.8/bin/activate export PATH=$JAVA_HOME/bin:$PATH pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt pip3 uninstall -y cqlsh pip3 freeze - run: - name: Run repeated Python dtest + name: Determine Tests to Run (j11_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_without_vnodes) no_output_timeout: 15m command: | - if [ "${REPEATED_DTEST_NAME}" == "" ]; then - echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" == "" ]; then - echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then - echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi + echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_DTEST_NAME} $count times" + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_compression_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi - stop_on_failure_arg="" - if ${REPEATED_DTEST_STOP_ON_FAILURE}; then - stop_on_failure_arg="-x" + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-compression + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" fi - vnodes_args="" - if ${REPEATED_DTEST_VNODES}; then - vnodes_args="--use-vnodes --num-tokens=16" + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" fi - upgrade_arg="" - if false; then - upgrade_arg="--execute-upgrade-tests" + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" fi - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt - fi - fi + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-compression $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) - store_test_results: - path: /tmp/results + path: /tmp/results/repeated_utests/output - store_artifacts: - path: /tmp/dtest - destination: dtest + path: /tmp/results/repeated_utests/stdout + destination: stdout - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_logs + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -1328,28 +1957,35 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - CASSANDRA_USE_JDK11: true - j8_repeated_dtest: + j11_repeated_ant_test: docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l @@ -1358,83 +1994,5876 @@ jobs: - attach_workspace: at: /home/cassandra - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies + name: Log Environment Information command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version - run: - name: Run repeated Python dtest + name: Run repeated JUnit test no_output_timeout: 15m command: | - if [ "${REPEATED_DTEST_NAME}" == "" ]; then - echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" == "" ]; then - echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then - echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + if [ "${REPEATED_ANT_TEST_CLASS}" == "" ]; then + echo "Repeated utest class name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_ANT_TEST_COUNT}" == "" ]; then + echo "Repeated utest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_ANT_TEST_COUNT}" -le 0 ]; then + echo "Repeated utest count is lesser or equals than zero, exiting without running any test" else # Calculate the number of test iterations to be run by the current parallel runner. # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((${REPEATED_ANT_TEST_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_ANT_TEST_COUNT} % CIRCLE_NODE_TOTAL))); then count=$((count+1)) fi if (($count <= 0)); then echo "No tests to run in this runner" else - echo "Running ${REPEATED_DTEST_NAME} $count times" + echo "Running ${REPEATED_ANT_TEST_TARGET} ${REPEATED_ANT_TEST_CLASS} ${REPEATED_ANT_TEST_METHODS} ${REPEATED_ANT_TEST_COUNT} times" - source ~/env3.6/bin/activate + set -x export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests + target=${REPEATED_ANT_TEST_TARGET} + class_path=${REPEATED_ANT_TEST_CLASS} + class_name="${class_path##*.}" - stop_on_failure_arg="" - if ${REPEATED_DTEST_STOP_ON_FAILURE}; then - stop_on_failure_arg="-x" + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name="-Dtest.name=$class_name" + else + name="-Dtest.name=$class_path" fi - vnodes_args="" - if ${REPEATED_DTEST_VNODES}; then - vnodes_args="--use-vnodes --num-tokens=16" + # Prepare the -Dtest.methods argument, which is optional + if [ "${REPEATED_ANT_TEST_METHODS}" == "" ]; then + methods="" + else + methods="-Dtest.methods=${REPEATED_ANT_TEST_METHODS}" fi - upgrade_arg="" - if false; then - upgrade_arg="--execute-upgrade-tests" - fi + # Run the test target as many times as requested collecting the exit code, + # stopping the iteration only if stop_on_failure is set. + exit_code="$?" + for i in $(seq -w 1 $count); do - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt - fi + echo "Running test iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utest/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${REPEATED_ANT_TEST_TARGET}-${REPEATED_ANT_TEST_CLASS}.txt + + # move the XML output files + source=build/test/output + dest=/tmp/results/repeated_utest/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs + dest=/tmp/results/repeated_utest/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + + (exit ${exit_code}) + fi + fi + - store_test_results: + path: /tmp/results/repeated_utest/output + - store_artifacts: + path: /tmp/results/repeated_utest/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utest/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utest/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_dtests_large_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_large_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_large_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --only-resource-intensive-tests --force-resource-intensive-tests --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_large_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_large_with_vnodes_raw /tmp/all_dtest_tests_j11_large_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_large_with_vnodes_raw > /tmp/all_dtest_tests_j11_large_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_large_with_vnodes > /tmp/split_dtest_tests_j11_large_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_large_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_large_with_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_large_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_large_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_large_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_dtests_large_vnode_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_LARGE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_LARGE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_LARGE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_LARGE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_LARGE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --only-resource-intensive-tests --force-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py311: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py38_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_dtests_large: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_large_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_large_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --only-resource-intensive-tests --force-resource-intensive-tests --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_large_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_large_without_vnodes_raw /tmp/all_dtest_tests_j11_large_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_large_without_vnodes_raw > /tmp/all_dtest_tests_j11_large_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_large_without_vnodes > /tmp/split_dtest_tests_j11_large_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_large_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_large_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_large_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_large_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_large_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_system_keyspace_directory_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-system-keyspace-directory + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-system-keyspace-directory $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py3_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py3: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_cdc: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-cdc) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-cdc -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_fqltool: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (fqltool-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant fqltool-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_system_keyspace_directory: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-system-keyspace-directory) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-system-keyspace-directory -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_offheap_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --use-off-heap-memtables --skip-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh-dtests-py2-offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_offheap_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --use-off-heap-memtables --skip-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtests_large_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_LARGE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_LARGE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_LARGE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_LARGE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_LARGE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --only-resource-intensive-tests --force-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_utests_compression: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-compression) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-compression -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_long: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (long-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant long-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_unit_tests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=testsome + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant testsome $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_large: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_large_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_large_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --only-resource-intensive-tests --force-resource-intensive-tests --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_large_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_large_without_vnodes_raw /tmp/all_dtest_tests_j8_large_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_large_without_vnodes_raw > /tmp/all_dtest_tests_j8_large_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_large_without_vnodes > /tmp/split_dtest_tests_j8_large_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_large_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_large_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_large_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_large_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_large_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh-dtests-py2-no-vnodes: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_stress: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (stress-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant stress-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py38_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_upgrade_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_UPGRADE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_UPGRADE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_UPGRADE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_UPGRADE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UPGRADE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_UPGRADE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_UPGRADE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if true; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh-dtests-py2-with-vnodes: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_cdc_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-cdc + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-cdc $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_fqltool_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_FQLTOOL_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_FQLTOOL_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_FQLTOOL} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=fqltool-test + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant fqltool-test $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_compression: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-compression) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-compression -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py38: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py3_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_cqlsh_dtests_py311_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_system_keyspace_directory_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-system-keyspace-directory + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-system-keyspace-directory $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_large_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_LARGE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_LARGE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_LARGE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_LARGE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_LARGE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --only-resource-intensive-tests --force-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py3_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_utests_stress_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_STRESS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_STRESS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_STRESS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=stress-test-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant stress-test-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_vnode_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py3_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_upgrade_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_upgradetests_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_upgradetests_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw /tmp/all_dtest_tests_j8_upgradetests_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw > /tmp/all_dtest_tests_j8_upgradetests_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_upgradetests_without_vnodes > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_upgradetests_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_upgradetests_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_upgradetests_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_upgradetests_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\"\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_jvm_upgrade_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_JVM_UPGRADE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_UPGRADE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_JVM_UPGRADE_DTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-jvm-dtest-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-jvm-dtest-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py38_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_jvm_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_JVM_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_JVM_DTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-jvm-dtest-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-jvm-dtest-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_cqlsh_dtests_py311_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_long_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_LONG_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_LONG_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_LONG} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=long-testsome + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant long-testsome $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\"\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_unit_tests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m - store_test_results: - path: /tmp/results + path: /tmp/cassandra/build/test/output/ - store_artifacts: - path: /tmp/dtest - destination: dtest + path: /tmp/cassandra/build/test/output + destination: junitxml - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_logs + path: /tmp/cassandra/build/test/logs + destination: logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -1451,85 +7880,106 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_cqlsh-dtests-py2-with-vnodes: + j11_jvm_dtests: docker: - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l - parallelism: 4 + parallelism: 1 steps: - attach_workspace: at: /home/cassandra - run: - name: Clone Cassandra dtest Repository (via git) + name: Determine distributed Tests to Run command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m - run: - name: Configure virtualenv and python Dependencies + name: Log Environment Information command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m + name: Run Unit Tests (testclasslist) command: | - echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt - - source ~/env3.6/bin/activate + set -x export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed -Dno-build-test=true + no_output_timeout: 15m - store_test_results: - path: /tmp/results + path: /tmp/cassandra/build/test/output/ - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes + path: /tmp/cassandra/build/test/output + destination: junitxml - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs + path: /tmp/cassandra/build/test/logs + destination: logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -1546,35 +7996,40 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - CASSANDRA_USE_JDK11: true - j11_dtests-with-vnodes: + j11_build: docker: - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l - parallelism: 4 + parallelism: 1 steps: - - attach_workspace: - at: /home/cassandra - run: name: Log Environment Information command: | @@ -1598,37 +8053,39 @@ jobs: which java java -version - run: - name: Clone Cassandra dtest Repository (via git) + name: Clone Cassandra Repository (via git) command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra - run: - name: Configure virtualenv and python Dependencies + name: Build Cassandra command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) + cd ~/cassandra + # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. + for x in $(seq 1 3); do + ${ANT_HOME}/bin/ant clean realclean jar + RETURN="$?" + if [ "${RETURN}" -eq "0" ]; then + break + fi + done + # Exit, if we didn't build successfully + if [ "${RETURN}" -ne "0" ]; then + echo "Build failed with exit code: ${RETURN}" + exit ${RETURN} + fi no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs + - run: + name: Run eclipse-warnings + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + ant eclipse-warnings + - persist_to_workspace: + root: /home/cassandra + paths: + - cassandra + - .m2 environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -1645,26 +8102,33 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - CASSANDRA_USE_JDK11: true - j8_dtests-no-vnodes: + j8_dtests: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium @@ -1697,7 +8161,7 @@ jobs: - run: name: Run dtests (j8_without_vnodes) no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" + command: "echo \"cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" - store_test_results: path: /tmp/results - store_artifacts: @@ -1722,27 +8186,34 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_upgradetests-no-vnodes: + j11_cqlsh-dtests-py2-no-vnodes: docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l @@ -1767,84 +8238,45 @@ jobs: pip3 uninstall -y cqlsh pip3 freeze - run: - name: Determine Tests to Run (j8_upgradetests_without_vnodes) + name: Determine Tests to Run (j11_without_vnodes) no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_upgradetests_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw /tmp/all_dtest_tests_j8_upgradetests_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw > /tmp/all_dtest_tests_j8_upgradetests_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_upgradetests_without_vnodes > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n" + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - run: - name: Run dtests (j8_upgradetests_without_vnodes) + name: Run dtests (j11_without_vnodes) no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_upgradetests_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_upgradetests_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_upgradetests_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_stress: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run Unit Tests (stress-test) command: | + echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt + + source ~/env3.6/bin/activate export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) fi - ant stress-test - no_output_timeout: 15m - store_test_results: - path: /tmp/cassandra/build/test/output/ + path: /tmp/results - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml + path: /tmp/dtest + destination: dtest_j11_without_vnodes - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -1861,25 +8293,33 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_unit_tests: + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtests_vnode: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium @@ -1890,70 +8330,37 @@ jobs: - attach_workspace: at: /home/cassandra - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information + name: Clone Cassandra dtest Repository (via git) command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - run: - name: Run Unit Tests (testclasslist) + name: Configure virtualenv and python Dependencies command: | - set -x + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" - store_test_results: - path: /tmp/cassandra/build/test/output/ + path: /tmp/results - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml + path: /tmp/dtest + destination: dtest_j8_with_vnodes - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -1970,25 +8377,32 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_jvm_dtests: + j11_cqlshlib_tests: docker: - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium @@ -1999,70 +8413,15 @@ jobs: - attach_workspace: at: /home/cassandra - run: - name: Determine distributed Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) + name: Run cqlshlib Unit Tests command: | - set -x export PATH=$JAVA_HOME/bin:$PATH time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed + cd /tmp/cassandra/pylib + ./cassandra-cqlsh-tests.sh .. no_output_timeout: 15m - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs + path: /tmp/cassandra/pylib environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -2079,33 +8438,42 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - CASSANDRA_USE_JDK11: true - j11_build: + j8_jvm_dtests_repeat: docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l - parallelism: 1 + parallelism: 4 steps: + - attach_workspace: + at: /home/cassandra - run: name: Log Environment Information command: | @@ -2129,39 +8497,124 @@ jobs: which java java -version - run: - name: Clone Cassandra Repository (via git) - command: | - git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra - - run: - name: Build Cassandra - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant clean realclean jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi + name: Repeatedly run new or modifed JUnit tests no_output_timeout: 15m - - run: - name: Run eclipse-warnings command: | + set -x export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - ant eclipse-warnings - - persist_to_workspace: - root: /home/cassandra - paths: - - cassandra - - .m2 + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_JVM_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_JVM_DTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-jvm-dtest-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-jvm-dtest-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -2178,26 +8631,32 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlsh-dtests-py2-no-vnodes: + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests: docker: - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium @@ -2207,6 +8666,28 @@ jobs: steps: - attach_workspace: at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version - run: name: Clone Cassandra dtest Repository (via git) command: | @@ -2226,30 +8707,11 @@ jobs: - run: name: Determine Tests to Run (j11_without_vnodes) no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - run: name: Run dtests (j11_without_vnodes) no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + command: "echo \"cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" - store_test_results: path: /tmp/results - store_artifacts: @@ -2274,83 +8736,35 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - CASSANDRA_USE_JDK11: true - j11_cqlshlib_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run cqlshlib Unit Tests - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra/pylib - ./cassandra-cqlsh-tests.sh .. - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/pylib - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_repeated_utest: + j8_repeated_ant_test: docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l @@ -2384,25 +8798,25 @@ jobs: name: Run repeated JUnit test no_output_timeout: 15m command: | - if [ "${REPEATED_UTEST_CLASS}" == "" ]; then + if [ "${REPEATED_ANT_TEST_CLASS}" == "" ]; then echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" == "" ]; then + elif [ "${REPEATED_ANT_TEST_COUNT}" == "" ]; then echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" -le 0 ]; then + elif [ "${REPEATED_ANT_TEST_COUNT}" -le 0 ]; then echo "Repeated utest count is lesser or equals than zero, exiting without running any test" else # Calculate the number of test iterations to be run by the current parallel runner. # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_UTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_UTEST_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((${REPEATED_ANT_TEST_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_ANT_TEST_COUNT} % CIRCLE_NODE_TOTAL))); then count=$((count+1)) fi if (($count <= 0)); then echo "No tests to run in this runner" else - echo "Running ${REPEATED_UTEST_TARGET} ${REPEATED_UTEST_CLASS} ${REPEATED_UTEST_METHODS} ${REPEATED_UTEST_COUNT} times" + echo "Running ${REPEATED_ANT_TEST_TARGET} ${REPEATED_ANT_TEST_CLASS} ${REPEATED_ANT_TEST_METHODS} ${REPEATED_ANT_TEST_COUNT} times" set -x export PATH=$JAVA_HOME/bin:$PATH @@ -2412,8 +8826,8 @@ jobs: cp ~/dtest_jars/dtest* /tmp/cassandra/build/ fi - target=${REPEATED_UTEST_TARGET} - class_path=${REPEATED_UTEST_CLASS} + target=${REPEATED_ANT_TEST_TARGET} + class_path=${REPEATED_ANT_TEST_CLASS} class_name="${class_path##*.}" # Prepare the -Dtest.name argument. @@ -2421,17 +8835,20 @@ jobs: if [[ $target == "test" || \ $target == "test-cdc" || \ $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then name="-Dtest.name=$class_name" else name="-Dtest.name=$class_path" fi # Prepare the -Dtest.methods argument, which is optional - if [ "${REPEATED_UTEST_METHODS}" == "" ]; then + if [ "${REPEATED_ANT_TEST_METHODS}" == "" ]; then methods="" else - methods="-Dtest.methods=${REPEATED_UTEST_METHODS}" + methods="-Dtest.methods=${REPEATED_ANT_TEST_METHODS}" fi # Run the test target as many times as requested collecting the exit code, @@ -2451,7 +8868,7 @@ jobs: # move the stdout output file dest=/tmp/results/repeated_utest/stdout/${status}/${i} mkdir -p $dest - mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt + mv stdout.txt $dest/${REPEATED_ANT_TEST_TARGET}-${REPEATED_ANT_TEST_CLASS}.txt # move the XML output files source=build/test/output @@ -2470,7 +8887,7 @@ jobs: fi # maybe stop iterations on test failure - if [[ ${REPEATED_UTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then break fi done @@ -2505,26 +8922,218 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_dtests-with-vnodes: + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_utests_long: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (long-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant long-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_utests_cdc: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-cdc) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-cdc -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh_dtests_py311_offheap: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium @@ -2545,27 +9154,51 @@ jobs: # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate + source ~/env3.11/bin/activate export PATH=$JAVA_HOME/bin:$PATH pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt pip3 uninstall -y cqlsh pip3 freeze - run: - name: Determine Tests to Run (j8_with_vnodes) + name: Determine Tests to Run (j8_dtests_offheap) no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" - run: - name: Run dtests (j8_with_vnodes) + name: Run dtests (j8_dtests_offheap) no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" + command: | + echo "cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi - store_test_results: path: /tmp/results - store_artifacts: path: /tmp/dtest - destination: dtest_j8_with_vnodes + destination: dtest_j8_dtests_offheap - store_artifacts: path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs + destination: dtest_j8_dtests_offheap_logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -2582,27 +9215,34 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_cqlsh-dtests-py38-no-vnodes: + j8_dtests_repeat: docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l @@ -2621,46 +9261,75 @@ jobs: # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate + source ~/env3.6/bin/activate export PATH=$JAVA_HOME/bin:$PATH pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt pip3 uninstall -y cqlsh pip3 freeze - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) + name: Run repeated Python dtest no_output_timeout: 15m command: | - echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi - store_test_results: path: /tmp/results - store_artifacts: path: /tmp/dtest - destination: dtest_j11_without_vnodes + destination: dtest - store_artifacts: path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs + destination: dtest_logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -2677,25 +9346,31 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 j8_jvm_dtests: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest @@ -2761,7 +9436,7 @@ jobs: if [ -z "$test_timeout" ]; then test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed -Dno-build-test=true no_output_timeout: 15m - store_test_results: path: /tmp/cassandra/build/test/output/ @@ -2787,22 +9462,29 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 j8_build: @@ -2885,25 +9567,32 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py3-no-vnodes: + j8_cqlsh_dtests_py38_vnode: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium @@ -2924,26 +9613,26 @@ jobs: # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate + source ~/env3.8/bin/activate export PATH=$JAVA_HOME/bin:$PATH pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt pip3 uninstall -y cqlsh pip3 freeze - run: - name: Determine Tests to Run (j8_without_vnodes) + name: Determine Tests to Run (j8_with_vnodes) no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - run: - name: Run dtests (j8_without_vnodes) + name: Run dtests (j8_with_vnodes) no_output_timeout: 15m command: | - echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt + echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt - source ~/env3.6/bin/activate + source ~/env3.8/bin/activate export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 fi java -version @@ -2954,16 +9643,21 @@ jobs: echo "** done env" mkdir -p /tmp/results/dtests # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi - store_test_results: path: /tmp/results - store_artifacts: path: /tmp/dtest - destination: dtest_j8_without_vnodes + destination: dtest_j8_with_vnodes - store_artifacts: path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs + destination: dtest_j8_with_vnodes_logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -2980,27 +9674,34 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py38-with-vnodes: + j11_cqlsh-dtests-py2-offheap: docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l @@ -3019,26 +9720,26 @@ jobs: # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate + source ~/env3.6/bin/activate export PATH=$JAVA_HOME/bin:$PATH pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt pip3 uninstall -y cqlsh pip3 freeze - run: - name: Determine Tests to Run (j8_with_vnodes) + name: Determine Tests to Run (j11_dtests_offheap) no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" - run: - name: Run dtests (j8_with_vnodes) + name: Run dtests (j11_dtests_offheap) no_output_timeout: 15m command: | - echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt + echo "cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt - source ~/env3.8/bin/activate + source ~/env3.6/bin/activate export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 fi java -version @@ -3049,79 +9750,21 @@ jobs: echo "** done env" mkdir -p /tmp/results/dtests # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi - store_test_results: path: /tmp/results - store_artifacts: path: /tmp/dtest - destination: dtest_j8_with_vnodes + destination: dtest_j11_dtests_offheap - store_artifacts: path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_long: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run Unit Tests (long-test) - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - ant long-test - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs + destination: dtest_j11_dtests_offheap_logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -3138,27 +9781,35 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_system_keyspace_directory: + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_unit_tests_repeat: docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium working_directory: ~/ shell: /bin/bash -eo pipefail -l @@ -3166,25 +9817,6 @@ jobs: steps: - attach_workspace: at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - run: name: Log Environment Information command: | @@ -3208,7 +9840,8 @@ jobs: which java java -version - run: - name: Run Unit Tests (testclasslist-system-keyspace-directory) + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m command: | set -x export PATH=$JAVA_HOME/bin:$PATH @@ -3217,19 +9850,113 @@ jobs: if [ -d ~/dtest_jars ]; then cp ~/dtest_jars/dtest* /tmp/cassandra/build/ fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) fi - ant testclasslist-system-keyspace-directory -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=testsome + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant testsome $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) - store_test_results: - path: /tmp/cassandra/build/test/output/ + path: /tmp/results/repeated_utests/output - store_artifacts: - path: /tmp/cassandra/build/test/output + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output destination: junitxml - store_artifacts: - path: /tmp/cassandra/build/test/logs + path: /tmp/results/repeated_utests/logs destination: logs environment: - ANT_HOME: /usr/share/ant @@ -3247,25 +9974,33 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlshlib_tests: + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_fqltool: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium @@ -3276,15 +10011,24 @@ jobs: - attach_workspace: at: /home/cassandra - run: - name: Run cqlshlib Unit Tests + name: Run Unit Tests (fqltool-test) command: | export PATH=$JAVA_HOME/bin:$PATH time mv ~/cassandra /tmp - cd /tmp/cassandra/pylib - ./cassandra-cqlsh-tests.sh .. + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant fqltool-test -Dno-build-test=true no_output_timeout: 15m - store_test_results: - path: /tmp/cassandra/pylib + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -3301,25 +10045,32 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_fqltool: + j8_cqlshlib_tests: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium @@ -3330,24 +10081,15 @@ jobs: - attach_workspace: at: /home/cassandra - run: - name: Run Unit Tests (fqltool-test) + name: Run cqlshlib Unit Tests command: | export PATH=$JAVA_HOME/bin:$PATH time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - ant fqltool-test + cd /tmp/cassandra/pylib + ./cassandra-cqlsh-tests.sh .. no_output_timeout: 15m - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs + path: /tmp/cassandra/pylib environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -3364,25 +10106,32 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_dtests-no-vnodes: + j11_dtests_vnode: docker: - image: apache/cassandra-testing-ubuntu2004-java11:latest resource_class: medium @@ -3431,21 +10180,21 @@ jobs: pip3 uninstall -y cqlsh pip3 freeze - run: - name: Determine Tests to Run (j11_without_vnodes) + name: Determine Tests to Run (j11_with_vnodes) no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - run: - name: Run dtests (j11_without_vnodes) + name: Run dtests (j11_with_vnodes) no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" + command: "echo \"cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" - store_test_results: path: /tmp/results - store_artifacts: path: /tmp/dtest - destination: dtest_j11_without_vnodes + destination: dtest_j11_with_vnodes - store_artifacts: path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs + destination: dtest_j11_with_vnodes_logs environment: - ANT_HOME: /usr/share/ant - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 @@ -3462,26 +10211,33 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - CASSANDRA_USE_JDK11: true - utests_compression: + j8_utests_cdc_repeat: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium @@ -3491,25 +10247,6 @@ jobs: steps: - attach_workspace: at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - run: name: Log Environment Information command: | @@ -3533,7 +10270,8 @@ jobs: which java java -version - run: - name: Run Unit Tests (testclasslist-compression) + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m command: | set -x export PATH=$JAVA_HOME/bin:$PATH @@ -3542,19 +10280,113 @@ jobs: if [ -d ~/dtest_jars ]; then cp ~/dtest_jars/dtest* /tmp/cassandra/build/ fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) fi - ant testclasslist-compression -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-cdc + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-cdc $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) - store_test_results: - path: /tmp/cassandra/build/test/output/ + path: /tmp/results/repeated_utests/output - store_artifacts: - path: /tmp/cassandra/build/test/output + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output destination: junitxml - store_artifacts: - path: /tmp/cassandra/build/test/logs + path: /tmp/results/repeated_utests/logs destination: logs environment: - ANT_HOME: /usr/share/ant @@ -3572,25 +10404,32 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_repeated_utest: + j8_dtests_large_vnode_repeat: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest resource_class: medium @@ -3598,6 +10437,137 @@ jobs: shell: /bin/bash -eo pipefail -l parallelism: 4 steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_LARGE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_LARGE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_LARGE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_LARGE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_LARGE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --only-resource-intensive-tests --force-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_long_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: - attach_workspace: at: /home/cassandra - run: @@ -3623,113 +10593,123 @@ jobs: which java java -version - run: - name: Run repeated JUnit test + name: Repeatedly run new or modifed JUnit tests no_output_timeout: 15m command: | - if [ "${REPEATED_UTEST_CLASS}" == "" ]; then - echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" == "" ]; then - echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" -le 0 ]; then - echo "Repeated utest count is lesser or equals than zero, exiting without running any test" - else + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_UTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_UTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_LONG_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_LONG_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_UTEST_TARGET} ${REPEATED_UTEST_CLASS} ${REPEATED_UTEST_METHODS} ${REPEATED_UTEST_COUNT} times" + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_LONG} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=long-testsome + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi - target=${REPEATED_UTEST_TARGET} - class_path=${REPEATED_UTEST_CLASS} - class_name="${class_path##*.}" + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi # Prepare the -Dtest.name argument. # It can be the fully qualified class name or the short class name, depending on the target. if [[ $target == "test" || \ $target == "test-cdc" || \ $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then - name="-Dtest.name=$class_name" + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" else - name="-Dtest.name=$class_path" + name_arg="-Dtest.name=$class" fi # Prepare the -Dtest.methods argument, which is optional - if [ "${REPEATED_UTEST_METHODS}" == "" ]; then - methods="" + if [[ $method == "" ]]; then + methods_arg="" else - methods="-Dtest.methods=${REPEATED_UTEST_METHODS}" + methods_arg="-Dtest.methods=$method" fi - # Run the test target as many times as requested collecting the exit code, - # stopping the iteration only if stop_on_failure is set. - exit_code="$?" for i in $(seq -w 1 $count); do - - echo "Running test iteration $i of $count" + echo "Running test $test, iteration $i of $count" # run the test status="passes" - if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then + if !( set -o pipefail && \ + ant long-testsome $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then status="fails" exit_code=1 fi # move the stdout output file - dest=/tmp/results/repeated_utest/stdout/${status}/${i} + dest=/tmp/results/repeated_utests/stdout/${status}/${i} mkdir -p $dest - mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt + mv stdout.txt $dest/${test}.txt # move the XML output files - source=build/test/output - dest=/tmp/results/repeated_utest/output/${status}/${i} + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} mkdir -p $dest if [[ -d $source && -n "$(ls $source)" ]]; then mv $source/* $dest/ fi # move the log files - source=build/test/logs - dest=/tmp/results/repeated_utest/logs/${status}/${i} + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} mkdir -p $dest if [[ -d $source && -n "$(ls $source)" ]]; then mv $source/* $dest/ fi # maybe stop iterations on test failure - if [[ ${REPEATED_UTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then break fi done - - (exit ${exit_code}) - fi - fi + done + (exit ${exit_code}) - store_test_results: - path: /tmp/results/repeated_utest/output + path: /tmp/results/repeated_utests/output - store_artifacts: - path: /tmp/results/repeated_utest/stdout + path: /tmp/results/repeated_utests/stdout destination: stdout - store_artifacts: - path: /tmp/results/repeated_utest/output + path: /tmp/results/repeated_utests/output destination: junitxml - store_artifacts: - path: /tmp/results/repeated_utest/logs + path: /tmp/results/repeated_utests/logs destination: logs environment: - ANT_HOME: /usr/share/ant @@ -3747,24 +10727,32 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true j8_dtest_jars_build: docker: - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest @@ -3842,22 +10830,29 @@ jobs: - DTEST_BRANCH: trunk - CCM_MAX_HEAP_SIZE: 1024M - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 workflows: @@ -3893,35 +10888,77 @@ workflows: requires: - start_j11_unit_tests - j8_build - - start_utests_long: + - start_j8_utests_long: type: approval - - utests_long: + - j8_utests_long: requires: - - start_utests_long + - start_j8_utests_long - j8_build - - start_utests_compression: + - start_j11_utests_long: type: approval - - utests_compression: + - j11_utests_long: requires: - - start_utests_compression + - start_j11_utests_long - j8_build - - start_utests_stress: + - start_j8_utests_cdc: type: approval - - utests_stress: + - j8_utests_cdc: requires: - - start_utests_stress + - start_j8_utests_cdc - j8_build - - start_utests_fqltool: + - start_j11_utests_cdc: type: approval - - utests_fqltool: + - j11_utests_cdc: requires: - - start_utests_fqltool + - start_j11_utests_cdc - j8_build - - start_utests_system_keyspace_directory: + - start_j8_utests_compression: type: approval - - utests_system_keyspace_directory: + - j8_utests_compression: requires: - - start_utests_system_keyspace_directory + - start_j8_utests_compression + - j8_build + - start_j11_utests_compression: + type: approval + - j11_utests_compression: + requires: + - start_j11_utests_compression + - j8_build + - start_j8_utests_stress: + type: approval + - j8_utests_stress: + requires: + - start_j8_utests_stress + - j8_build + - start_j11_utests_stress: + type: approval + - j11_utests_stress: + requires: + - start_j11_utests_stress + - j8_build + - start_j8_utests_fqltool: + type: approval + - j8_utests_fqltool: + requires: + - start_j8_utests_fqltool + - j8_build + - start_j11_utests_fqltool: + type: approval + - j11_utests_fqltool: + requires: + - start_j11_utests_fqltool + - j8_build + - start_j8_utests_system_keyspace_directory: + type: approval + - j8_utests_system_keyspace_directory: + requires: + - start_j8_utests_system_keyspace_directory + - j8_build + - start_j11_utests_system_keyspace_directory: + type: approval + - j11_utests_system_keyspace_directory: + requires: + - start_j11_utests_system_keyspace_directory - j8_build - start_j8_dtest_jars_build: type: approval @@ -3929,35 +10966,75 @@ workflows: requires: - j8_build - start_j8_dtest_jars_build - - start_jvm_upgrade_dtest: + - start_jvm_upgrade_dtests: type: approval - j8_jvm_upgrade_dtests: requires: - - start_jvm_upgrade_dtest + - start_jvm_upgrade_dtests - j8_dtest_jars_build - start_j8_dtests: type: approval - - j8_dtests-with-vnodes: + - j8_dtests: requires: - start_j8_dtests - j8_build - - j8_dtests-no-vnodes: + - start_j8_dtests_vnode: + type: approval + - j8_dtests_vnode: requires: - - start_j8_dtests + - start_j8_dtests_vnode + - j8_build + - start_j8_dtests_offheap: + type: approval + - j8_dtests_offheap: + requires: + - start_j8_dtests_offheap + - j8_build + - start_j8_dtests_large: + type: approval + - j8_dtests_large: + requires: + - start_j8_dtests_large + - j8_build + - start_j8_dtests_large_vnode: + type: approval + - j8_dtests_large_vnode: + requires: + - start_j8_dtests_large_vnode - j8_build - start_j11_dtests: type: approval - - j11_dtests-with-vnodes: + - j11_dtests: requires: - start_j11_dtests - j8_build - - j11_dtests-no-vnodes: + - start_j11_dtests_vnode: + type: approval + - j11_dtests_vnode: + requires: + - start_j11_dtests_vnode + - j8_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: requires: - - start_j11_dtests + - start_j11_dtests_offheap + - j8_build + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j8_build + - start_j11_dtests_large_vnode: + type: approval + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large_vnode - j8_build - start_upgrade_tests: type: approval - - j8_upgradetests-no-vnodes: + - j8_upgrade_dtests: requires: - start_upgrade_tests - j8_build @@ -3967,11 +11044,15 @@ workflows: requires: - start_j8_cqlsh_tests - j8_build - - j8_cqlsh-dtests-py3-with-vnodes: + - j8_cqlsh_dtests_py3_vnode: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py38_vnode: requires: - start_j8_cqlsh_tests - j8_build - - j8_cqlsh-dtests-py38-with-vnodes: + - j8_cqlsh_dtests_py311_vnode: requires: - start_j8_cqlsh_tests - j8_build @@ -3979,76 +11060,88 @@ workflows: requires: - start_j8_cqlsh_tests - j8_build - - j8_cqlsh-dtests-py3-no-vnodes: + - j8_cqlsh_dtests_py3: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py38: requires: - start_j8_cqlsh_tests - j8_build - - j8_cqlsh-dtests-py38-no-vnodes: + - j8_cqlsh_dtests_py311: requires: - start_j8_cqlsh_tests - j8_build + - start_j8_cqlsh_tests_offheap: + type: approval + - j8_cqlsh-dtests-py2-offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - j8_cqlsh_dtests_py3_offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - j8_cqlsh_dtests_py38_offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - j8_cqlsh_dtests_py311_offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build - start_j11_cqlsh_tests: type: approval - j11_cqlsh-dtests-py2-with-vnodes: requires: - start_j11_cqlsh_tests - j8_build - - j11_cqlsh-dtests-py3-with-vnodes: + - j11_cqlsh_dtests_py3_vnode: requires: - start_j11_cqlsh_tests - j8_build - - j11_cqlsh-dtests-py38-with-vnodes: + - j11_cqlsh_dtests_py38_vnode: requires: - start_j11_cqlsh_tests - j8_build - - j11_cqlsh-dtests-py2-no-vnodes: + - j11_cqlsh_dtests_py311_vnode: requires: - start_j11_cqlsh_tests - j8_build - - j11_cqlsh-dtests-py3-no-vnodes: + - j11_cqlsh-dtests-py2-no-vnodes: requires: - start_j11_cqlsh_tests - j8_build - - j11_cqlsh-dtests-py38-no-vnodes: + - j11_cqlsh_dtests_py3: requires: - start_j11_cqlsh_tests - j8_build - - start_j8_repeated_utest: - type: approval - - j8_repeated_utest: + - j11_cqlsh_dtests_py38: requires: - - start_j8_repeated_utest + - start_j11_cqlsh_tests - j8_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: + - j11_cqlsh_dtests_py311: requires: - - start_j11_repeated_utest + - start_j11_cqlsh_tests - j8_build - - start_j8_repeated_dtest: + - start_j11_cqlsh_tests_offheap: type: approval - - j8_repeated_dtest: + - j11_cqlsh-dtests-py2-offheap: requires: - - start_j8_repeated_dtest + - start_j11_cqlsh_tests_offheap - j8_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: + - j11_cqlsh_dtests_py3_offheap: requires: - - start_j11_repeated_dtest + - start_j11_cqlsh_tests_offheap - j8_build - - start_repeated_upgrade_dtest: - type: approval - - repeated_upgrade_dtest: + - j11_cqlsh_dtests_py38_offheap: requires: - - start_repeated_upgrade_dtest + - start_j11_cqlsh_tests_offheap - j8_build - - start_repeated_jvm_upgrade_dtest: - type: approval - - repeated_jvm_upgrade_dtest: + - j11_cqlsh_dtests_py311_offheap: requires: - - start_repeated_jvm_upgrade_dtest - - j8_dtest_jars_build + - start_j11_cqlsh_tests_offheap + - j8_build java8_pre-commit_tests: jobs: - start_pre-commit_tests: @@ -4070,133 +11163,206 @@ workflows: - j8_build - start_utests_long: type: approval - - utests_long: + - j8_utests_long: requires: - start_utests_long - j8_build + - j11_utests_long: + requires: + - start_utests_long + - j8_build + - start_utests_cdc: + type: approval + - j8_utests_cdc: + requires: + - start_utests_cdc + - j8_build + - j11_utests_cdc: + requires: + - start_utests_cdc + - j8_build - start_utests_compression: type: approval - - utests_compression: + - j8_utests_compression: + requires: + - start_utests_compression + - j8_build + - j11_utests_compression: requires: - start_utests_compression - j8_build - start_utests_stress: type: approval - - utests_stress: + - j8_utests_stress: + requires: + - start_utests_stress + - j8_build + - j11_utests_stress: requires: - start_utests_stress - j8_build - start_utests_fqltool: type: approval - - utests_fqltool: + - j8_utests_fqltool: requires: - start_utests_fqltool - j8_build - - utests_system_keyspace_directory: + - j11_utests_fqltool: requires: + - start_utests_fqltool - j8_build - - start_j8_dtest_jars_build: + - start_utests_system_keyspace_directory: type: approval - - j8_dtest_jars_build: + - j8_utests_system_keyspace_directory: requires: - j8_build - - start_j8_dtest_jars_build - - start_jvm_upgrade_dtest: + - j11_utests_system_keyspace_directory: + requires: + - start_utests_system_keyspace_directory + - j8_build + - start_jvm_upgrade_dtests: type: approval + - j8_dtest_jars_build: + requires: + - j8_build + - start_jvm_upgrade_dtests - j8_jvm_upgrade_dtests: requires: - - start_jvm_upgrade_dtest - j8_dtest_jars_build - - j8_dtests-with-vnodes: + - j8_dtests: + requires: + - j8_build + - j8_dtests_vnode: + requires: + - j8_build + - start_j8_dtests_offheap: + type: approval + - j8_dtests_offheap: + requires: + - start_j8_dtests_offheap + - j8_build + - j11_dtests: requires: - j8_build - - j8_dtests-no-vnodes: + - j11_dtests_vnode: requires: - j8_build - - j11_dtests-with-vnodes: + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j8_build + - start_j8_dtests_large: + type: approval + - j8_dtests_large: + requires: + - start_j8_dtests_large + - j8_build + - j8_dtests_large_vnode: + requires: + - start_j8_dtests_large + - j8_build + - start_j11_dtests_large: + type: approval + - j11_dtests_large: requires: + - start_j11_dtests_large - j8_build - - j11_dtests-no-vnodes: + - j11_dtests_large_vnode: requires: + - start_j11_dtests_large - j8_build - start_upgrade_tests: type: approval - - j8_upgradetests-no-vnodes: + - j8_upgrade_dtests: requires: - j8_build - start_upgrade_tests - j8_cqlsh-dtests-py2-with-vnodes: requires: - j8_build - - j8_cqlsh-dtests-py3-with-vnodes: + - j8_cqlsh_dtests_py3_vnode: + requires: + - j8_build + - j8_cqlsh_dtests_py38_vnode: requires: - j8_build - - j8_cqlsh-dtests-py38-with-vnodes: + - j8_cqlsh_dtests_py311_vnode: requires: - j8_build - j8_cqlsh-dtests-py2-no-vnodes: requires: - j8_build - - j8_cqlsh-dtests-py3-no-vnodes: + - j8_cqlsh_dtests_py3: + requires: + - j8_build + - j8_cqlsh_dtests_py38: + requires: + - j8_build + - j8_cqlsh_dtests_py311: + requires: + - j8_build + - start_j8_cqlsh_dtests_offheap: + type: approval + - j8_cqlsh-dtests-py2-offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j8_cqlsh_dtests_py3_offheap: requires: + - start_j8_cqlsh_dtests_offheap - j8_build - - j8_cqlsh-dtests-py38-no-vnodes: + - j8_cqlsh_dtests_py38_offheap: requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j8_cqlsh_dtests_py311_offheap: + requires: + - start_j8_cqlsh_dtests_offheap - j8_build - j11_cqlsh-dtests-py2-with-vnodes: requires: - j8_build - - j11_cqlsh-dtests-py3-with-vnodes: + - j11_cqlsh_dtests_py3_vnode: requires: - j8_build - - j11_cqlsh-dtests-py38-with-vnodes: + - j11_cqlsh_dtests_py38_vnode: requires: - j8_build - - j11_cqlsh-dtests-py2-no-vnodes: + - j11_cqlsh_dtests_py311_vnode: requires: - j8_build - - j11_cqlsh-dtests-py3-no-vnodes: + - j11_cqlsh-dtests-py2-no-vnodes: requires: - j8_build - - j11_cqlsh-dtests-py38-no-vnodes: + - j11_cqlsh_dtests_py3: requires: - j8_build - - start_j8_repeated_utest: - type: approval - - j8_repeated_utest: + - j11_cqlsh_dtests_py38: requires: - - start_j8_repeated_utest - j8_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: + - j11_cqlsh_dtests_py311: requires: - - start_j11_repeated_utest - j8_build - - start_j8_repeated_dtest: + - start_j11_cqlsh-dtests-offheap: type: approval - - j8_repeated_dtest: + - j11_cqlsh-dtests-py2-offheap: requires: - - start_j8_repeated_dtest + - start_j11_cqlsh-dtests-offheap - j8_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: + - j11_cqlsh_dtests_py3_offheap: requires: - - start_j11_repeated_dtest + - start_j11_cqlsh-dtests-offheap - j8_build - - start_repeated_upgrade_dtest: - type: approval - - repeated_upgrade_dtest: + - j11_cqlsh_dtests_py38_offheap: requires: - - start_repeated_upgrade_dtest + - start_j11_cqlsh-dtests-offheap - j8_build - - start_repeated_jvm_upgrade_dtest: - type: approval - - repeated_jvm_upgrade_dtest: + - j11_cqlsh_dtests_py311_offheap: requires: - - start_repeated_jvm_upgrade_dtest - - j8_dtest_jars_build + - start_j11_cqlsh-dtests-offheap + - j8_build java11_separate_tests: jobs: - start_j11_build: @@ -4224,13 +11390,21 @@ workflows: - j11_build - start_j11_dtests: type: approval - - j11_dtests-with-vnodes: + - j11_dtests: requires: - start_j11_dtests - j11_build - - j11_dtests-no-vnodes: + - start_j11_dtests_vnode: + type: approval + - j11_dtests_vnode: + requires: + - start_j11_dtests_vnode + - j11_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: requires: - - start_j11_dtests + - start_j11_dtests_offheap - j11_build - start_j11_cqlsh_tests: type: approval @@ -4238,11 +11412,15 @@ workflows: requires: - start_j11_cqlsh_tests - j11_build - - j11_cqlsh-dtests-py3-with-vnodes: + - j11_cqlsh_dtests_py3_vnode: requires: - start_j11_cqlsh_tests - j11_build - - j11_cqlsh-dtests-py38-with-vnodes: + - j11_cqlsh_dtests_py38_vnode: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py311_vnode: requires: - start_j11_cqlsh_tests - j11_build @@ -4250,25 +11428,83 @@ workflows: requires: - start_j11_cqlsh_tests - j11_build - - j11_cqlsh-dtests-py3-no-vnodes: + - j11_cqlsh_dtests_py3: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py38: requires: - start_j11_cqlsh_tests - j11_build - - j11_cqlsh-dtests-py38-no-vnodes: + - j11_cqlsh_dtests_py311: requires: - start_j11_cqlsh_tests - j11_build - - start_j11_repeated_utest: + - start_j11_cqlsh-dtests-offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j11_build + - start_j11_dtests_large_vnode: + type: approval + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large_vnode + - j11_build + - start_j11_utests_long: + type: approval + - j11_utests_long: + requires: + - start_j11_utests_long + - j11_build + - start_j11_utests_cdc: + type: approval + - j11_utests_cdc: + requires: + - start_j11_utests_cdc + - j11_build + - start_j11_utests_compression: + type: approval + - j11_utests_compression: + requires: + - start_j11_utests_compression + - j11_build + - start_j11_utests_stress: + type: approval + - j11_utests_stress: + requires: + - start_j11_utests_stress + - j11_build + - start_j11_utests_fqltool: type: approval - - j11_repeated_utest: + - j11_utests_fqltool: requires: - - start_j11_repeated_utest + - start_j11_utests_fqltool - j11_build - - start_j11_repeated_dtest: + - start_j11_utests_system_keyspace_directory: type: approval - - j11_repeated_dtest: + - j11_utests_system_keyspace_directory: requires: - - start_j11_repeated_dtest + - start_j11_utests_system_keyspace_directory - j11_build java11_pre-commit_tests: jobs: @@ -4292,39 +11528,103 @@ workflows: - j11_cqlshlib_tests: requires: - j11_build - - j11_dtests-with-vnodes: + - j11_dtests: + requires: + - j11_build + - j11_dtests_vnode: requires: - j11_build - - j11_dtests-no-vnodes: + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: requires: + - start_j11_dtests_offheap - j11_build - j11_cqlsh-dtests-py2-with-vnodes: requires: - j11_build - - j11_cqlsh-dtests-py3-with-vnodes: + - j11_cqlsh_dtests_py3_vnode: requires: - j11_build - - j11_cqlsh-dtests-py38-with-vnodes: + - j11_cqlsh_dtests_py38_vnode: + requires: + - j11_build + - j11_cqlsh_dtests_py311_vnode: requires: - j11_build - j11_cqlsh-dtests-py2-no-vnodes: requires: - j11_build - - j11_cqlsh-dtests-py3-no-vnodes: + - j11_cqlsh_dtests_py3: + requires: + - j11_build + - j11_cqlsh_dtests_py38: + requires: + - j11_build + - j11_cqlsh_dtests_py311: + requires: + - j11_build + - start_j11_cqlsh-dtests-offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j11_build + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large + - j11_build + - start_utests_long: + type: approval + - j11_utests_long: requires: + - start_utests_long + - j11_build + - start_utests_cdc: + type: approval + - j11_utests_cdc: + requires: + - start_utests_cdc + - j11_build + - start_utests_compression: + type: approval + - j11_utests_compression: + requires: + - start_utests_compression - j11_build - - j11_cqlsh-dtests-py38-no-vnodes: + - start_utests_stress: + type: approval + - j11_utests_stress: requires: + - start_utests_stress - j11_build - - start_j11_repeated_utest: + - start_utests_fqltool: type: approval - - j11_repeated_utest: + - j11_utests_fqltool: requires: - - start_j11_repeated_utest + - start_utests_fqltool - j11_build - - start_j11_repeated_dtest: + - start_utests_system_keyspace_directory: type: approval - - j11_repeated_dtest: + - j11_utests_system_keyspace_directory: requires: - - start_j11_repeated_dtest + - start_utests_system_keyspace_directory - j11_build diff --git a/.circleci/config.yml.FREE b/.circleci/config.yml.FREE new file mode 100644 index 000000000000..1bae80b66cc7 --- /dev/null +++ b/.circleci/config.yml.FREE @@ -0,0 +1,11630 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +version: 2 +jobs: + j8_jvm_upgrade_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine distributed Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_fqltool_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_FQLTOOL_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_FQLTOOL_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_FQLTOOL} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=fqltool-test + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant fqltool-test $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh-dtests-py2-with-vnodes: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh_dtests_py311_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_vnode_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtests_large_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_large_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_large_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --only-resource-intensive-tests --force-resource-intensive-tests --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_large_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_large_with_vnodes_raw /tmp/all_dtest_tests_j8_large_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_large_with_vnodes_raw > /tmp/all_dtest_tests_j8_large_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_large_with_vnodes > /tmp/split_dtest_tests_j8_large_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_large_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_large_with_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_large_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_large_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_large_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_system_keyspace_directory: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-system-keyspace-directory) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-system-keyspace-directory -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_stress: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (stress-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant stress-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_stress_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_STRESS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_STRESS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_STRESS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=stress-test-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant stress-test-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_cqlsh_dtests_py311: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_compression_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-compression + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-compression $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_unit_tests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py3: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py38: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_compression_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-compression + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-compression $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_repeated_ant_test: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run repeated JUnit test + no_output_timeout: 15m + command: | + if [ "${REPEATED_ANT_TEST_CLASS}" == "" ]; then + echo "Repeated utest class name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_ANT_TEST_COUNT}" == "" ]; then + echo "Repeated utest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_ANT_TEST_COUNT}" -le 0 ]; then + echo "Repeated utest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_ANT_TEST_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_ANT_TEST_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_ANT_TEST_TARGET} ${REPEATED_ANT_TEST_CLASS} ${REPEATED_ANT_TEST_METHODS} ${REPEATED_ANT_TEST_COUNT} times" + + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + target=${REPEATED_ANT_TEST_TARGET} + class_path=${REPEATED_ANT_TEST_CLASS} + class_name="${class_path##*.}" + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name="-Dtest.name=$class_name" + else + name="-Dtest.name=$class_path" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [ "${REPEATED_ANT_TEST_METHODS}" == "" ]; then + methods="" + else + methods="-Dtest.methods=${REPEATED_ANT_TEST_METHODS}" + fi + + # Run the test target as many times as requested collecting the exit code, + # stopping the iteration only if stop_on_failure is set. + exit_code="$?" + for i in $(seq -w 1 $count); do + + echo "Running test iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utest/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${REPEATED_ANT_TEST_TARGET}-${REPEATED_ANT_TEST_CLASS}.txt + + # move the XML output files + source=build/test/output + dest=/tmp/results/repeated_utest/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs + dest=/tmp/results/repeated_utest/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + + (exit ${exit_code}) + fi + fi + - store_test_results: + path: /tmp/results/repeated_utest/output + - store_artifacts: + path: /tmp/results/repeated_utest/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utest/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utest/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_dtests_large_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_large_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_large_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --only-resource-intensive-tests --force-resource-intensive-tests --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_large_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_large_with_vnodes_raw /tmp/all_dtest_tests_j11_large_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_large_with_vnodes_raw > /tmp/all_dtest_tests_j11_large_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_large_with_vnodes > /tmp/split_dtest_tests_j11_large_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_large_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_large_with_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_large_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_large_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_large_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_dtests_large_vnode_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_LARGE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_LARGE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_LARGE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_LARGE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_LARGE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --only-resource-intensive-tests --force-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py311: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py38_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_dtests_large: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_large_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_large_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --only-resource-intensive-tests --force-resource-intensive-tests --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_large_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_large_without_vnodes_raw /tmp/all_dtest_tests_j11_large_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_large_without_vnodes_raw > /tmp/all_dtest_tests_j11_large_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_large_without_vnodes > /tmp/split_dtest_tests_j11_large_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_large_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_large_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_large_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_large_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_large_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_system_keyspace_directory_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-system-keyspace-directory + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-system-keyspace-directory $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py3_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py3: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_cdc: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-cdc) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-cdc -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_fqltool: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (fqltool-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant fqltool-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_system_keyspace_directory: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-system-keyspace-directory) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-system-keyspace-directory -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_offheap_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --use-off-heap-memtables --skip-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh-dtests-py2-offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_offheap_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --use-off-heap-memtables --skip-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtests_large_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_LARGE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_LARGE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_LARGE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_LARGE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_LARGE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --only-resource-intensive-tests --force-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_utests_compression: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-compression) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-compression -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_long: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (long-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant long-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_unit_tests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=testsome + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant testsome $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_large: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_large_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_large_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --only-resource-intensive-tests --force-resource-intensive-tests --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_large_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_large_without_vnodes_raw /tmp/all_dtest_tests_j8_large_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_large_without_vnodes_raw > /tmp/all_dtest_tests_j8_large_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_large_without_vnodes > /tmp/split_dtest_tests_j8_large_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_large_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_large_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_large_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_large_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_large_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh-dtests-py2-no-vnodes: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_stress: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (stress-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant stress-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py38_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_upgrade_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_UPGRADE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_UPGRADE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_UPGRADE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_UPGRADE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UPGRADE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_UPGRADE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_UPGRADE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if true; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh-dtests-py2-with-vnodes: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_cdc_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-cdc + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-cdc $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_fqltool_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_FQLTOOL_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_FQLTOOL_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_FQLTOOL} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=fqltool-test + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant fqltool-test $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_compression: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-compression) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-compression -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py38: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py3_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_cqlsh_dtests_py311_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_system_keyspace_directory_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-system-keyspace-directory + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-system-keyspace-directory $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_large_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_LARGE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_LARGE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_LARGE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_LARGE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_LARGE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --only-resource-intensive-tests --force-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py3_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_utests_stress_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_STRESS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_STRESS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_STRESS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=stress-test-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant stress-test-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_vnode_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py3_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_upgrade_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_upgradetests_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_upgradetests_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw /tmp/all_dtest_tests_j8_upgradetests_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw > /tmp/all_dtest_tests_j8_upgradetests_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_upgradetests_without_vnodes > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_upgradetests_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_upgradetests_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_upgradetests_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_upgradetests_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\"\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_jvm_upgrade_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_JVM_UPGRADE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_UPGRADE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_JVM_UPGRADE_DTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-jvm-dtest-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-jvm-dtest-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py38_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_jvm_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_JVM_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_JVM_DTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-jvm-dtest-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-jvm-dtest-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_cqlsh_dtests_py311_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_long_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_LONG_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_LONG_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_LONG} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=long-testsome + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant long-testsome $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\"\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_unit_tests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_jvm_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine distributed Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_build: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra Repository (via git) + command: | + git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra + - run: + name: Build Cassandra + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. + for x in $(seq 1 3); do + ${ANT_HOME}/bin/ant clean realclean jar + RETURN="$?" + if [ "${RETURN}" -eq "0" ]; then + break + fi + done + # Exit, if we didn't build successfully + if [ "${RETURN}" -ne "0" ]; then + echo "Build failed with exit code: ${RETURN}" + exit ${RETURN} + fi + no_output_timeout: 15m + - run: + name: Run eclipse-warnings + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + ant eclipse-warnings + - persist_to_workspace: + root: /home/cassandra + paths: + - cassandra + - .m2 + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh-dtests-py2-no-vnodes: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtests_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlshlib_tests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run cqlshlib Unit Tests + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra/pylib + ./cassandra-cqlsh-tests.sh .. + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/pylib + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_jvm_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_JVM_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_JVM_DTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-jvm-dtest-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-jvm-dtest-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_repeated_ant_test: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run repeated JUnit test + no_output_timeout: 15m + command: | + if [ "${REPEATED_ANT_TEST_CLASS}" == "" ]; then + echo "Repeated utest class name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_ANT_TEST_COUNT}" == "" ]; then + echo "Repeated utest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_ANT_TEST_COUNT}" -le 0 ]; then + echo "Repeated utest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_ANT_TEST_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_ANT_TEST_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_ANT_TEST_TARGET} ${REPEATED_ANT_TEST_CLASS} ${REPEATED_ANT_TEST_METHODS} ${REPEATED_ANT_TEST_COUNT} times" + + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + target=${REPEATED_ANT_TEST_TARGET} + class_path=${REPEATED_ANT_TEST_CLASS} + class_name="${class_path##*.}" + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name="-Dtest.name=$class_name" + else + name="-Dtest.name=$class_path" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [ "${REPEATED_ANT_TEST_METHODS}" == "" ]; then + methods="" + else + methods="-Dtest.methods=${REPEATED_ANT_TEST_METHODS}" + fi + + # Run the test target as many times as requested collecting the exit code, + # stopping the iteration only if stop_on_failure is set. + exit_code="$?" + for i in $(seq -w 1 $count); do + + echo "Running test iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utest/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${REPEATED_ANT_TEST_TARGET}-${REPEATED_ANT_TEST_CLASS}.txt + + # move the XML output files + source=build/test/output + dest=/tmp/results/repeated_utest/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs + dest=/tmp/results/repeated_utest/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + + (exit ${exit_code}) + fi + fi + - store_test_results: + path: /tmp/results/repeated_utest/output + - store_artifacts: + path: /tmp/results/repeated_utest/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utest/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utest/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_utests_long: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (long-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant long-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_utests_cdc: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-cdc) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-cdc -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh_dtests_py311_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_jvm_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine distributed Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_build: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra Repository (via git) + command: | + git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra + - run: + name: Build Cassandra + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. + for x in $(seq 1 3); do + ${ANT_HOME}/bin/ant clean realclean jar + RETURN="$?" + if [ "${RETURN}" -eq "0" ]; then + break + fi + done + # Exit, if we didn't build successfully + if [ "${RETURN}" -ne "0" ]; then + echo "Build failed with exit code: ${RETURN}" + exit ${RETURN} + fi + no_output_timeout: 15m + - run: + name: Run eclipse-warnings + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + ant eclipse-warnings + - persist_to_workspace: + root: /home/cassandra + paths: + - cassandra + - .m2 + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh_dtests_py38_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh-dtests-py2-offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_unit_tests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=testsome + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant testsome $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_fqltool: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (fqltool-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant fqltool-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlshlib_tests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run cqlshlib Unit Tests + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra/pylib + ./cassandra-cqlsh-tests.sh .. + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/pylib + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_cdc_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-cdc + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-cdc $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_large_vnode_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_LARGE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_LARGE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_LARGE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_LARGE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_LARGE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --only-resource-intensive-tests --force-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_long_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_LONG_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_LONG_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_LONG} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=long-testsome + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant long-testsome $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtest_jars_build: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Build Cassandra DTest jars + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + mkdir ~/dtest_jars + git remote add apache https://github.com/apache/cassandra.git + for branch in cassandra-2.2 cassandra-3.0 cassandra-3.11 cassandra-4.0 trunk; do + # check out the correct cassandra version: + git remote set-branches --add apache '$branch' + git fetch --depth 1 apache $branch + git checkout $branch + git clean -fd + # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. + for x in $(seq 1 3); do + ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar + RETURN="$?" + if [ "${RETURN}" -eq "0" ]; then + cp build/dtest*.jar ~/dtest_jars + break + fi + done + # Exit, if we didn't build successfully + if [ "${RETURN}" -ne "0" ]; then + echo "Build failed with exit code: ${RETURN}" + exit ${RETURN} + fi + done + # and build the dtest-jar for the branch under test + ${ANT_HOME}/bin/ant realclean + git checkout origin/$CIRCLE_BRANCH + git clean -fd + for x in $(seq 1 3); do + ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar + RETURN="$?" + if [ "${RETURN}" -eq "0" ]; then + cp build/dtest*.jar ~/dtest_jars + break + fi + done + # Exit, if we didn't build successfully + if [ "${RETURN}" -ne "0" ]; then + echo "Build failed with exit code: ${RETURN}" + exit ${RETURN} + fi + ls -l ~/dtest_jars + no_output_timeout: 15m + - persist_to_workspace: + root: /home/cassandra + paths: + - dtest_jars + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 +workflows: + version: 2 + java8_separate_tests: + jobs: + - start_j8_build: + type: approval + - j8_build: + requires: + - start_j8_build + - start_j8_unit_tests: + type: approval + - j8_unit_tests: + requires: + - start_j8_unit_tests + - j8_build + - start_j8_jvm_dtests: + type: approval + - j8_jvm_dtests: + requires: + - start_j8_jvm_dtests + - j8_build + - start_j8_cqlshlib_tests: + type: approval + - j8_cqlshlib_tests: + requires: + - start_j8_cqlshlib_tests + - j8_build + - start_j11_unit_tests: + type: approval + - j11_unit_tests: + requires: + - start_j11_unit_tests + - j8_build + - start_j8_utests_long: + type: approval + - j8_utests_long: + requires: + - start_j8_utests_long + - j8_build + - start_j11_utests_long: + type: approval + - j11_utests_long: + requires: + - start_j11_utests_long + - j8_build + - start_j8_utests_cdc: + type: approval + - j8_utests_cdc: + requires: + - start_j8_utests_cdc + - j8_build + - start_j11_utests_cdc: + type: approval + - j11_utests_cdc: + requires: + - start_j11_utests_cdc + - j8_build + - start_j8_utests_compression: + type: approval + - j8_utests_compression: + requires: + - start_j8_utests_compression + - j8_build + - start_j11_utests_compression: + type: approval + - j11_utests_compression: + requires: + - start_j11_utests_compression + - j8_build + - start_j8_utests_stress: + type: approval + - j8_utests_stress: + requires: + - start_j8_utests_stress + - j8_build + - start_j11_utests_stress: + type: approval + - j11_utests_stress: + requires: + - start_j11_utests_stress + - j8_build + - start_j8_utests_fqltool: + type: approval + - j8_utests_fqltool: + requires: + - start_j8_utests_fqltool + - j8_build + - start_j11_utests_fqltool: + type: approval + - j11_utests_fqltool: + requires: + - start_j11_utests_fqltool + - j8_build + - start_j8_utests_system_keyspace_directory: + type: approval + - j8_utests_system_keyspace_directory: + requires: + - start_j8_utests_system_keyspace_directory + - j8_build + - start_j11_utests_system_keyspace_directory: + type: approval + - j11_utests_system_keyspace_directory: + requires: + - start_j11_utests_system_keyspace_directory + - j8_build + - start_j8_dtest_jars_build: + type: approval + - j8_dtest_jars_build: + requires: + - j8_build + - start_j8_dtest_jars_build + - start_jvm_upgrade_dtests: + type: approval + - j8_jvm_upgrade_dtests: + requires: + - start_jvm_upgrade_dtests + - j8_dtest_jars_build + - start_j8_dtests: + type: approval + - j8_dtests: + requires: + - start_j8_dtests + - j8_build + - start_j8_dtests_vnode: + type: approval + - j8_dtests_vnode: + requires: + - start_j8_dtests_vnode + - j8_build + - start_j8_dtests_offheap: + type: approval + - j8_dtests_offheap: + requires: + - start_j8_dtests_offheap + - j8_build + - start_j8_dtests_large: + type: approval + - j8_dtests_large: + requires: + - start_j8_dtests_large + - j8_build + - start_j8_dtests_large_vnode: + type: approval + - j8_dtests_large_vnode: + requires: + - start_j8_dtests_large_vnode + - j8_build + - start_j11_dtests: + type: approval + - j11_dtests: + requires: + - start_j11_dtests + - j8_build + - start_j11_dtests_vnode: + type: approval + - j11_dtests_vnode: + requires: + - start_j11_dtests_vnode + - j8_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j8_build + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j8_build + - start_j11_dtests_large_vnode: + type: approval + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large_vnode + - j8_build + - start_upgrade_tests: + type: approval + - j8_upgrade_dtests: + requires: + - start_upgrade_tests + - j8_build + - start_j8_cqlsh_tests: + type: approval + - j8_cqlsh-dtests-py2-with-vnodes: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py3_vnode: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py38_vnode: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py311_vnode: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh-dtests-py2-no-vnodes: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py3: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py38: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py311: + requires: + - start_j8_cqlsh_tests + - j8_build + - start_j8_cqlsh_tests_offheap: + type: approval + - j8_cqlsh-dtests-py2-offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - j8_cqlsh_dtests_py3_offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - j8_cqlsh_dtests_py38_offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - j8_cqlsh_dtests_py311_offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - start_j11_cqlsh_tests: + type: approval + - j11_cqlsh-dtests-py2-with-vnodes: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py3_vnode: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py38_vnode: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py311_vnode: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh-dtests-py2-no-vnodes: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py3: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py38: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py311: + requires: + - start_j11_cqlsh_tests + - j8_build + - start_j11_cqlsh_tests_offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh_tests_offheap + - j8_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh_tests_offheap + - j8_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh_tests_offheap + - j8_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh_tests_offheap + - j8_build + java8_pre-commit_tests: + jobs: + - start_pre-commit_tests: + type: approval + - j8_build: + requires: + - start_pre-commit_tests + - j8_unit_tests: + requires: + - j8_build + - j8_jvm_dtests: + requires: + - j8_build + - j8_cqlshlib_tests: + requires: + - j8_build + - j11_unit_tests: + requires: + - j8_build + - start_utests_long: + type: approval + - j8_utests_long: + requires: + - start_utests_long + - j8_build + - j11_utests_long: + requires: + - start_utests_long + - j8_build + - start_utests_cdc: + type: approval + - j8_utests_cdc: + requires: + - start_utests_cdc + - j8_build + - j11_utests_cdc: + requires: + - start_utests_cdc + - j8_build + - start_utests_compression: + type: approval + - j8_utests_compression: + requires: + - start_utests_compression + - j8_build + - j11_utests_compression: + requires: + - start_utests_compression + - j8_build + - start_utests_stress: + type: approval + - j8_utests_stress: + requires: + - start_utests_stress + - j8_build + - j11_utests_stress: + requires: + - start_utests_stress + - j8_build + - start_utests_fqltool: + type: approval + - j8_utests_fqltool: + requires: + - start_utests_fqltool + - j8_build + - j11_utests_fqltool: + requires: + - start_utests_fqltool + - j8_build + - start_utests_system_keyspace_directory: + type: approval + - j8_utests_system_keyspace_directory: + requires: + - j8_build + - j11_utests_system_keyspace_directory: + requires: + - start_utests_system_keyspace_directory + - j8_build + - start_jvm_upgrade_dtests: + type: approval + - j8_dtest_jars_build: + requires: + - j8_build + - start_jvm_upgrade_dtests + - j8_jvm_upgrade_dtests: + requires: + - j8_dtest_jars_build + - j8_dtests: + requires: + - j8_build + - j8_dtests_vnode: + requires: + - j8_build + - start_j8_dtests_offheap: + type: approval + - j8_dtests_offheap: + requires: + - start_j8_dtests_offheap + - j8_build + - j11_dtests: + requires: + - j8_build + - j11_dtests_vnode: + requires: + - j8_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j8_build + - start_j8_dtests_large: + type: approval + - j8_dtests_large: + requires: + - start_j8_dtests_large + - j8_build + - j8_dtests_large_vnode: + requires: + - start_j8_dtests_large + - j8_build + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j8_build + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large + - j8_build + - start_upgrade_tests: + type: approval + - j8_upgrade_dtests: + requires: + - j8_build + - start_upgrade_tests + - j8_cqlsh-dtests-py2-with-vnodes: + requires: + - j8_build + - j8_cqlsh_dtests_py3_vnode: + requires: + - j8_build + - j8_cqlsh_dtests_py38_vnode: + requires: + - j8_build + - j8_cqlsh_dtests_py311_vnode: + requires: + - j8_build + - j8_cqlsh-dtests-py2-no-vnodes: + requires: + - j8_build + - j8_cqlsh_dtests_py3: + requires: + - j8_build + - j8_cqlsh_dtests_py38: + requires: + - j8_build + - j8_cqlsh_dtests_py311: + requires: + - j8_build + - start_j8_cqlsh_dtests_offheap: + type: approval + - j8_cqlsh-dtests-py2-offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j8_cqlsh_dtests_py3_offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j8_cqlsh_dtests_py38_offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j8_cqlsh_dtests_py311_offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j11_cqlsh-dtests-py2-with-vnodes: + requires: + - j8_build + - j11_cqlsh_dtests_py3_vnode: + requires: + - j8_build + - j11_cqlsh_dtests_py38_vnode: + requires: + - j8_build + - j11_cqlsh_dtests_py311_vnode: + requires: + - j8_build + - j11_cqlsh-dtests-py2-no-vnodes: + requires: + - j8_build + - j11_cqlsh_dtests_py3: + requires: + - j8_build + - j11_cqlsh_dtests_py38: + requires: + - j8_build + - j11_cqlsh_dtests_py311: + requires: + - j8_build + - start_j11_cqlsh-dtests-offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j8_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j8_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j8_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j8_build + java11_separate_tests: + jobs: + - start_j11_build: + type: approval + - j11_build: + requires: + - start_j11_build + - start_j11_unit_tests: + type: approval + - j11_unit_tests: + requires: + - start_j11_unit_tests + - j11_build + - start_j11_jvm_dtests: + type: approval + - j11_jvm_dtests: + requires: + - start_j11_jvm_dtests + - j11_build + - start_j11_cqlshlib_tests: + type: approval + - j11_cqlshlib_tests: + requires: + - start_j11_cqlshlib_tests + - j11_build + - start_j11_dtests: + type: approval + - j11_dtests: + requires: + - start_j11_dtests + - j11_build + - start_j11_dtests_vnode: + type: approval + - j11_dtests_vnode: + requires: + - start_j11_dtests_vnode + - j11_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j11_build + - start_j11_cqlsh_tests: + type: approval + - j11_cqlsh-dtests-py2-with-vnodes: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py3_vnode: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py38_vnode: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py311_vnode: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh-dtests-py2-no-vnodes: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py3: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py38: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py311: + requires: + - start_j11_cqlsh_tests + - j11_build + - start_j11_cqlsh-dtests-offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j11_build + - start_j11_dtests_large_vnode: + type: approval + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large_vnode + - j11_build + - start_j11_utests_long: + type: approval + - j11_utests_long: + requires: + - start_j11_utests_long + - j11_build + - start_j11_utests_cdc: + type: approval + - j11_utests_cdc: + requires: + - start_j11_utests_cdc + - j11_build + - start_j11_utests_compression: + type: approval + - j11_utests_compression: + requires: + - start_j11_utests_compression + - j11_build + - start_j11_utests_stress: + type: approval + - j11_utests_stress: + requires: + - start_j11_utests_stress + - j11_build + - start_j11_utests_fqltool: + type: approval + - j11_utests_fqltool: + requires: + - start_j11_utests_fqltool + - j11_build + - start_j11_utests_system_keyspace_directory: + type: approval + - j11_utests_system_keyspace_directory: + requires: + - start_j11_utests_system_keyspace_directory + - j11_build + java11_pre-commit_tests: + jobs: + - start_pre-commit_tests: + type: approval + - j11_build: + requires: + - start_pre-commit_tests + - j11_unit_tests: + requires: + - j11_build + - j11_jvm_dtests: + requires: + - j11_build + - j11_cqlshlib_tests: + requires: + - j11_build + - j11_jvm_dtests: + requires: + - j11_build + - j11_cqlshlib_tests: + requires: + - j11_build + - j11_dtests: + requires: + - j11_build + - j11_dtests_vnode: + requires: + - j11_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j11_build + - j11_cqlsh-dtests-py2-with-vnodes: + requires: + - j11_build + - j11_cqlsh_dtests_py3_vnode: + requires: + - j11_build + - j11_cqlsh_dtests_py38_vnode: + requires: + - j11_build + - j11_cqlsh_dtests_py311_vnode: + requires: + - j11_build + - j11_cqlsh-dtests-py2-no-vnodes: + requires: + - j11_build + - j11_cqlsh_dtests_py3: + requires: + - j11_build + - j11_cqlsh_dtests_py38: + requires: + - j11_build + - j11_cqlsh_dtests_py311: + requires: + - j11_build + - start_j11_cqlsh-dtests-offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j11_build + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large + - j11_build + - start_utests_long: + type: approval + - j11_utests_long: + requires: + - start_utests_long + - j11_build + - start_utests_cdc: + type: approval + - j11_utests_cdc: + requires: + - start_utests_cdc + - j11_build + - start_utests_compression: + type: approval + - j11_utests_compression: + requires: + - start_utests_compression + - j11_build + - start_utests_stress: + type: approval + - j11_utests_stress: + requires: + - start_utests_stress + - j11_build + - start_utests_fqltool: + type: approval + - j11_utests_fqltool: + requires: + - start_utests_fqltool + - j11_build + - start_utests_system_keyspace_directory: + type: approval + - j11_utests_system_keyspace_directory: + requires: + - start_utests_system_keyspace_directory + - j11_build diff --git a/.circleci/config.yml.HIGHRES b/.circleci/config.yml.HIGHRES deleted file mode 100644 index 3b71be7b4bff..000000000000 --- a/.circleci/config.yml.HIGHRES +++ /dev/null @@ -1,4330 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -version: 2 -jobs: - j8_jvm_upgrade_dtests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 2 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine distributed Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - repeated_jvm_upgrade_dtest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run repeated JUnit test - no_output_timeout: 15m - command: | - if [ "${REPEATED_JVM_UPGRADE_DTEST_CLASS}" == "" ]; then - echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" == "" ]; then - echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" -le 0 ]; then - echo "Repeated utest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_JVM_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running test-jvm-dtest-some ${REPEATED_JVM_UPGRADE_DTEST_CLASS} ${REPEATED_JVM_UPGRADE_DTEST_METHODS} ${REPEATED_JVM_UPGRADE_DTEST_COUNT} times" - - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - - target=test-jvm-dtest-some - class_path=${REPEATED_JVM_UPGRADE_DTEST_CLASS} - class_name="${class_path##*.}" - - # Prepare the -Dtest.name argument. - # It can be the fully qualified class name or the short class name, depending on the target. - if [[ $target == "test" || \ - $target == "test-cdc" || \ - $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then - name="-Dtest.name=$class_name" - else - name="-Dtest.name=$class_path" - fi - - # Prepare the -Dtest.methods argument, which is optional - if [ "${REPEATED_JVM_UPGRADE_DTEST_METHODS}" == "" ]; then - methods="" - else - methods="-Dtest.methods=${REPEATED_JVM_UPGRADE_DTEST_METHODS}" - fi - - # Run the test target as many times as requested collecting the exit code, - # stopping the iteration only if stop_on_failure is set. - exit_code="$?" - for i in $(seq -w 1 $count); do - - echo "Running test iteration $i of $count" - - # run the test - status="passes" - if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then - status="fails" - exit_code=1 - fi - - # move the stdout output file - dest=/tmp/results/repeated_utest/stdout/${status}/${i} - mkdir -p $dest - mv stdout.txt $dest/test-jvm-dtest-some-${REPEATED_JVM_UPGRADE_DTEST_CLASS}.txt - - # move the XML output files - source=build/test/output - dest=/tmp/results/repeated_utest/output/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # move the log files - source=build/test/logs - dest=/tmp/results/repeated_utest/logs/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # maybe stop iterations on test failure - if [[ ${REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then - break - fi - done - - (exit ${exit_code}) - fi - fi - - store_test_results: - path: /tmp/results/repeated_utest/output - - store_artifacts: - path: /tmp/results/repeated_utest/stdout - destination: stdout - - store_artifacts: - path: /tmp/results/repeated_utest/output - destination: junitxml - - store_artifacts: - path: /tmp/results/repeated_utest/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py2-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - - run: - name: Run dtests (j8_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_unit_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - repeated_upgrade_dtest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Run repeated Python dtest - no_output_timeout: 15m - command: | - if [ "${REPEATED_UPGRADE_DTEST_NAME}" == "" ]; then - echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" == "" ]; then - echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" -le 0 ]; then - echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_UPGRADE_DTEST_NAME} $count times" - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - - stop_on_failure_arg="" - if ${REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE}; then - stop_on_failure_arg="-x" - fi - - vnodes_args="" - if false; then - vnodes_args="--use-vnodes --num-tokens=16" - fi - - upgrade_arg="" - if true; then - upgrade_arg="--execute-upgrade-tests" - fi - - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_UPGRADE_DTEST_NAME} | tee /tmp/dtest/stdout.txt - fi - fi - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py38-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_cqlsh-dtests-py3-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlsh-dtests-py3-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlsh-dtests-py38-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_cqlsh-dtests-py3-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - - run: - name: Run dtests (j8_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py2-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_repeated_dtest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Run repeated Python dtest - no_output_timeout: 15m - command: | - if [ "${REPEATED_DTEST_NAME}" == "" ]; then - echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" == "" ]; then - echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then - echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_DTEST_NAME} $count times" - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - - stop_on_failure_arg="" - if ${REPEATED_DTEST_STOP_ON_FAILURE}; then - stop_on_failure_arg="-x" - fi - - vnodes_args="" - if ${REPEATED_DTEST_VNODES}; then - vnodes_args="--use-vnodes --num-tokens=16" - fi - - upgrade_arg="" - if false; then - upgrade_arg="--execute-upgrade-tests" - fi - - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt - fi - fi - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_repeated_dtest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Run repeated Python dtest - no_output_timeout: 15m - command: | - if [ "${REPEATED_DTEST_NAME}" == "" ]; then - echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" == "" ]; then - echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then - echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_DTEST_NAME} $count times" - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - - stop_on_failure_arg="" - if ${REPEATED_DTEST_STOP_ON_FAILURE}; then - stop_on_failure_arg="-x" - fi - - vnodes_args="" - if ${REPEATED_DTEST_VNODES}; then - vnodes_args="--use-vnodes --num-tokens=16" - fi - - upgrade_arg="" - if false; then - upgrade_arg="--execute-upgrade-tests" - fi - - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt - fi - fi - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_cqlsh-dtests-py2-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_dtests-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_dtests-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_without_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_upgradetests-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_upgradetests_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_upgradetests_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw /tmp/all_dtest_tests_j8_upgradetests_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw > /tmp/all_dtest_tests_j8_upgradetests_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_upgradetests_without_vnodes > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_upgradetests_without_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_upgradetests_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_upgradetests_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_upgradetests_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_stress: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run Unit Tests (stress-test) - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - ant stress-test - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_unit_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_jvm_dtests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 5 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine distributed Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_build: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra Repository (via git) - command: | - git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra - - run: - name: Build Cassandra - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant clean realclean jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - no_output_timeout: 15m - - run: - name: Run eclipse-warnings - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - ant eclipse-warnings - - persist_to_workspace: - root: /home/cassandra - paths: - - cassandra - - .m2 - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlsh-dtests-py2-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlshlib_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run cqlshlib Unit Tests - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra/pylib - ./cassandra-cqlsh-tests.sh .. - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/pylib - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_repeated_utest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run repeated JUnit test - no_output_timeout: 15m - command: | - if [ "${REPEATED_UTEST_CLASS}" == "" ]; then - echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" == "" ]; then - echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" -le 0 ]; then - echo "Repeated utest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_UTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_UTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_UTEST_TARGET} ${REPEATED_UTEST_CLASS} ${REPEATED_UTEST_METHODS} ${REPEATED_UTEST_COUNT} times" - - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - - target=${REPEATED_UTEST_TARGET} - class_path=${REPEATED_UTEST_CLASS} - class_name="${class_path##*.}" - - # Prepare the -Dtest.name argument. - # It can be the fully qualified class name or the short class name, depending on the target. - if [[ $target == "test" || \ - $target == "test-cdc" || \ - $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then - name="-Dtest.name=$class_name" - else - name="-Dtest.name=$class_path" - fi - - # Prepare the -Dtest.methods argument, which is optional - if [ "${REPEATED_UTEST_METHODS}" == "" ]; then - methods="" - else - methods="-Dtest.methods=${REPEATED_UTEST_METHODS}" - fi - - # Run the test target as many times as requested collecting the exit code, - # stopping the iteration only if stop_on_failure is set. - exit_code="$?" - for i in $(seq -w 1 $count); do - - echo "Running test iteration $i of $count" - - # run the test - status="passes" - if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then - status="fails" - exit_code=1 - fi - - # move the stdout output file - dest=/tmp/results/repeated_utest/stdout/${status}/${i} - mkdir -p $dest - mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt - - # move the XML output files - source=build/test/output - dest=/tmp/results/repeated_utest/output/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # move the log files - source=build/test/logs - dest=/tmp/results/repeated_utest/logs/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # maybe stop iterations on test failure - if [[ ${REPEATED_UTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then - break - fi - done - - (exit ${exit_code}) - fi - fi - - store_test_results: - path: /tmp/results/repeated_utest/output - - store_artifacts: - path: /tmp/results/repeated_utest/stdout - destination: stdout - - store_artifacts: - path: /tmp/results/repeated_utest/output - destination: junitxml - - store_artifacts: - path: /tmp/results/repeated_utest/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_dtests-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - - run: - name: Run dtests (j8_with_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_cqlsh-dtests-py38-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_jvm_dtests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 5 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine distributed Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_build: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra Repository (via git) - command: | - git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra - - run: - name: Build Cassandra - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant clean realclean jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - no_output_timeout: 15m - - run: - name: Run eclipse-warnings - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - ant eclipse-warnings - - persist_to_workspace: - root: /home/cassandra - paths: - - cassandra - - .m2 - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py3-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py38-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - - run: - name: Run dtests (j8_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_long: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run Unit Tests (long-test) - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - ant long-test - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_system_keyspace_directory: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist-system-keyspace-directory) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist-system-keyspace-directory -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlshlib_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run cqlshlib Unit Tests - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra/pylib - ./cassandra-cqlsh-tests.sh .. - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/pylib - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_fqltool: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run Unit Tests (fqltool-test) - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - ant fqltool-test - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_dtests-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - utests_compression: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist-compression) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist-compression -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_repeated_utest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run repeated JUnit test - no_output_timeout: 15m - command: | - if [ "${REPEATED_UTEST_CLASS}" == "" ]; then - echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" == "" ]; then - echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" -le 0 ]; then - echo "Repeated utest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_UTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_UTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_UTEST_TARGET} ${REPEATED_UTEST_CLASS} ${REPEATED_UTEST_METHODS} ${REPEATED_UTEST_COUNT} times" - - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - - target=${REPEATED_UTEST_TARGET} - class_path=${REPEATED_UTEST_CLASS} - class_name="${class_path##*.}" - - # Prepare the -Dtest.name argument. - # It can be the fully qualified class name or the short class name, depending on the target. - if [[ $target == "test" || \ - $target == "test-cdc" || \ - $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then - name="-Dtest.name=$class_name" - else - name="-Dtest.name=$class_path" - fi - - # Prepare the -Dtest.methods argument, which is optional - if [ "${REPEATED_UTEST_METHODS}" == "" ]; then - methods="" - else - methods="-Dtest.methods=${REPEATED_UTEST_METHODS}" - fi - - # Run the test target as many times as requested collecting the exit code, - # stopping the iteration only if stop_on_failure is set. - exit_code="$?" - for i in $(seq -w 1 $count); do - - echo "Running test iteration $i of $count" - - # run the test - status="passes" - if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then - status="fails" - exit_code=1 - fi - - # move the stdout output file - dest=/tmp/results/repeated_utest/stdout/${status}/${i} - mkdir -p $dest - mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt - - # move the XML output files - source=build/test/output - dest=/tmp/results/repeated_utest/output/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # move the log files - source=build/test/logs - dest=/tmp/results/repeated_utest/logs/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # maybe stop iterations on test failure - if [[ ${REPEATED_UTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then - break - fi - done - - (exit ${exit_code}) - fi - fi - - store_test_results: - path: /tmp/results/repeated_utest/output - - store_artifacts: - path: /tmp/results/repeated_utest/stdout - destination: stdout - - store_artifacts: - path: /tmp/results/repeated_utest/output - destination: junitxml - - store_artifacts: - path: /tmp/results/repeated_utest/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_dtest_jars_build: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Build Cassandra DTest jars - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - mkdir ~/dtest_jars - git remote add apache https://github.com/apache/cassandra.git - for branch in cassandra-2.2 cassandra-3.0 cassandra-3.11 cassandra-4.0 trunk; do - # check out the correct cassandra version: - git remote set-branches --add apache '$branch' - git fetch --depth 1 apache $branch - git checkout $branch - git clean -fd - # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - cp build/dtest*.jar ~/dtest_jars - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - done - # and build the dtest-jar for the branch under test - ${ANT_HOME}/bin/ant realclean - git checkout origin/$CIRCLE_BRANCH - git clean -fd - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - cp build/dtest*.jar ~/dtest_jars - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - ls -l ~/dtest_jars - no_output_timeout: 15m - - persist_to_workspace: - root: /home/cassandra - paths: - - dtest_jars - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 -workflows: - version: 2 - java8_separate_tests: - jobs: - - start_j8_build: - type: approval - - j8_build: - requires: - - start_j8_build - - start_j8_unit_tests: - type: approval - - j8_unit_tests: - requires: - - start_j8_unit_tests - - j8_build - - start_j8_jvm_dtests: - type: approval - - j8_jvm_dtests: - requires: - - start_j8_jvm_dtests - - j8_build - - start_j8_cqlshlib_tests: - type: approval - - j8_cqlshlib_tests: - requires: - - start_j8_cqlshlib_tests - - j8_build - - start_j11_unit_tests: - type: approval - - j11_unit_tests: - requires: - - start_j11_unit_tests - - j8_build - - start_utests_long: - type: approval - - utests_long: - requires: - - start_utests_long - - j8_build - - start_utests_compression: - type: approval - - utests_compression: - requires: - - start_utests_compression - - j8_build - - start_utests_stress: - type: approval - - utests_stress: - requires: - - start_utests_stress - - j8_build - - start_utests_fqltool: - type: approval - - utests_fqltool: - requires: - - start_utests_fqltool - - j8_build - - start_utests_system_keyspace_directory: - type: approval - - utests_system_keyspace_directory: - requires: - - start_utests_system_keyspace_directory - - j8_build - - start_j8_dtest_jars_build: - type: approval - - j8_dtest_jars_build: - requires: - - j8_build - - start_j8_dtest_jars_build - - start_jvm_upgrade_dtest: - type: approval - - j8_jvm_upgrade_dtests: - requires: - - start_jvm_upgrade_dtest - - j8_dtest_jars_build - - start_j8_dtests: - type: approval - - j8_dtests-with-vnodes: - requires: - - start_j8_dtests - - j8_build - - j8_dtests-no-vnodes: - requires: - - start_j8_dtests - - j8_build - - start_j11_dtests: - type: approval - - j11_dtests-with-vnodes: - requires: - - start_j11_dtests - - j8_build - - j11_dtests-no-vnodes: - requires: - - start_j11_dtests - - j8_build - - start_upgrade_tests: - type: approval - - j8_upgradetests-no-vnodes: - requires: - - start_upgrade_tests - - j8_build - - start_j8_cqlsh_tests: - type: approval - - j8_cqlsh-dtests-py2-with-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py3-with-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py38-with-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py2-no-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py3-no-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py38-no-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - start_j11_cqlsh_tests: - type: approval - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - start_j8_repeated_utest: - type: approval - - j8_repeated_utest: - requires: - - start_j8_repeated_utest - - j8_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j8_build - - start_j8_repeated_dtest: - type: approval - - j8_repeated_dtest: - requires: - - start_j8_repeated_dtest - - j8_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j8_build - - start_repeated_upgrade_dtest: - type: approval - - repeated_upgrade_dtest: - requires: - - start_repeated_upgrade_dtest - - j8_build - - start_repeated_jvm_upgrade_dtest: - type: approval - - repeated_jvm_upgrade_dtest: - requires: - - start_repeated_jvm_upgrade_dtest - - j8_dtest_jars_build - java8_pre-commit_tests: - jobs: - - start_pre-commit_tests: - type: approval - - j8_build: - requires: - - start_pre-commit_tests - - j8_unit_tests: - requires: - - j8_build - - j8_jvm_dtests: - requires: - - j8_build - - j8_cqlshlib_tests: - requires: - - j8_build - - j11_unit_tests: - requires: - - j8_build - - start_utests_long: - type: approval - - utests_long: - requires: - - start_utests_long - - j8_build - - start_utests_compression: - type: approval - - utests_compression: - requires: - - start_utests_compression - - j8_build - - start_utests_stress: - type: approval - - utests_stress: - requires: - - start_utests_stress - - j8_build - - start_utests_fqltool: - type: approval - - utests_fqltool: - requires: - - start_utests_fqltool - - j8_build - - utests_system_keyspace_directory: - requires: - - j8_build - - start_j8_dtest_jars_build: - type: approval - - j8_dtest_jars_build: - requires: - - j8_build - - start_j8_dtest_jars_build - - start_jvm_upgrade_dtest: - type: approval - - j8_jvm_upgrade_dtests: - requires: - - start_jvm_upgrade_dtest - - j8_dtest_jars_build - - j8_dtests-with-vnodes: - requires: - - j8_build - - j8_dtests-no-vnodes: - requires: - - j8_build - - j11_dtests-with-vnodes: - requires: - - j8_build - - j11_dtests-no-vnodes: - requires: - - j8_build - - start_upgrade_tests: - type: approval - - j8_upgradetests-no-vnodes: - requires: - - j8_build - - start_upgrade_tests - - j8_cqlsh-dtests-py2-with-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py3-with-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py38-with-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py2-no-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py3-no-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py38-no-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - j8_build - - start_j8_repeated_utest: - type: approval - - j8_repeated_utest: - requires: - - start_j8_repeated_utest - - j8_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j8_build - - start_j8_repeated_dtest: - type: approval - - j8_repeated_dtest: - requires: - - start_j8_repeated_dtest - - j8_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j8_build - - start_repeated_upgrade_dtest: - type: approval - - repeated_upgrade_dtest: - requires: - - start_repeated_upgrade_dtest - - j8_build - - start_repeated_jvm_upgrade_dtest: - type: approval - - repeated_jvm_upgrade_dtest: - requires: - - start_repeated_jvm_upgrade_dtest - - j8_dtest_jars_build - java11_separate_tests: - jobs: - - start_j11_build: - type: approval - - j11_build: - requires: - - start_j11_build - - start_j11_unit_tests: - type: approval - - j11_unit_tests: - requires: - - start_j11_unit_tests - - j11_build - - start_j11_jvm_dtests: - type: approval - - j11_jvm_dtests: - requires: - - start_j11_jvm_dtests - - j11_build - - start_j11_cqlshlib_tests: - type: approval - - j11_cqlshlib_tests: - requires: - - start_j11_cqlshlib_tests - - j11_build - - start_j11_dtests: - type: approval - - j11_dtests-with-vnodes: - requires: - - start_j11_dtests - - j11_build - - j11_dtests-no-vnodes: - requires: - - start_j11_dtests - - j11_build - - start_j11_cqlsh_tests: - type: approval - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j11_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j11_build - java11_pre-commit_tests: - jobs: - - start_pre-commit_tests: - type: approval - - j11_build: - requires: - - start_pre-commit_tests - - j11_unit_tests: - requires: - - j11_build - - j11_jvm_dtests: - requires: - - j11_build - - j11_cqlshlib_tests: - requires: - - j11_build - - j11_jvm_dtests: - requires: - - j11_build - - j11_cqlshlib_tests: - requires: - - j11_build - - j11_dtests-with-vnodes: - requires: - - j11_build - - j11_dtests-no-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - j11_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j11_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j11_build diff --git a/.circleci/config.yml.LOWRES b/.circleci/config.yml.LOWRES deleted file mode 100644 index 6e941ea5ea52..000000000000 --- a/.circleci/config.yml.LOWRES +++ /dev/null @@ -1,4330 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -version: 2 -jobs: - j8_jvm_upgrade_dtests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine distributed Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - repeated_jvm_upgrade_dtest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run repeated JUnit test - no_output_timeout: 15m - command: | - if [ "${REPEATED_JVM_UPGRADE_DTEST_CLASS}" == "" ]; then - echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" == "" ]; then - echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" -le 0 ]; then - echo "Repeated utest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_JVM_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running test-jvm-dtest-some ${REPEATED_JVM_UPGRADE_DTEST_CLASS} ${REPEATED_JVM_UPGRADE_DTEST_METHODS} ${REPEATED_JVM_UPGRADE_DTEST_COUNT} times" - - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - - target=test-jvm-dtest-some - class_path=${REPEATED_JVM_UPGRADE_DTEST_CLASS} - class_name="${class_path##*.}" - - # Prepare the -Dtest.name argument. - # It can be the fully qualified class name or the short class name, depending on the target. - if [[ $target == "test" || \ - $target == "test-cdc" || \ - $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then - name="-Dtest.name=$class_name" - else - name="-Dtest.name=$class_path" - fi - - # Prepare the -Dtest.methods argument, which is optional - if [ "${REPEATED_JVM_UPGRADE_DTEST_METHODS}" == "" ]; then - methods="" - else - methods="-Dtest.methods=${REPEATED_JVM_UPGRADE_DTEST_METHODS}" - fi - - # Run the test target as many times as requested collecting the exit code, - # stopping the iteration only if stop_on_failure is set. - exit_code="$?" - for i in $(seq -w 1 $count); do - - echo "Running test iteration $i of $count" - - # run the test - status="passes" - if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then - status="fails" - exit_code=1 - fi - - # move the stdout output file - dest=/tmp/results/repeated_utest/stdout/${status}/${i} - mkdir -p $dest - mv stdout.txt $dest/test-jvm-dtest-some-${REPEATED_JVM_UPGRADE_DTEST_CLASS}.txt - - # move the XML output files - source=build/test/output - dest=/tmp/results/repeated_utest/output/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # move the log files - source=build/test/logs - dest=/tmp/results/repeated_utest/logs/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # maybe stop iterations on test failure - if [[ ${REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then - break - fi - done - - (exit ${exit_code}) - fi - fi - - store_test_results: - path: /tmp/results/repeated_utest/output - - store_artifacts: - path: /tmp/results/repeated_utest/stdout - destination: stdout - - store_artifacts: - path: /tmp/results/repeated_utest/output - destination: junitxml - - store_artifacts: - path: /tmp/results/repeated_utest/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py2-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - - run: - name: Run dtests (j8_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_unit_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - repeated_upgrade_dtest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Run repeated Python dtest - no_output_timeout: 15m - command: | - if [ "${REPEATED_UPGRADE_DTEST_NAME}" == "" ]; then - echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" == "" ]; then - echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" -le 0 ]; then - echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_UPGRADE_DTEST_NAME} $count times" - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - - stop_on_failure_arg="" - if ${REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE}; then - stop_on_failure_arg="-x" - fi - - vnodes_args="" - if false; then - vnodes_args="--use-vnodes --num-tokens=16" - fi - - upgrade_arg="" - if true; then - upgrade_arg="--execute-upgrade-tests" - fi - - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_UPGRADE_DTEST_NAME} | tee /tmp/dtest/stdout.txt - fi - fi - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py38-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_cqlsh-dtests-py3-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlsh-dtests-py3-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlsh-dtests-py38-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_cqlsh-dtests-py3-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - - run: - name: Run dtests (j8_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py2-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_repeated_dtest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Run repeated Python dtest - no_output_timeout: 15m - command: | - if [ "${REPEATED_DTEST_NAME}" == "" ]; then - echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" == "" ]; then - echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then - echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_DTEST_NAME} $count times" - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - - stop_on_failure_arg="" - if ${REPEATED_DTEST_STOP_ON_FAILURE}; then - stop_on_failure_arg="-x" - fi - - vnodes_args="" - if ${REPEATED_DTEST_VNODES}; then - vnodes_args="--use-vnodes --num-tokens=16" - fi - - upgrade_arg="" - if false; then - upgrade_arg="--execute-upgrade-tests" - fi - - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt - fi - fi - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_repeated_dtest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Run repeated Python dtest - no_output_timeout: 15m - command: | - if [ "${REPEATED_DTEST_NAME}" == "" ]; then - echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" == "" ]; then - echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then - echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_DTEST_NAME} $count times" - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - - stop_on_failure_arg="" - if ${REPEATED_DTEST_STOP_ON_FAILURE}; then - stop_on_failure_arg="-x" - fi - - vnodes_args="" - if ${REPEATED_DTEST_VNODES}; then - vnodes_args="--use-vnodes --num-tokens=16" - fi - - upgrade_arg="" - if false; then - upgrade_arg="--execute-upgrade-tests" - fi - - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt - fi - fi - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_cqlsh-dtests-py2-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_dtests-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_dtests-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_without_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_upgradetests-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_upgradetests_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_upgradetests_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw /tmp/all_dtest_tests_j8_upgradetests_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw > /tmp/all_dtest_tests_j8_upgradetests_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_upgradetests_without_vnodes > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_upgradetests_without_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_upgradetests_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_upgradetests_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_upgradetests_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_stress: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run Unit Tests (stress-test) - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - ant stress-test - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_unit_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_jvm_dtests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine distributed Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_build: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra Repository (via git) - command: | - git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra - - run: - name: Build Cassandra - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant clean realclean jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - no_output_timeout: 15m - - run: - name: Run eclipse-warnings - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - ant eclipse-warnings - - persist_to_workspace: - root: /home/cassandra - paths: - - cassandra - - .m2 - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlsh-dtests-py2-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlshlib_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run cqlshlib Unit Tests - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra/pylib - ./cassandra-cqlsh-tests.sh .. - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/pylib - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_repeated_utest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run repeated JUnit test - no_output_timeout: 15m - command: | - if [ "${REPEATED_UTEST_CLASS}" == "" ]; then - echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" == "" ]; then - echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" -le 0 ]; then - echo "Repeated utest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_UTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_UTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_UTEST_TARGET} ${REPEATED_UTEST_CLASS} ${REPEATED_UTEST_METHODS} ${REPEATED_UTEST_COUNT} times" - - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - - target=${REPEATED_UTEST_TARGET} - class_path=${REPEATED_UTEST_CLASS} - class_name="${class_path##*.}" - - # Prepare the -Dtest.name argument. - # It can be the fully qualified class name or the short class name, depending on the target. - if [[ $target == "test" || \ - $target == "test-cdc" || \ - $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then - name="-Dtest.name=$class_name" - else - name="-Dtest.name=$class_path" - fi - - # Prepare the -Dtest.methods argument, which is optional - if [ "${REPEATED_UTEST_METHODS}" == "" ]; then - methods="" - else - methods="-Dtest.methods=${REPEATED_UTEST_METHODS}" - fi - - # Run the test target as many times as requested collecting the exit code, - # stopping the iteration only if stop_on_failure is set. - exit_code="$?" - for i in $(seq -w 1 $count); do - - echo "Running test iteration $i of $count" - - # run the test - status="passes" - if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then - status="fails" - exit_code=1 - fi - - # move the stdout output file - dest=/tmp/results/repeated_utest/stdout/${status}/${i} - mkdir -p $dest - mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt - - # move the XML output files - source=build/test/output - dest=/tmp/results/repeated_utest/output/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # move the log files - source=build/test/logs - dest=/tmp/results/repeated_utest/logs/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # maybe stop iterations on test failure - if [[ ${REPEATED_UTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then - break - fi - done - - (exit ${exit_code}) - fi - fi - - store_test_results: - path: /tmp/results/repeated_utest/output - - store_artifacts: - path: /tmp/results/repeated_utest/stdout - destination: stdout - - store_artifacts: - path: /tmp/results/repeated_utest/output - destination: junitxml - - store_artifacts: - path: /tmp/results/repeated_utest/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_dtests-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - - run: - name: Run dtests (j8_with_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_cqlsh-dtests-py38-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_jvm_dtests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine distributed Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_build: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra Repository (via git) - command: | - git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra - - run: - name: Build Cassandra - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant clean realclean jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - no_output_timeout: 15m - - run: - name: Run eclipse-warnings - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - ant eclipse-warnings - - persist_to_workspace: - root: /home/cassandra - paths: - - cassandra - - .m2 - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py3-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py38-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - - run: - name: Run dtests (j8_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_long: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run Unit Tests (long-test) - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - ant long-test - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_system_keyspace_directory: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist-system-keyspace-directory) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist-system-keyspace-directory -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlshlib_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run cqlshlib Unit Tests - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra/pylib - ./cassandra-cqlsh-tests.sh .. - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/pylib - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_fqltool: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run Unit Tests (fqltool-test) - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - ant fqltool-test - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_dtests-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - utests_compression: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist-compression) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist-compression -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_repeated_utest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run repeated JUnit test - no_output_timeout: 15m - command: | - if [ "${REPEATED_UTEST_CLASS}" == "" ]; then - echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" == "" ]; then - echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" -le 0 ]; then - echo "Repeated utest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_UTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_UTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_UTEST_TARGET} ${REPEATED_UTEST_CLASS} ${REPEATED_UTEST_METHODS} ${REPEATED_UTEST_COUNT} times" - - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - - target=${REPEATED_UTEST_TARGET} - class_path=${REPEATED_UTEST_CLASS} - class_name="${class_path##*.}" - - # Prepare the -Dtest.name argument. - # It can be the fully qualified class name or the short class name, depending on the target. - if [[ $target == "test" || \ - $target == "test-cdc" || \ - $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then - name="-Dtest.name=$class_name" - else - name="-Dtest.name=$class_path" - fi - - # Prepare the -Dtest.methods argument, which is optional - if [ "${REPEATED_UTEST_METHODS}" == "" ]; then - methods="" - else - methods="-Dtest.methods=${REPEATED_UTEST_METHODS}" - fi - - # Run the test target as many times as requested collecting the exit code, - # stopping the iteration only if stop_on_failure is set. - exit_code="$?" - for i in $(seq -w 1 $count); do - - echo "Running test iteration $i of $count" - - # run the test - status="passes" - if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then - status="fails" - exit_code=1 - fi - - # move the stdout output file - dest=/tmp/results/repeated_utest/stdout/${status}/${i} - mkdir -p $dest - mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt - - # move the XML output files - source=build/test/output - dest=/tmp/results/repeated_utest/output/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # move the log files - source=build/test/logs - dest=/tmp/results/repeated_utest/logs/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # maybe stop iterations on test failure - if [[ ${REPEATED_UTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then - break - fi - done - - (exit ${exit_code}) - fi - fi - - store_test_results: - path: /tmp/results/repeated_utest/output - - store_artifacts: - path: /tmp/results/repeated_utest/stdout - destination: stdout - - store_artifacts: - path: /tmp/results/repeated_utest/output - destination: junitxml - - store_artifacts: - path: /tmp/results/repeated_utest/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_dtest_jars_build: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Build Cassandra DTest jars - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - mkdir ~/dtest_jars - git remote add apache https://github.com/apache/cassandra.git - for branch in cassandra-2.2 cassandra-3.0 cassandra-3.11 cassandra-4.0 trunk; do - # check out the correct cassandra version: - git remote set-branches --add apache '$branch' - git fetch --depth 1 apache $branch - git checkout $branch - git clean -fd - # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - cp build/dtest*.jar ~/dtest_jars - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - done - # and build the dtest-jar for the branch under test - ${ANT_HOME}/bin/ant realclean - git checkout origin/$CIRCLE_BRANCH - git clean -fd - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - cp build/dtest*.jar ~/dtest_jars - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - ls -l ~/dtest_jars - no_output_timeout: 15m - - persist_to_workspace: - root: /home/cassandra - paths: - - dtest_jars - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 -workflows: - version: 2 - java8_separate_tests: - jobs: - - start_j8_build: - type: approval - - j8_build: - requires: - - start_j8_build - - start_j8_unit_tests: - type: approval - - j8_unit_tests: - requires: - - start_j8_unit_tests - - j8_build - - start_j8_jvm_dtests: - type: approval - - j8_jvm_dtests: - requires: - - start_j8_jvm_dtests - - j8_build - - start_j8_cqlshlib_tests: - type: approval - - j8_cqlshlib_tests: - requires: - - start_j8_cqlshlib_tests - - j8_build - - start_j11_unit_tests: - type: approval - - j11_unit_tests: - requires: - - start_j11_unit_tests - - j8_build - - start_utests_long: - type: approval - - utests_long: - requires: - - start_utests_long - - j8_build - - start_utests_compression: - type: approval - - utests_compression: - requires: - - start_utests_compression - - j8_build - - start_utests_stress: - type: approval - - utests_stress: - requires: - - start_utests_stress - - j8_build - - start_utests_fqltool: - type: approval - - utests_fqltool: - requires: - - start_utests_fqltool - - j8_build - - start_utests_system_keyspace_directory: - type: approval - - utests_system_keyspace_directory: - requires: - - start_utests_system_keyspace_directory - - j8_build - - start_j8_dtest_jars_build: - type: approval - - j8_dtest_jars_build: - requires: - - j8_build - - start_j8_dtest_jars_build - - start_jvm_upgrade_dtest: - type: approval - - j8_jvm_upgrade_dtests: - requires: - - start_jvm_upgrade_dtest - - j8_dtest_jars_build - - start_j8_dtests: - type: approval - - j8_dtests-with-vnodes: - requires: - - start_j8_dtests - - j8_build - - j8_dtests-no-vnodes: - requires: - - start_j8_dtests - - j8_build - - start_j11_dtests: - type: approval - - j11_dtests-with-vnodes: - requires: - - start_j11_dtests - - j8_build - - j11_dtests-no-vnodes: - requires: - - start_j11_dtests - - j8_build - - start_upgrade_tests: - type: approval - - j8_upgradetests-no-vnodes: - requires: - - start_upgrade_tests - - j8_build - - start_j8_cqlsh_tests: - type: approval - - j8_cqlsh-dtests-py2-with-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py3-with-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py38-with-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py2-no-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py3-no-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py38-no-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - start_j11_cqlsh_tests: - type: approval - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - start_j8_repeated_utest: - type: approval - - j8_repeated_utest: - requires: - - start_j8_repeated_utest - - j8_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j8_build - - start_j8_repeated_dtest: - type: approval - - j8_repeated_dtest: - requires: - - start_j8_repeated_dtest - - j8_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j8_build - - start_repeated_upgrade_dtest: - type: approval - - repeated_upgrade_dtest: - requires: - - start_repeated_upgrade_dtest - - j8_build - - start_repeated_jvm_upgrade_dtest: - type: approval - - repeated_jvm_upgrade_dtest: - requires: - - start_repeated_jvm_upgrade_dtest - - j8_dtest_jars_build - java8_pre-commit_tests: - jobs: - - start_pre-commit_tests: - type: approval - - j8_build: - requires: - - start_pre-commit_tests - - j8_unit_tests: - requires: - - j8_build - - j8_jvm_dtests: - requires: - - j8_build - - j8_cqlshlib_tests: - requires: - - j8_build - - j11_unit_tests: - requires: - - j8_build - - start_utests_long: - type: approval - - utests_long: - requires: - - start_utests_long - - j8_build - - start_utests_compression: - type: approval - - utests_compression: - requires: - - start_utests_compression - - j8_build - - start_utests_stress: - type: approval - - utests_stress: - requires: - - start_utests_stress - - j8_build - - start_utests_fqltool: - type: approval - - utests_fqltool: - requires: - - start_utests_fqltool - - j8_build - - utests_system_keyspace_directory: - requires: - - j8_build - - start_j8_dtest_jars_build: - type: approval - - j8_dtest_jars_build: - requires: - - j8_build - - start_j8_dtest_jars_build - - start_jvm_upgrade_dtest: - type: approval - - j8_jvm_upgrade_dtests: - requires: - - start_jvm_upgrade_dtest - - j8_dtest_jars_build - - j8_dtests-with-vnodes: - requires: - - j8_build - - j8_dtests-no-vnodes: - requires: - - j8_build - - j11_dtests-with-vnodes: - requires: - - j8_build - - j11_dtests-no-vnodes: - requires: - - j8_build - - start_upgrade_tests: - type: approval - - j8_upgradetests-no-vnodes: - requires: - - j8_build - - start_upgrade_tests - - j8_cqlsh-dtests-py2-with-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py3-with-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py38-with-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py2-no-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py3-no-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py38-no-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - j8_build - - start_j8_repeated_utest: - type: approval - - j8_repeated_utest: - requires: - - start_j8_repeated_utest - - j8_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j8_build - - start_j8_repeated_dtest: - type: approval - - j8_repeated_dtest: - requires: - - start_j8_repeated_dtest - - j8_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j8_build - - start_repeated_upgrade_dtest: - type: approval - - repeated_upgrade_dtest: - requires: - - start_repeated_upgrade_dtest - - j8_build - - start_repeated_jvm_upgrade_dtest: - type: approval - - repeated_jvm_upgrade_dtest: - requires: - - start_repeated_jvm_upgrade_dtest - - j8_dtest_jars_build - java11_separate_tests: - jobs: - - start_j11_build: - type: approval - - j11_build: - requires: - - start_j11_build - - start_j11_unit_tests: - type: approval - - j11_unit_tests: - requires: - - start_j11_unit_tests - - j11_build - - start_j11_jvm_dtests: - type: approval - - j11_jvm_dtests: - requires: - - start_j11_jvm_dtests - - j11_build - - start_j11_cqlshlib_tests: - type: approval - - j11_cqlshlib_tests: - requires: - - start_j11_cqlshlib_tests - - j11_build - - start_j11_dtests: - type: approval - - j11_dtests-with-vnodes: - requires: - - start_j11_dtests - - j11_build - - j11_dtests-no-vnodes: - requires: - - start_j11_dtests - - j11_build - - start_j11_cqlsh_tests: - type: approval - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j11_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j11_build - java11_pre-commit_tests: - jobs: - - start_pre-commit_tests: - type: approval - - j11_build: - requires: - - start_pre-commit_tests - - j11_unit_tests: - requires: - - j11_build - - j11_jvm_dtests: - requires: - - j11_build - - j11_cqlshlib_tests: - requires: - - j11_build - - j11_jvm_dtests: - requires: - - j11_build - - j11_cqlshlib_tests: - requires: - - j11_build - - j11_dtests-with-vnodes: - requires: - - j11_build - - j11_dtests-no-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - j11_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j11_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j11_build diff --git a/.circleci/config.yml.MIDRES b/.circleci/config.yml.MIDRES deleted file mode 100644 index be925747c993..000000000000 --- a/.circleci/config.yml.MIDRES +++ /dev/null @@ -1,4330 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -version: 2 -jobs: - j8_jvm_upgrade_dtests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 4 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine distributed Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - repeated_jvm_upgrade_dtest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 25 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run repeated JUnit test - no_output_timeout: 15m - command: | - if [ "${REPEATED_JVM_UPGRADE_DTEST_CLASS}" == "" ]; then - echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" == "" ]; then - echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" -le 0 ]; then - echo "Repeated utest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_JVM_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running test-jvm-dtest-some ${REPEATED_JVM_UPGRADE_DTEST_CLASS} ${REPEATED_JVM_UPGRADE_DTEST_METHODS} ${REPEATED_JVM_UPGRADE_DTEST_COUNT} times" - - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - - target=test-jvm-dtest-some - class_path=${REPEATED_JVM_UPGRADE_DTEST_CLASS} - class_name="${class_path##*.}" - - # Prepare the -Dtest.name argument. - # It can be the fully qualified class name or the short class name, depending on the target. - if [[ $target == "test" || \ - $target == "test-cdc" || \ - $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then - name="-Dtest.name=$class_name" - else - name="-Dtest.name=$class_path" - fi - - # Prepare the -Dtest.methods argument, which is optional - if [ "${REPEATED_JVM_UPGRADE_DTEST_METHODS}" == "" ]; then - methods="" - else - methods="-Dtest.methods=${REPEATED_JVM_UPGRADE_DTEST_METHODS}" - fi - - # Run the test target as many times as requested collecting the exit code, - # stopping the iteration only if stop_on_failure is set. - exit_code="$?" - for i in $(seq -w 1 $count); do - - echo "Running test iteration $i of $count" - - # run the test - status="passes" - if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then - status="fails" - exit_code=1 - fi - - # move the stdout output file - dest=/tmp/results/repeated_utest/stdout/${status}/${i} - mkdir -p $dest - mv stdout.txt $dest/test-jvm-dtest-some-${REPEATED_JVM_UPGRADE_DTEST_CLASS}.txt - - # move the XML output files - source=build/test/output - dest=/tmp/results/repeated_utest/output/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # move the log files - source=build/test/logs - dest=/tmp/results/repeated_utest/logs/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # maybe stop iterations on test failure - if [[ ${REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then - break - fi - done - - (exit ${exit_code}) - fi - fi - - store_test_results: - path: /tmp/results/repeated_utest/output - - store_artifacts: - path: /tmp/results/repeated_utest/stdout - destination: stdout - - store_artifacts: - path: /tmp/results/repeated_utest/output - destination: junitxml - - store_artifacts: - path: /tmp/results/repeated_utest/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py2-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - - run: - name: Run dtests (j8_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_unit_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 25 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - repeated_upgrade_dtest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 25 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Run repeated Python dtest - no_output_timeout: 15m - command: | - if [ "${REPEATED_UPGRADE_DTEST_NAME}" == "" ]; then - echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" == "" ]; then - echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" -le 0 ]; then - echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_UPGRADE_DTEST_NAME} $count times" - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - - stop_on_failure_arg="" - if ${REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE}; then - stop_on_failure_arg="-x" - fi - - vnodes_args="" - if false; then - vnodes_args="--use-vnodes --num-tokens=16" - fi - - upgrade_arg="" - if true; then - upgrade_arg="--execute-upgrade-tests" - fi - - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_UPGRADE_DTEST_NAME} | tee /tmp/dtest/stdout.txt - fi - fi - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py38-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_cqlsh-dtests-py3-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlsh-dtests-py3-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlsh-dtests-py38-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_cqlsh-dtests-py3-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - - run: - name: Run dtests (j8_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py2-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_repeated_dtest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 25 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Run repeated Python dtest - no_output_timeout: 15m - command: | - if [ "${REPEATED_DTEST_NAME}" == "" ]; then - echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" == "" ]; then - echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then - echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_DTEST_NAME} $count times" - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - - stop_on_failure_arg="" - if ${REPEATED_DTEST_STOP_ON_FAILURE}; then - stop_on_failure_arg="-x" - fi - - vnodes_args="" - if ${REPEATED_DTEST_VNODES}; then - vnodes_args="--use-vnodes --num-tokens=16" - fi - - upgrade_arg="" - if false; then - upgrade_arg="--execute-upgrade-tests" - fi - - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt - fi - fi - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_repeated_dtest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 25 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Run repeated Python dtest - no_output_timeout: 15m - command: | - if [ "${REPEATED_DTEST_NAME}" == "" ]; then - echo "Repeated dtest name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" == "" ]; then - echo "Repeated dtest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then - echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_DTEST_NAME} $count times" - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - - stop_on_failure_arg="" - if ${REPEATED_DTEST_STOP_ON_FAILURE}; then - stop_on_failure_arg="-x" - fi - - vnodes_args="" - if ${REPEATED_DTEST_VNODES}; then - vnodes_args="--use-vnodes --num-tokens=16" - fi - - upgrade_arg="" - if false; then - upgrade_arg="--execute-upgrade-tests" - fi - - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt - fi - fi - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_cqlsh-dtests-py2-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_dtests-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" - - run: - name: Run dtests (j11_with_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_dtests-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_without_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_upgradetests-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: xlarge - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 100 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_upgradetests_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_upgradetests_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw /tmp/all_dtest_tests_j8_upgradetests_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw > /tmp/all_dtest_tests_j8_upgradetests_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_upgradetests_without_vnodes > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_upgradetests_without_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_upgradetests_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_upgradetests_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_upgradetests_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_stress: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run Unit Tests (stress-test) - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - ant stress-test - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_unit_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 25 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_jvm_dtests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 10 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine distributed Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_build: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra Repository (via git) - command: | - git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra - - run: - name: Build Cassandra - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant clean realclean jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - no_output_timeout: 15m - - run: - name: Run eclipse-warnings - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - ant eclipse-warnings - - persist_to_workspace: - root: /home/cassandra - paths: - - cassandra - - .m2 - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlsh-dtests-py2-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then - export CQLSH_PYTHON=/usr/bin/python2.7 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_cqlshlib_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run cqlshlib Unit Tests - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra/pylib - ./cassandra-cqlsh-tests.sh .. - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/pylib - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j11_repeated_utest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 25 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run repeated JUnit test - no_output_timeout: 15m - command: | - if [ "${REPEATED_UTEST_CLASS}" == "" ]; then - echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" == "" ]; then - echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" -le 0 ]; then - echo "Repeated utest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_UTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_UTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_UTEST_TARGET} ${REPEATED_UTEST_CLASS} ${REPEATED_UTEST_METHODS} ${REPEATED_UTEST_COUNT} times" - - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - - target=${REPEATED_UTEST_TARGET} - class_path=${REPEATED_UTEST_CLASS} - class_name="${class_path##*.}" - - # Prepare the -Dtest.name argument. - # It can be the fully qualified class name or the short class name, depending on the target. - if [[ $target == "test" || \ - $target == "test-cdc" || \ - $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then - name="-Dtest.name=$class_name" - else - name="-Dtest.name=$class_path" - fi - - # Prepare the -Dtest.methods argument, which is optional - if [ "${REPEATED_UTEST_METHODS}" == "" ]; then - methods="" - else - methods="-Dtest.methods=${REPEATED_UTEST_METHODS}" - fi - - # Run the test target as many times as requested collecting the exit code, - # stopping the iteration only if stop_on_failure is set. - exit_code="$?" - for i in $(seq -w 1 $count); do - - echo "Running test iteration $i of $count" - - # run the test - status="passes" - if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then - status="fails" - exit_code=1 - fi - - # move the stdout output file - dest=/tmp/results/repeated_utest/stdout/${status}/${i} - mkdir -p $dest - mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt - - # move the XML output files - source=build/test/output - dest=/tmp/results/repeated_utest/output/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # move the log files - source=build/test/logs - dest=/tmp/results/repeated_utest/logs/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # maybe stop iterations on test failure - if [[ ${REPEATED_UTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then - break - fi - done - - (exit ${exit_code}) - fi - fi - - store_test_results: - path: /tmp/results/repeated_utest/output - - store_artifacts: - path: /tmp/results/repeated_utest/stdout - destination: stdout - - store_artifacts: - path: /tmp/results/repeated_utest/output - destination: junitxml - - store_artifacts: - path: /tmp/results/repeated_utest/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_dtests-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - - run: - name: Run dtests (j8_with_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_cqlsh-dtests-py38-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - j8_jvm_dtests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 10 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine distributed Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_build: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra Repository (via git) - command: | - git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra - - run: - name: Build Cassandra - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant clean realclean jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - no_output_timeout: 15m - - run: - name: Run eclipse-warnings - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - ant eclipse-warnings - - persist_to_workspace: - root: /home/cassandra - paths: - - cassandra - - .m2 - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py3-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" - - run: - name: Run dtests (j8_without_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt - - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then - export CQLSH_PYTHON=/usr/bin/python3.6 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlsh-dtests-py38-with-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j8_with_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" - - run: - name: Run dtests (j8_with_vnodes) - no_output_timeout: 15m - command: | - echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" - cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt - - source ~/env3.8/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then - export CQLSH_PYTHON=/usr/bin/python3.8 - fi - - java -version - cd ~/cassandra-dtest - mkdir -p /tmp/dtest - - echo "env: $(env)" - echo "** done env" - mkdir -p /tmp/results/dtests - # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee - export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` - set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j8_with_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j8_with_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_long: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run Unit Tests (long-test) - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - ant long-test - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_system_keyspace_directory: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 25 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist-system-keyspace-directory) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist-system-keyspace-directory -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_cqlshlib_tests: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run cqlshlib Unit Tests - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra/pylib - ./cassandra-cqlsh-tests.sh .. - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/pylib - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - utests_fqltool: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Run Unit Tests (fqltool-test) - command: | - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - ant fqltool-test - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j11_dtests-no-vnodes: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11:latest - resource_class: large - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 50 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Clone Cassandra dtest Repository (via git) - command: | - git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest - - run: - name: Configure virtualenv and python Dependencies - command: | - # note, this should be super quick as all dependencies should be pre-installed in the docker image - # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated - # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and - # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) - source ~/env3.6/bin/activate - export PATH=$JAVA_HOME/bin:$PATH - pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt - pip3 uninstall -y cqlsh - pip3 freeze - - run: - name: Determine Tests to Run (j11_without_vnodes) - no_output_timeout: 5m - command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" - - run: - name: Run dtests (j11_without_vnodes) - no_output_timeout: 15m - command: "echo \"cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`\nset -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\n" - - store_test_results: - path: /tmp/results - - store_artifacts: - path: /tmp/dtest - destination: dtest_j11_without_vnodes - - store_artifacts: - path: ~/cassandra-dtest/logs - destination: dtest_j11_without_vnodes_logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - CASSANDRA_USE_JDK11: true - utests_compression: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 25 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Determine unit Tests to Run - command: | - # reminder: this code (along with all the steps) is independently executed on every circle container - # so the goal here is to get the circleci script to return the tests *this* container will run - # which we do via the `circleci` cli tool. - - rm -fr ~/cassandra-dtest/upgrade_tests - echo "***java tests***" - - # get all of our unit test filenames - set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt - - # split up the unit tests into groups based on the number of containers we have - set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt - set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" - cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt - no_output_timeout: 15m - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run Unit Tests (testclasslist-compression) - command: | - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) - if [ -z "$test_timeout" ]; then - test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') - fi - ant testclasslist-compression -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit - no_output_timeout: 15m - - store_test_results: - path: /tmp/cassandra/build/test/output/ - - store_artifacts: - path: /tmp/cassandra/build/test/output - destination: junitxml - - store_artifacts: - path: /tmp/cassandra/build/test/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_repeated_utest: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 25 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Log Environment Information - command: | - echo '*** id ***' - id - echo '*** cat /proc/cpuinfo ***' - cat /proc/cpuinfo - echo '*** free -m ***' - free -m - echo '*** df -m ***' - df -m - echo '*** ifconfig -a ***' - ifconfig -a - echo '*** uname -a ***' - uname -a - echo '*** mount ***' - mount - echo '*** env ***' - env - echo '*** java ***' - which java - java -version - - run: - name: Run repeated JUnit test - no_output_timeout: 15m - command: | - if [ "${REPEATED_UTEST_CLASS}" == "" ]; then - echo "Repeated utest class name hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" == "" ]; then - echo "Repeated utest count hasn't been defined, exiting without running any test" - elif [ "${REPEATED_UTEST_COUNT}" -le 0 ]; then - echo "Repeated utest count is lesser or equals than zero, exiting without running any test" - else - - # Calculate the number of test iterations to be run by the current parallel runner. - # Since we are running the same test multiple times there is no need to use `circleci tests split`. - count=$((${REPEATED_UTEST_COUNT} / CIRCLE_NODE_TOTAL)) - if (($CIRCLE_NODE_INDEX < (${REPEATED_UTEST_COUNT} % CIRCLE_NODE_TOTAL))); then - count=$((count+1)) - fi - - if (($count <= 0)); then - echo "No tests to run in this runner" - else - echo "Running ${REPEATED_UTEST_TARGET} ${REPEATED_UTEST_CLASS} ${REPEATED_UTEST_METHODS} ${REPEATED_UTEST_COUNT} times" - - set -x - export PATH=$JAVA_HOME/bin:$PATH - time mv ~/cassandra /tmp - cd /tmp/cassandra - if [ -d ~/dtest_jars ]; then - cp ~/dtest_jars/dtest* /tmp/cassandra/build/ - fi - - target=${REPEATED_UTEST_TARGET} - class_path=${REPEATED_UTEST_CLASS} - class_name="${class_path##*.}" - - # Prepare the -Dtest.name argument. - # It can be the fully qualified class name or the short class name, depending on the target. - if [[ $target == "test" || \ - $target == "test-cdc" || \ - $target == "test-compression" || \ - $target == "test-system-keyspace-directory" ]]; then - name="-Dtest.name=$class_name" - else - name="-Dtest.name=$class_path" - fi - - # Prepare the -Dtest.methods argument, which is optional - if [ "${REPEATED_UTEST_METHODS}" == "" ]; then - methods="" - else - methods="-Dtest.methods=${REPEATED_UTEST_METHODS}" - fi - - # Run the test target as many times as requested collecting the exit code, - # stopping the iteration only if stop_on_failure is set. - exit_code="$?" - for i in $(seq -w 1 $count); do - - echo "Running test iteration $i of $count" - - # run the test - status="passes" - if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then - status="fails" - exit_code=1 - fi - - # move the stdout output file - dest=/tmp/results/repeated_utest/stdout/${status}/${i} - mkdir -p $dest - mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt - - # move the XML output files - source=build/test/output - dest=/tmp/results/repeated_utest/output/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # move the log files - source=build/test/logs - dest=/tmp/results/repeated_utest/logs/${status}/${i} - mkdir -p $dest - if [[ -d $source && -n "$(ls $source)" ]]; then - mv $source/* $dest/ - fi - - # maybe stop iterations on test failure - if [[ ${REPEATED_UTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then - break - fi - done - - (exit ${exit_code}) - fi - fi - - store_test_results: - path: /tmp/results/repeated_utest/output - - store_artifacts: - path: /tmp/results/repeated_utest/stdout - destination: stdout - - store_artifacts: - path: /tmp/results/repeated_utest/output - destination: junitxml - - store_artifacts: - path: /tmp/results/repeated_utest/logs - destination: logs - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - j8_dtest_jars_build: - docker: - - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest - resource_class: medium - working_directory: ~/ - shell: /bin/bash -eo pipefail -l - parallelism: 1 - steps: - - attach_workspace: - at: /home/cassandra - - run: - name: Build Cassandra DTest jars - command: | - export PATH=$JAVA_HOME/bin:$PATH - cd ~/cassandra - mkdir ~/dtest_jars - git remote add apache https://github.com/apache/cassandra.git - for branch in cassandra-2.2 cassandra-3.0 cassandra-3.11 cassandra-4.0 trunk; do - # check out the correct cassandra version: - git remote set-branches --add apache '$branch' - git fetch --depth 1 apache $branch - git checkout $branch - git clean -fd - # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - cp build/dtest*.jar ~/dtest_jars - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - done - # and build the dtest-jar for the branch under test - ${ANT_HOME}/bin/ant realclean - git checkout origin/$CIRCLE_BRANCH - git clean -fd - for x in $(seq 1 3); do - ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar - RETURN="$?" - if [ "${RETURN}" -eq "0" ]; then - cp build/dtest*.jar ~/dtest_jars - break - fi - done - # Exit, if we didn't build successfully - if [ "${RETURN}" -ne "0" ]; then - echo "Build failed with exit code: ${RETURN}" - exit ${RETURN} - fi - ls -l ~/dtest_jars - no_output_timeout: 15m - - persist_to_workspace: - root: /home/cassandra - paths: - - dtest_jars - environment: - - ANT_HOME: /usr/share/ant - - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - LANG: en_US.UTF-8 - - KEEP_TEST_DIR: true - - DEFAULT_DIR: /home/cassandra/cassandra-dtest - - PYTHONIOENCODING: utf-8 - - PYTHONUNBUFFERED: true - - CASS_DRIVER_NO_EXTENSIONS: true - - CASS_DRIVER_NO_CYTHON: true - - CASSANDRA_SKIP_SYNC: true - - DTEST_REPO: https://github.com/apache/cassandra-dtest.git - - DTEST_BRANCH: trunk - - CCM_MAX_HEAP_SIZE: 1024M - - CCM_HEAP_NEWSIZE: 256M - - REPEATED_UTEST_TARGET: testsome - - REPEATED_UTEST_CLASS: null - - REPEATED_UTEST_METHODS: null - - REPEATED_UTEST_COUNT: 100 - - REPEATED_UTEST_STOP_ON_FAILURE: false - - REPEATED_DTEST_NAME: null - - REPEATED_DTEST_VNODES: false - - REPEATED_DTEST_COUNT: 100 - - REPEATED_DTEST_STOP_ON_FAILURE: false - - REPEATED_UPGRADE_DTEST_NAME: null - - REPEATED_UPGRADE_DTEST_COUNT: 100 - - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false - - REPEATED_JVM_UPGRADE_DTEST_CLASS: null - - REPEATED_JVM_UPGRADE_DTEST_METHODS: null - - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100 - - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false - - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 -workflows: - version: 2 - java8_separate_tests: - jobs: - - start_j8_build: - type: approval - - j8_build: - requires: - - start_j8_build - - start_j8_unit_tests: - type: approval - - j8_unit_tests: - requires: - - start_j8_unit_tests - - j8_build - - start_j8_jvm_dtests: - type: approval - - j8_jvm_dtests: - requires: - - start_j8_jvm_dtests - - j8_build - - start_j8_cqlshlib_tests: - type: approval - - j8_cqlshlib_tests: - requires: - - start_j8_cqlshlib_tests - - j8_build - - start_j11_unit_tests: - type: approval - - j11_unit_tests: - requires: - - start_j11_unit_tests - - j8_build - - start_utests_long: - type: approval - - utests_long: - requires: - - start_utests_long - - j8_build - - start_utests_compression: - type: approval - - utests_compression: - requires: - - start_utests_compression - - j8_build - - start_utests_stress: - type: approval - - utests_stress: - requires: - - start_utests_stress - - j8_build - - start_utests_fqltool: - type: approval - - utests_fqltool: - requires: - - start_utests_fqltool - - j8_build - - start_utests_system_keyspace_directory: - type: approval - - utests_system_keyspace_directory: - requires: - - start_utests_system_keyspace_directory - - j8_build - - start_j8_dtest_jars_build: - type: approval - - j8_dtest_jars_build: - requires: - - j8_build - - start_j8_dtest_jars_build - - start_jvm_upgrade_dtest: - type: approval - - j8_jvm_upgrade_dtests: - requires: - - start_jvm_upgrade_dtest - - j8_dtest_jars_build - - start_j8_dtests: - type: approval - - j8_dtests-with-vnodes: - requires: - - start_j8_dtests - - j8_build - - j8_dtests-no-vnodes: - requires: - - start_j8_dtests - - j8_build - - start_j11_dtests: - type: approval - - j11_dtests-with-vnodes: - requires: - - start_j11_dtests - - j8_build - - j11_dtests-no-vnodes: - requires: - - start_j11_dtests - - j8_build - - start_upgrade_tests: - type: approval - - j8_upgradetests-no-vnodes: - requires: - - start_upgrade_tests - - j8_build - - start_j8_cqlsh_tests: - type: approval - - j8_cqlsh-dtests-py2-with-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py3-with-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py38-with-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py2-no-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py3-no-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - j8_cqlsh-dtests-py38-no-vnodes: - requires: - - start_j8_cqlsh_tests - - j8_build - - start_j11_cqlsh_tests: - type: approval - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j8_build - - start_j8_repeated_utest: - type: approval - - j8_repeated_utest: - requires: - - start_j8_repeated_utest - - j8_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j8_build - - start_j8_repeated_dtest: - type: approval - - j8_repeated_dtest: - requires: - - start_j8_repeated_dtest - - j8_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j8_build - - start_repeated_upgrade_dtest: - type: approval - - repeated_upgrade_dtest: - requires: - - start_repeated_upgrade_dtest - - j8_build - - start_repeated_jvm_upgrade_dtest: - type: approval - - repeated_jvm_upgrade_dtest: - requires: - - start_repeated_jvm_upgrade_dtest - - j8_dtest_jars_build - java8_pre-commit_tests: - jobs: - - start_pre-commit_tests: - type: approval - - j8_build: - requires: - - start_pre-commit_tests - - j8_unit_tests: - requires: - - j8_build - - j8_jvm_dtests: - requires: - - j8_build - - j8_cqlshlib_tests: - requires: - - j8_build - - j11_unit_tests: - requires: - - j8_build - - start_utests_long: - type: approval - - utests_long: - requires: - - start_utests_long - - j8_build - - start_utests_compression: - type: approval - - utests_compression: - requires: - - start_utests_compression - - j8_build - - start_utests_stress: - type: approval - - utests_stress: - requires: - - start_utests_stress - - j8_build - - start_utests_fqltool: - type: approval - - utests_fqltool: - requires: - - start_utests_fqltool - - j8_build - - utests_system_keyspace_directory: - requires: - - j8_build - - start_j8_dtest_jars_build: - type: approval - - j8_dtest_jars_build: - requires: - - j8_build - - start_j8_dtest_jars_build - - start_jvm_upgrade_dtest: - type: approval - - j8_jvm_upgrade_dtests: - requires: - - start_jvm_upgrade_dtest - - j8_dtest_jars_build - - j8_dtests-with-vnodes: - requires: - - j8_build - - j8_dtests-no-vnodes: - requires: - - j8_build - - j11_dtests-with-vnodes: - requires: - - j8_build - - j11_dtests-no-vnodes: - requires: - - j8_build - - start_upgrade_tests: - type: approval - - j8_upgradetests-no-vnodes: - requires: - - j8_build - - start_upgrade_tests - - j8_cqlsh-dtests-py2-with-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py3-with-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py38-with-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py2-no-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py3-no-vnodes: - requires: - - j8_build - - j8_cqlsh-dtests-py38-no-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - j8_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - j8_build - - start_j8_repeated_utest: - type: approval - - j8_repeated_utest: - requires: - - start_j8_repeated_utest - - j8_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j8_build - - start_j8_repeated_dtest: - type: approval - - j8_repeated_dtest: - requires: - - start_j8_repeated_dtest - - j8_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j8_build - - start_repeated_upgrade_dtest: - type: approval - - repeated_upgrade_dtest: - requires: - - start_repeated_upgrade_dtest - - j8_build - - start_repeated_jvm_upgrade_dtest: - type: approval - - repeated_jvm_upgrade_dtest: - requires: - - start_repeated_jvm_upgrade_dtest - - j8_dtest_jars_build - java11_separate_tests: - jobs: - - start_j11_build: - type: approval - - j11_build: - requires: - - start_j11_build - - start_j11_unit_tests: - type: approval - - j11_unit_tests: - requires: - - start_j11_unit_tests - - j11_build - - start_j11_jvm_dtests: - type: approval - - j11_jvm_dtests: - requires: - - start_j11_jvm_dtests - - j11_build - - start_j11_cqlshlib_tests: - type: approval - - j11_cqlshlib_tests: - requires: - - start_j11_cqlshlib_tests - - j11_build - - start_j11_dtests: - type: approval - - j11_dtests-with-vnodes: - requires: - - start_j11_dtests - - j11_build - - j11_dtests-no-vnodes: - requires: - - start_j11_dtests - - j11_build - - start_j11_cqlsh_tests: - type: approval - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - start_j11_cqlsh_tests - - j11_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j11_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j11_build - java11_pre-commit_tests: - jobs: - - start_pre-commit_tests: - type: approval - - j11_build: - requires: - - start_pre-commit_tests - - j11_unit_tests: - requires: - - j11_build - - j11_jvm_dtests: - requires: - - j11_build - - j11_cqlshlib_tests: - requires: - - j11_build - - j11_jvm_dtests: - requires: - - j11_build - - j11_cqlshlib_tests: - requires: - - j11_build - - j11_dtests-with-vnodes: - requires: - - j11_build - - j11_dtests-no-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py2-with-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py3-with-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py38-with-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py2-no-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py3-no-vnodes: - requires: - - j11_build - - j11_cqlsh-dtests-py38-no-vnodes: - requires: - - j11_build - - start_j11_repeated_utest: - type: approval - - j11_repeated_utest: - requires: - - start_j11_repeated_utest - - j11_build - - start_j11_repeated_dtest: - type: approval - - j11_repeated_dtest: - requires: - - start_j11_repeated_dtest - - j11_build diff --git a/.circleci/config.yml.PAID b/.circleci/config.yml.PAID new file mode 100644 index 000000000000..be6f6f8d34d9 --- /dev/null +++ b/.circleci/config.yml.PAID @@ -0,0 +1,11630 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +version: 2 +jobs: + j8_jvm_upgrade_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: xlarge + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine distributed Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_fqltool_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_FQLTOOL_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_FQLTOOL_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_FQLTOOL} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=fqltool-test + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant fqltool-test $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh-dtests-py2-with-vnodes: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh_dtests_py311_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_vnode_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtests_large_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: xlarge + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_large_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_large_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --only-resource-intensive-tests --force-resource-intensive-tests --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_large_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_large_with_vnodes_raw /tmp/all_dtest_tests_j8_large_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_large_with_vnodes_raw > /tmp/all_dtest_tests_j8_large_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_large_with_vnodes > /tmp/split_dtest_tests_j8_large_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_large_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_large_with_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_large_with_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_large_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_large_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_large_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_system_keyspace_directory: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-system-keyspace-directory) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-system-keyspace-directory -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_stress: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (stress-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant stress-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_stress_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_STRESS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_STRESS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_STRESS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=stress-test-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant stress-test-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_cqlsh_dtests_py311: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_compression_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-compression + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-compression $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_unit_tests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py3: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py38: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_compression_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-compression + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-compression $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_repeated_ant_test: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run repeated JUnit test + no_output_timeout: 15m + command: | + if [ "${REPEATED_ANT_TEST_CLASS}" == "" ]; then + echo "Repeated utest class name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_ANT_TEST_COUNT}" == "" ]; then + echo "Repeated utest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_ANT_TEST_COUNT}" -le 0 ]; then + echo "Repeated utest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_ANT_TEST_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_ANT_TEST_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_ANT_TEST_TARGET} ${REPEATED_ANT_TEST_CLASS} ${REPEATED_ANT_TEST_METHODS} ${REPEATED_ANT_TEST_COUNT} times" + + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + target=${REPEATED_ANT_TEST_TARGET} + class_path=${REPEATED_ANT_TEST_CLASS} + class_name="${class_path##*.}" + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name="-Dtest.name=$class_name" + else + name="-Dtest.name=$class_path" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [ "${REPEATED_ANT_TEST_METHODS}" == "" ]; then + methods="" + else + methods="-Dtest.methods=${REPEATED_ANT_TEST_METHODS}" + fi + + # Run the test target as many times as requested collecting the exit code, + # stopping the iteration only if stop_on_failure is set. + exit_code="$?" + for i in $(seq -w 1 $count); do + + echo "Running test iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utest/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${REPEATED_ANT_TEST_TARGET}-${REPEATED_ANT_TEST_CLASS}.txt + + # move the XML output files + source=build/test/output + dest=/tmp/results/repeated_utest/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs + dest=/tmp/results/repeated_utest/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + + (exit ${exit_code}) + fi + fi + - store_test_results: + path: /tmp/results/repeated_utest/output + - store_artifacts: + path: /tmp/results/repeated_utest/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utest/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utest/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_dtests_large_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: xlarge + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_large_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_large_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --only-resource-intensive-tests --force-resource-intensive-tests --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_large_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_large_with_vnodes_raw /tmp/all_dtest_tests_j11_large_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_large_with_vnodes_raw > /tmp/all_dtest_tests_j11_large_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_large_with_vnodes > /tmp/split_dtest_tests_j11_large_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_large_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_large_with_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_large_with_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_large_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_large_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_large_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_dtests_large_vnode_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_LARGE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_LARGE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_LARGE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_LARGE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_LARGE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --only-resource-intensive-tests --force-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py311: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py38_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_dtests_large: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: xlarge + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_large_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_large_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --only-resource-intensive-tests --force-resource-intensive-tests --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_large_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_large_without_vnodes_raw /tmp/all_dtest_tests_j11_large_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_large_without_vnodes_raw > /tmp/all_dtest_tests_j11_large_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_large_without_vnodes > /tmp/split_dtest_tests_j11_large_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_large_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_large_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_large_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_large_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_large_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_large_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_system_keyspace_directory_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-system-keyspace-directory + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-system-keyspace-directory $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py3_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py3: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_cdc: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-cdc) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-cdc -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_fqltool: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (fqltool-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant fqltool-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_system_keyspace_directory: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-system-keyspace-directory) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-system-keyspace-directory -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_offheap_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --use-off-heap-memtables --skip-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh-dtests-py2-offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_offheap_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --use-off-heap-memtables --skip-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtests_large_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_LARGE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_LARGE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_LARGE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_LARGE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_LARGE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --only-resource-intensive-tests --force-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_utests_compression: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-compression) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-compression -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_long: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (long-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant long-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_unit_tests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=testsome + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant testsome $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_large: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: xlarge + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 4 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_large_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_large_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --only-resource-intensive-tests --force-resource-intensive-tests --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_large_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_large_without_vnodes_raw /tmp/all_dtest_tests_j8_large_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_large_without_vnodes_raw > /tmp/all_dtest_tests_j8_large_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_large_without_vnodes > /tmp/split_dtest_tests_j8_large_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_large_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_large_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_large_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_large_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_large_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_large_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh-dtests-py2-no-vnodes: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_stress: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (stress-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant stress-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py38_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_upgrade_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: xlarge + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_UPGRADE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_UPGRADE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_UPGRADE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_UPGRADE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UPGRADE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_UPGRADE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_UPGRADE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if true; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh-dtests-py2-with-vnodes: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_utests_cdc_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-cdc + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-cdc $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_fqltool_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_FQLTOOL_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_FQLTOOL_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_FQLTOOL} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=fqltool-test + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant fqltool-test $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_compression: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-compression) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-compression -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py38: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py3_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_cqlsh_dtests_py311_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_system_keyspace_directory_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-system-keyspace-directory + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-system-keyspace-directory $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_large_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_LARGE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_LARGE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_LARGE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_LARGE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_LARGE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --only-resource-intensive-tests --force-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_cqlsh_dtests_py3_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_utests_stress_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_STRESS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_STRESS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_STRESS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=stress-test-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant stress-test-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_vnode_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py3_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then + export CQLSH_PYTHON=/usr/bin/python3.6 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_upgrade_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: xlarge + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 100 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_upgradetests_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_upgradetests_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw /tmp/all_dtest_tests_j8_upgradetests_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw > /tmp/all_dtest_tests_j8_upgradetests_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_upgradetests_without_vnodes > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_upgradetests_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_upgradetests_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_upgradetests_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_upgradetests_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\"\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_jvm_upgrade_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_JVM_UPGRADE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_UPGRADE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_JVM_UPGRADE_DTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-jvm-dtest-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-jvm-dtest-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh_dtests_py38_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_jvm_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_JVM_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_JVM_DTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-jvm-dtest-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-jvm-dtest-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_cqlsh_dtests_py311_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_long_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_LONG_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_LONG_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_LONG} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=long-testsome + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant long-testsome $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\"\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_unit_tests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_jvm_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 10 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine distributed Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_build: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra Repository (via git) + command: | + git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra + - run: + name: Build Cassandra + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. + for x in $(seq 1 3); do + ${ANT_HOME}/bin/ant clean realclean jar + RETURN="$?" + if [ "${RETURN}" -eq "0" ]; then + break + fi + done + # Exit, if we didn't build successfully + if [ "${RETURN}" -ne "0" ]; then + echo "Build failed with exit code: ${RETURN}" + exit ${RETURN} + fi + no_output_timeout: 15m + - run: + name: Run eclipse-warnings + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + ant eclipse-warnings + - persist_to_workspace: + root: /home/cassandra + paths: + - cassandra + - .m2 + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_without_vnodes_raw /tmp/all_dtest_tests_j8_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_without_vnodes_raw > /tmp/all_dtest_tests_j8_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_without_vnodes > /tmp/split_dtest_tests_j8_without_vnodes.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n" + - run: + name: Run dtests (j8_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh-dtests-py2-no-vnodes: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_without_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt" + cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtests_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlshlib_tests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run cqlshlib Unit Tests + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra/pylib + ./cassandra-cqlsh-tests.sh .. + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/pylib + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_jvm_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_JVM_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_JVM_DTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-jvm-dtest-some + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-jvm-dtest-some $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_without_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_without_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_without_vnodes_raw /tmp/all_dtest_tests_j11_without_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_without_vnodes_raw > /tmp/all_dtest_tests_j11_without_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_without_vnodes > /tmp/split_dtest_tests_j11_without_vnodes.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_without_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n" + - run: + name: Run dtests (j11_without_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_without_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_without_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_repeated_ant_test: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run repeated JUnit test + no_output_timeout: 15m + command: | + if [ "${REPEATED_ANT_TEST_CLASS}" == "" ]; then + echo "Repeated utest class name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_ANT_TEST_COUNT}" == "" ]; then + echo "Repeated utest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_ANT_TEST_COUNT}" -le 0 ]; then + echo "Repeated utest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_ANT_TEST_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_ANT_TEST_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_ANT_TEST_TARGET} ${REPEATED_ANT_TEST_CLASS} ${REPEATED_ANT_TEST_METHODS} ${REPEATED_ANT_TEST_COUNT} times" + + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + target=${REPEATED_ANT_TEST_TARGET} + class_path=${REPEATED_ANT_TEST_CLASS} + class_name="${class_path##*.}" + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name="-Dtest.name=$class_name" + else + name="-Dtest.name=$class_path" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [ "${REPEATED_ANT_TEST_METHODS}" == "" ]; then + methods="" + else + methods="-Dtest.methods=${REPEATED_ANT_TEST_METHODS}" + fi + + # Run the test target as many times as requested collecting the exit code, + # stopping the iteration only if stop_on_failure is set. + exit_code="$?" + for i in $(seq -w 1 $count); do + + echo "Running test iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utest/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${REPEATED_ANT_TEST_TARGET}-${REPEATED_ANT_TEST_CLASS}.txt + + # move the XML output files + source=build/test/output + dest=/tmp/results/repeated_utest/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs + dest=/tmp/results/repeated_utest/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + + (exit ${exit_code}) + fi + fi + - store_test_results: + path: /tmp/results/repeated_utest/output + - store_artifacts: + path: /tmp/results/repeated_utest/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utest/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utest/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_utests_long: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (long-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant long-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_utests_cdc: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine unit Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist-cdc) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist-cdc -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh_dtests_py311_offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.11/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_dtests_offheap_raw /tmp/all_dtest_tests_j8_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j8_dtests_offheap_raw > /tmp/all_dtest_tests_j8_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_dtests_offheap > /tmp/split_dtest_tests_j8_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j8_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt + + source ~/env3.11/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.11' ]; then + export CQLSH_PYTHON=/usr/bin/python3.11 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if false; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_jvm_dtests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 10 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Determine distributed Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + no_output_timeout: 15m + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Run Unit Tests (testclasslist) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_build: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra Repository (via git) + command: | + git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra + - run: + name: Build Cassandra + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. + for x in $(seq 1 3); do + ${ANT_HOME}/bin/ant clean realclean jar + RETURN="$?" + if [ "${RETURN}" -eq "0" ]; then + break + fi + done + # Exit, if we didn't build successfully + if [ "${RETURN}" -ne "0" ]; then + echo "Build failed with exit code: ${RETURN}" + exit ${RETURN} + fi + no_output_timeout: 15m + - run: + name: Run eclipse-warnings + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + ant eclipse-warnings + - persist_to_workspace: + root: /home/cassandra + paths: + - cassandra + - .m2 + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlsh_dtests_py38_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j8_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j8_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j8_with_vnodes_raw /tmp/all_dtest_tests_j8_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j8_with_vnodes_raw > /tmp/all_dtest_tests_j8_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j8_with_vnodes > /tmp/split_dtest_tests_j8_with_vnodes.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j8_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n" + - run: + name: Run dtests (j8_with_vnodes) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt" + cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt + + source ~/env3.8/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then + export CQLSH_PYTHON=/usr/bin/python3.8 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j8_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j8_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_cqlsh-dtests-py2-offheap: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_dtests_offheap) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_dtests_offheap)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_dtests_offheap_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_dtests_offheap_raw /tmp/all_dtest_tests_j11_dtests_offheap\nelse\n grep -e '' /tmp/all_dtest_tests_j11_dtests_offheap_raw > /tmp/all_dtest_tests_j11_dtests_offheap || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_dtests_offheap > /tmp/split_dtest_tests_j11_dtests_offheap.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\ncat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt\n" + - run: + name: Run dtests (j11_dtests_offheap) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt" + cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then + export CQLSH_PYTHON=/usr/bin/python2.7 + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_dtests_offheap_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_dtests_offheap.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_dtests_offheap + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_dtests_offheap_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j11_unit_tests_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=testsome + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant testsome $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_fqltool: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run Unit Tests (fqltool-test) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant fqltool-test -Dno-build-test=true + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_cqlshlib_tests: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Run cqlshlib Unit Tests + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra/pylib + ./cassandra-cqlsh-tests.sh .. + no_output_timeout: 15m + - store_test_results: + path: /tmp/cassandra/pylib + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_dtests_vnode: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 50 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Determine Tests to Run (j11_with_vnodes) + no_output_timeout: 5m + command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql' --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_j11_with_vnodes_raw --cassandra-dir=../cassandra\nif [ -z '' ]; then\n mv /tmp/all_dtest_tests_j11_with_vnodes_raw /tmp/all_dtest_tests_j11_with_vnodes\nelse\n grep -e '' /tmp/all_dtest_tests_j11_with_vnodes_raw > /tmp/all_dtest_tests_j11_with_vnodes || { echo \"Filter did not match any tests! Exiting build.\"; exit 0; }\nfi\nset -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_j11_with_vnodes > /tmp/split_dtest_tests_j11_with_vnodes.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes.txt | tr '\\n' ' ' > /tmp/split_dtest_tests_j11_with_vnodes_final.txt\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n" + - run: + name: Run dtests (j11_with_vnodes) + no_output_timeout: 15m + command: "echo \"cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee\nexport SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`\nif [ ! -z \"$SPLIT_TESTS\" ]; then\n set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt\nelse\n echo \"Tune your parallelism, there are more containers than test classes. Nothing to do in this container\"\n (exit 1)\nfi\n" + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_j11_with_vnodes + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_j11_with_vnodes_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_utests_cdc_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=test-cdc + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant test-cdc $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j8_dtests_large_vnode_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: large + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "${REPEATED_LARGE_DTESTS}" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "${REPEATED_LARGE_DTESTS_COUNT}" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((${REPEATED_LARGE_DTESTS_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_LARGE_DTESTS_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running ${REPEATED_LARGE_DTESTS} $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo ${REPEATED_LARGE_DTESTS} | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if ${REPEATED_TESTS_STOP_ON_FAILURE}; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if true; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if false; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir --only-resource-intensive-tests --force-resource-intensive-tests $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + j11_utests_long_repeat: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 25 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((${REPEATED_UTESTS_LONG_COUNT} / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (${REPEATED_UTESTS_LONG_COUNT} % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo ${REPEATED_UTESTS_LONG} | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=long-testsome + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant long-testsome $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ ${REPEATED_TESTS_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - CASSANDRA_USE_JDK11: true + j8_dtest_jars_build: + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: medium + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Build Cassandra DTest jars + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + mkdir ~/dtest_jars + git remote add apache https://github.com/apache/cassandra.git + for branch in cassandra-2.2 cassandra-3.0 cassandra-3.11 cassandra-4.0 trunk; do + # check out the correct cassandra version: + git remote set-branches --add apache '$branch' + git fetch --depth 1 apache $branch + git checkout $branch + git clean -fd + # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. + for x in $(seq 1 3); do + ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar + RETURN="$?" + if [ "${RETURN}" -eq "0" ]; then + cp build/dtest*.jar ~/dtest_jars + break + fi + done + # Exit, if we didn't build successfully + if [ "${RETURN}" -ne "0" ]; then + echo "Build failed with exit code: ${RETURN}" + exit ${RETURN} + fi + done + # and build the dtest-jar for the branch under test + ${ANT_HOME}/bin/ant realclean + git checkout origin/$CIRCLE_BRANCH + git clean -fd + for x in $(seq 1 3); do + ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar + RETURN="$?" + if [ "${RETURN}" -eq "0" ]; then + cp build/dtest*.jar ~/dtest_jars + break + fi + done + # Exit, if we didn't build successfully + if [ "${RETURN}" -ne "0" ]; then + echo "Build failed with exit code: ${RETURN}" + exit ${RETURN} + fi + ls -l ~/dtest_jars + no_output_timeout: 15m + - persist_to_workspace: + root: /home/cassandra + paths: + - dtest_jars + environment: + - ANT_HOME: /usr/share/ant + - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - LANG: en_US.UTF-8 + - KEEP_TEST_DIR: true + - DEFAULT_DIR: /home/cassandra/cassandra-dtest + - PYTHONIOENCODING: utf-8 + - PYTHONUNBUFFERED: true + - CASS_DRIVER_NO_EXTENSIONS: true + - CASS_DRIVER_NO_CYTHON: true + - CASSANDRA_SKIP_SYNC: true + - DTEST_REPO: https://github.com/apache/cassandra-dtest.git + - DTEST_BRANCH: trunk + - CCM_MAX_HEAP_SIZE: 1024M + - CCM_HEAP_NEWSIZE: 256M + - REPEATED_TESTS_STOP_ON_FAILURE: false + - REPEATED_UTESTS: null + - REPEATED_UTESTS_COUNT: 500 + - REPEATED_UTESTS_FQLTOOL: null + - REPEATED_UTESTS_FQLTOOL_COUNT: 500 + - REPEATED_UTESTS_LONG: null + - REPEATED_UTESTS_LONG_COUNT: 100 + - REPEATED_UTESTS_STRESS: null + - REPEATED_UTESTS_STRESS_COUNT: 500 + - REPEATED_JVM_DTESTS: null + - REPEATED_JVM_DTESTS_COUNT: 500 + - REPEATED_JVM_UPGRADE_DTESTS: null + - REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + - REPEATED_DTESTS: null + - REPEATED_DTESTS_COUNT: 500 + - REPEATED_LARGE_DTESTS: null + - REPEATED_LARGE_DTESTS_COUNT: 100 + - REPEATED_UPGRADE_DTESTS: null + - REPEATED_UPGRADE_DTESTS_COUNT: 25 + - REPEATED_ANT_TEST_TARGET: testsome + - REPEATED_ANT_TEST_CLASS: null + - REPEATED_ANT_TEST_METHODS: null + - REPEATED_ANT_TEST_COUNT: 500 + - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 +workflows: + version: 2 + java8_separate_tests: + jobs: + - start_j8_build: + type: approval + - j8_build: + requires: + - start_j8_build + - start_j8_unit_tests: + type: approval + - j8_unit_tests: + requires: + - start_j8_unit_tests + - j8_build + - start_j8_jvm_dtests: + type: approval + - j8_jvm_dtests: + requires: + - start_j8_jvm_dtests + - j8_build + - start_j8_cqlshlib_tests: + type: approval + - j8_cqlshlib_tests: + requires: + - start_j8_cqlshlib_tests + - j8_build + - start_j11_unit_tests: + type: approval + - j11_unit_tests: + requires: + - start_j11_unit_tests + - j8_build + - start_j8_utests_long: + type: approval + - j8_utests_long: + requires: + - start_j8_utests_long + - j8_build + - start_j11_utests_long: + type: approval + - j11_utests_long: + requires: + - start_j11_utests_long + - j8_build + - start_j8_utests_cdc: + type: approval + - j8_utests_cdc: + requires: + - start_j8_utests_cdc + - j8_build + - start_j11_utests_cdc: + type: approval + - j11_utests_cdc: + requires: + - start_j11_utests_cdc + - j8_build + - start_j8_utests_compression: + type: approval + - j8_utests_compression: + requires: + - start_j8_utests_compression + - j8_build + - start_j11_utests_compression: + type: approval + - j11_utests_compression: + requires: + - start_j11_utests_compression + - j8_build + - start_j8_utests_stress: + type: approval + - j8_utests_stress: + requires: + - start_j8_utests_stress + - j8_build + - start_j11_utests_stress: + type: approval + - j11_utests_stress: + requires: + - start_j11_utests_stress + - j8_build + - start_j8_utests_fqltool: + type: approval + - j8_utests_fqltool: + requires: + - start_j8_utests_fqltool + - j8_build + - start_j11_utests_fqltool: + type: approval + - j11_utests_fqltool: + requires: + - start_j11_utests_fqltool + - j8_build + - start_j8_utests_system_keyspace_directory: + type: approval + - j8_utests_system_keyspace_directory: + requires: + - start_j8_utests_system_keyspace_directory + - j8_build + - start_j11_utests_system_keyspace_directory: + type: approval + - j11_utests_system_keyspace_directory: + requires: + - start_j11_utests_system_keyspace_directory + - j8_build + - start_j8_dtest_jars_build: + type: approval + - j8_dtest_jars_build: + requires: + - j8_build + - start_j8_dtest_jars_build + - start_jvm_upgrade_dtests: + type: approval + - j8_jvm_upgrade_dtests: + requires: + - start_jvm_upgrade_dtests + - j8_dtest_jars_build + - start_j8_dtests: + type: approval + - j8_dtests: + requires: + - start_j8_dtests + - j8_build + - start_j8_dtests_vnode: + type: approval + - j8_dtests_vnode: + requires: + - start_j8_dtests_vnode + - j8_build + - start_j8_dtests_offheap: + type: approval + - j8_dtests_offheap: + requires: + - start_j8_dtests_offheap + - j8_build + - start_j8_dtests_large: + type: approval + - j8_dtests_large: + requires: + - start_j8_dtests_large + - j8_build + - start_j8_dtests_large_vnode: + type: approval + - j8_dtests_large_vnode: + requires: + - start_j8_dtests_large_vnode + - j8_build + - start_j11_dtests: + type: approval + - j11_dtests: + requires: + - start_j11_dtests + - j8_build + - start_j11_dtests_vnode: + type: approval + - j11_dtests_vnode: + requires: + - start_j11_dtests_vnode + - j8_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j8_build + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j8_build + - start_j11_dtests_large_vnode: + type: approval + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large_vnode + - j8_build + - start_upgrade_tests: + type: approval + - j8_upgrade_dtests: + requires: + - start_upgrade_tests + - j8_build + - start_j8_cqlsh_tests: + type: approval + - j8_cqlsh-dtests-py2-with-vnodes: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py3_vnode: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py38_vnode: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py311_vnode: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh-dtests-py2-no-vnodes: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py3: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py38: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py311: + requires: + - start_j8_cqlsh_tests + - j8_build + - start_j8_cqlsh_tests_offheap: + type: approval + - j8_cqlsh-dtests-py2-offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - j8_cqlsh_dtests_py3_offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - j8_cqlsh_dtests_py38_offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - j8_cqlsh_dtests_py311_offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - start_j11_cqlsh_tests: + type: approval + - j11_cqlsh-dtests-py2-with-vnodes: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py3_vnode: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py38_vnode: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py311_vnode: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh-dtests-py2-no-vnodes: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py3: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py38: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py311: + requires: + - start_j11_cqlsh_tests + - j8_build + - start_j11_cqlsh_tests_offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh_tests_offheap + - j8_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh_tests_offheap + - j8_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh_tests_offheap + - j8_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh_tests_offheap + - j8_build + java8_pre-commit_tests: + jobs: + - start_pre-commit_tests: + type: approval + - j8_build: + requires: + - start_pre-commit_tests + - j8_unit_tests: + requires: + - j8_build + - j8_jvm_dtests: + requires: + - j8_build + - j8_cqlshlib_tests: + requires: + - j8_build + - j11_unit_tests: + requires: + - j8_build + - start_utests_long: + type: approval + - j8_utests_long: + requires: + - start_utests_long + - j8_build + - j11_utests_long: + requires: + - start_utests_long + - j8_build + - start_utests_cdc: + type: approval + - j8_utests_cdc: + requires: + - start_utests_cdc + - j8_build + - j11_utests_cdc: + requires: + - start_utests_cdc + - j8_build + - start_utests_compression: + type: approval + - j8_utests_compression: + requires: + - start_utests_compression + - j8_build + - j11_utests_compression: + requires: + - start_utests_compression + - j8_build + - start_utests_stress: + type: approval + - j8_utests_stress: + requires: + - start_utests_stress + - j8_build + - j11_utests_stress: + requires: + - start_utests_stress + - j8_build + - start_utests_fqltool: + type: approval + - j8_utests_fqltool: + requires: + - start_utests_fqltool + - j8_build + - j11_utests_fqltool: + requires: + - start_utests_fqltool + - j8_build + - start_utests_system_keyspace_directory: + type: approval + - j8_utests_system_keyspace_directory: + requires: + - j8_build + - j11_utests_system_keyspace_directory: + requires: + - start_utests_system_keyspace_directory + - j8_build + - start_jvm_upgrade_dtests: + type: approval + - j8_dtest_jars_build: + requires: + - j8_build + - start_jvm_upgrade_dtests + - j8_jvm_upgrade_dtests: + requires: + - j8_dtest_jars_build + - j8_dtests: + requires: + - j8_build + - j8_dtests_vnode: + requires: + - j8_build + - start_j8_dtests_offheap: + type: approval + - j8_dtests_offheap: + requires: + - start_j8_dtests_offheap + - j8_build + - j11_dtests: + requires: + - j8_build + - j11_dtests_vnode: + requires: + - j8_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j8_build + - start_j8_dtests_large: + type: approval + - j8_dtests_large: + requires: + - start_j8_dtests_large + - j8_build + - j8_dtests_large_vnode: + requires: + - start_j8_dtests_large + - j8_build + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j8_build + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large + - j8_build + - start_upgrade_tests: + type: approval + - j8_upgrade_dtests: + requires: + - j8_build + - start_upgrade_tests + - j8_cqlsh-dtests-py2-with-vnodes: + requires: + - j8_build + - j8_cqlsh_dtests_py3_vnode: + requires: + - j8_build + - j8_cqlsh_dtests_py38_vnode: + requires: + - j8_build + - j8_cqlsh_dtests_py311_vnode: + requires: + - j8_build + - j8_cqlsh-dtests-py2-no-vnodes: + requires: + - j8_build + - j8_cqlsh_dtests_py3: + requires: + - j8_build + - j8_cqlsh_dtests_py38: + requires: + - j8_build + - j8_cqlsh_dtests_py311: + requires: + - j8_build + - start_j8_cqlsh_dtests_offheap: + type: approval + - j8_cqlsh-dtests-py2-offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j8_cqlsh_dtests_py3_offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j8_cqlsh_dtests_py38_offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j8_cqlsh_dtests_py311_offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j11_cqlsh-dtests-py2-with-vnodes: + requires: + - j8_build + - j11_cqlsh_dtests_py3_vnode: + requires: + - j8_build + - j11_cqlsh_dtests_py38_vnode: + requires: + - j8_build + - j11_cqlsh_dtests_py311_vnode: + requires: + - j8_build + - j11_cqlsh-dtests-py2-no-vnodes: + requires: + - j8_build + - j11_cqlsh_dtests_py3: + requires: + - j8_build + - j11_cqlsh_dtests_py38: + requires: + - j8_build + - j11_cqlsh_dtests_py311: + requires: + - j8_build + - start_j11_cqlsh-dtests-offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j8_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j8_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j8_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j8_build + java11_separate_tests: + jobs: + - start_j11_build: + type: approval + - j11_build: + requires: + - start_j11_build + - start_j11_unit_tests: + type: approval + - j11_unit_tests: + requires: + - start_j11_unit_tests + - j11_build + - start_j11_jvm_dtests: + type: approval + - j11_jvm_dtests: + requires: + - start_j11_jvm_dtests + - j11_build + - start_j11_cqlshlib_tests: + type: approval + - j11_cqlshlib_tests: + requires: + - start_j11_cqlshlib_tests + - j11_build + - start_j11_dtests: + type: approval + - j11_dtests: + requires: + - start_j11_dtests + - j11_build + - start_j11_dtests_vnode: + type: approval + - j11_dtests_vnode: + requires: + - start_j11_dtests_vnode + - j11_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j11_build + - start_j11_cqlsh_tests: + type: approval + - j11_cqlsh-dtests-py2-with-vnodes: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py3_vnode: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py38_vnode: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py311_vnode: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh-dtests-py2-no-vnodes: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py3: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py38: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py311: + requires: + - start_j11_cqlsh_tests + - j11_build + - start_j11_cqlsh-dtests-offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j11_build + - start_j11_dtests_large_vnode: + type: approval + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large_vnode + - j11_build + - start_j11_utests_long: + type: approval + - j11_utests_long: + requires: + - start_j11_utests_long + - j11_build + - start_j11_utests_cdc: + type: approval + - j11_utests_cdc: + requires: + - start_j11_utests_cdc + - j11_build + - start_j11_utests_compression: + type: approval + - j11_utests_compression: + requires: + - start_j11_utests_compression + - j11_build + - start_j11_utests_stress: + type: approval + - j11_utests_stress: + requires: + - start_j11_utests_stress + - j11_build + - start_j11_utests_fqltool: + type: approval + - j11_utests_fqltool: + requires: + - start_j11_utests_fqltool + - j11_build + - start_j11_utests_system_keyspace_directory: + type: approval + - j11_utests_system_keyspace_directory: + requires: + - start_j11_utests_system_keyspace_directory + - j11_build + java11_pre-commit_tests: + jobs: + - start_pre-commit_tests: + type: approval + - j11_build: + requires: + - start_pre-commit_tests + - j11_unit_tests: + requires: + - j11_build + - j11_jvm_dtests: + requires: + - j11_build + - j11_cqlshlib_tests: + requires: + - j11_build + - j11_jvm_dtests: + requires: + - j11_build + - j11_cqlshlib_tests: + requires: + - j11_build + - j11_dtests: + requires: + - j11_build + - j11_dtests_vnode: + requires: + - j11_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j11_build + - j11_cqlsh-dtests-py2-with-vnodes: + requires: + - j11_build + - j11_cqlsh_dtests_py3_vnode: + requires: + - j11_build + - j11_cqlsh_dtests_py38_vnode: + requires: + - j11_build + - j11_cqlsh_dtests_py311_vnode: + requires: + - j11_build + - j11_cqlsh-dtests-py2-no-vnodes: + requires: + - j11_build + - j11_cqlsh_dtests_py3: + requires: + - j11_build + - j11_cqlsh_dtests_py38: + requires: + - j11_build + - j11_cqlsh_dtests_py311: + requires: + - j11_build + - start_j11_cqlsh-dtests-offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j11_build + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large + - j11_build + - start_utests_long: + type: approval + - j11_utests_long: + requires: + - start_utests_long + - j11_build + - start_utests_cdc: + type: approval + - j11_utests_cdc: + requires: + - start_utests_cdc + - j11_build + - start_utests_compression: + type: approval + - j11_utests_compression: + requires: + - start_utests_compression + - j11_build + - start_utests_stress: + type: approval + - j11_utests_stress: + requires: + - start_utests_stress + - j11_build + - start_utests_fqltool: + type: approval + - j11_utests_fqltool: + requires: + - start_utests_fqltool + - j11_build + - start_utests_system_keyspace_directory: + type: approval + - j11_utests_system_keyspace_directory: + requires: + - start_utests_system_keyspace_directory + - j11_build diff --git a/.circleci/config_template.yml b/.circleci/config_template.yml new file mode 100644 index 000000000000..38f5c7539fab --- /dev/null +++ b/.circleci/config_template.yml @@ -0,0 +1,3338 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +version: 2.1 + +default_env_vars: &default_env_vars + + # The values of some of these environment variables are meant to be frequently changed by developers. + # The generate.sh script contains a list of accepted environment variables that should contain some of + # these variables. Also, some variables are mentioned in the documentation, at least in + # .circleci/readme.md and in doc/source/development/testing.rst. + # If you modify these variables, or if you add new variables whose values are meant to be changed frequently, + # please remember to modify the generate.sh script and the documentation accordingly. + + ANT_HOME: /usr/share/ant + JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + LANG: en_US.UTF-8 + KEEP_TEST_DIR: true + DEFAULT_DIR: /home/cassandra/cassandra-dtest + PYTHONIOENCODING: utf-8 + PYTHONUNBUFFERED: true + CASS_DRIVER_NO_EXTENSIONS: true + CASS_DRIVER_NO_CYTHON: true + #Skip all syncing to disk to avoid performance issues in flaky CI environments + CASSANDRA_SKIP_SYNC: true + DTEST_REPO: https://github.com/apache/cassandra-dtest.git + DTEST_BRANCH: trunk + CCM_MAX_HEAP_SIZE: 1024M + CCM_HEAP_NEWSIZE: 256M + + # Whether the repeated test iterations should stop on the first failure by default. + REPEATED_TESTS_STOP_ON_FAILURE: false + + # Comma-separated list of tests that should be included in the repeated run for regular unit tests, + # in addition to automatically detected new and modified tests. For example: + # REPEATED_UTESTS: org.apache.cassandra.cql3.ViewTest + # REPEATED_UTESTS: org.apache.cassandra.cql3.ViewTest#testCountersTable + # REPEATED_UTESTS: org.apache.cassandra.cql3.ViewTest,org.apache.cassandra.cql3.functions.TimeFctsTest + REPEATED_UTESTS: + # The number of times that new, modified or manually specified unit tests should be run. + REPEATED_UTESTS_COUNT: 500 + + # Comma-separated list of tests that should be included in the repeated run for fqltool unit tests, + # in addition to automatically detected new and modified tests. For example: + # REPEATED_UTESTS_FQLTOOL: org.apache.cassandra.fqltool.FQLCompareTest + # REPEATED_UTESTS_FQLTOOL: org.apache.cassandra.fqltool.FQLCompareTest#endToEnd + # REPEATED_UTESTS_FQLTOOL: org.apache.cassandra.fqltool.FQLCompareTest,org.apache.cassandra.fqltool.FQLReplayTest + REPEATED_UTESTS_FQLTOOL: + # The number of times that new, modified or manually specified fqltool unit tests should be run. + REPEATED_UTESTS_FQLTOOL_COUNT: 500 + + # Comma-separated list of tests that should be included in the repeated run for long unit tests, + # in addition to automatically detected new and modified tests. For example: + # REPEATED_UTESTS_LONG: org.apache.cassandra.db.commitlog.CommitLogStressTest + # REPEATED_UTESTS_LONG: org.apache.cassandra.db.commitlog.CommitLogStressTest#testRandomSize + REPEATED_UTESTS_LONG: + # The number of times that new, modified or manually specified long unit tests should be run. + REPEATED_UTESTS_LONG_COUNT: 100 + + # Comma-separated list of tests that should be included in the repeated run for stress unit tests, + # in addition to automatically detected new and modified tests. For example: + # REPEATED_UTESTS_STRESS: org.apache.cassandra.stress.generate.DistributionGaussianTest + # REPEATED_UTESTS_STRESS: org.apache.cassandra.stress.generate.DistributionGaussianTest#simpleGaussian + REPEATED_UTESTS_STRESS: + # The number of times that new, modified or manually specified stress unit tests should be run. + REPEATED_UTESTS_STRESS_COUNT: 500 + + # Comma-separated list of tests that should be included in the repeated run for JVM dtests, + # in addition to automatically detected new and modified tests. For example: + # REPEATED_JVM_DTESTS: org.apache.cassandra.distributed.test.PagingTest + # REPEATED_JVM_DTESTS: org.apache.cassandra.distributed.test.PagingTest#testPaging + REPEATED_JVM_DTESTS: + # The number of times that new, modified or manually specified JVM dtests should be run. + REPEATED_JVM_DTESTS_COUNT: 500 + + # Comma-separated list of tests that should be included in the repeated run for JVM upgrade dtests, + # in addition to automatically detected new and modified tests. For example: + # REPEATED_JVM_UPGRADE_DTESTS: org.apache.cassandra.distributed.upgrade.GroupByTest + # REPEATED_JVM_UPGRADE_DTESTS: org.apache.cassandra.distributed.upgrade.GroupByTest#testReads + REPEATED_JVM_UPGRADE_DTESTS: + # The number of times that new, modified or manually specified JVM upgrade dtests should be run. + REPEATED_JVM_UPGRADE_DTESTS_COUNT: 500 + + # Comma-separated list of Python dtests that should be repeatedly run, for example: + # REPEATED_DTESTS: cqlsh_tests/test_cqlsh.py + # REPEATED_DTESTS: cqlsh_tests/test_cqlsh.py::TestCqlshSmoke + # REPEATED_DTESTS: cqlsh_tests/test_cqlsh.py::TestCqlshSmoke::test_create_index + # REPEATED_DTESTS: cqlsh_tests/test_cqlsh.py,consistency_test.py + REPEATED_DTESTS: + # The number of times that the manually specified Python dtests should be run. + REPEATED_DTESTS_COUNT: 500 + + # Comma-separated list of Python large dtests that should be repeatedly run, for example: + # REPEATED_LARGE_DTESTS: replace_address_test.py + # REPEATED_LARGE_DTESTS: replace_address_test.py::TestReplaceAddress + # REPEATED_LARGE_DTESTS: replace_address_test.py::TestReplaceAddress::test_replace_stopped_node + # REPEATED_LARGE_DTESTS: replace_address_test.py,materialized_views_test.py + REPEATED_LARGE_DTESTS: + # The number of times that the manually specified Python large dtests should be run. + REPEATED_LARGE_DTESTS_COUNT: 100 + + # Comma-separated list of Python dtests that should be repeatedly run, for example: + # REPEATED_UPGRADE_DTESTS: upgrade_tests/cql_tests.py + # REPEATED_UPGRADE_DTESTS: upgrade_tests/cql_tests.py::TestCQLNodes2RF1_Upgrade_current_4_0_x_To_indev_4_0_x + # REPEATED_UPGRADE_DTESTS: upgrade_tests/cql_tests.py,upgrade_tests/paging_test.py + REPEATED_UPGRADE_DTESTS: + # The number of times that the manually specified Python upgrade dtests should be run. + REPEATED_UPGRADE_DTESTS_COUNT: 25 + + # The Ant test target to run, for example: + # REPEATED_ANT_TEST_TARGET: testsome + # REPEATED_ANT_TEST_TARGET: test-jvm-dtest-some + # REPEATED_ANT_TEST_TARGET: test-cdc + # REPEATED_ANT_TEST_TARGET: test-compression + # REPEATED_ANT_TEST_TARGET: test-system-keyspace-directory + REPEATED_ANT_TEST_TARGET: testsome + # The name of JUnit class to be run multiple times, for example: + # REPEATED_ANT_TEST_CLASS: org.apache.cassandra.cql3.ViewTest + # REPEATED_ANT_TEST_CLASS: org.apache.cassandra.distributed.test.PagingTest + REPEATED_ANT_TEST_CLASS: + # The optional specific methods within REPEATED_ANT_TEST_CLASS to be run, for example: + # REPEATED_ANT_TEST_METHODS: testCompoundPartitionKey + # REPEATED_ANT_TEST_METHODS: testCompoundPartitionKey,testStaticTable + # Please note that some Ant targets will ignore the -Dtest.methods argument produced by this. + REPEATED_ANT_TEST_METHODS: + # The number of times that the repeated JUnit test should be run. + REPEATED_ANT_TEST_COUNT: 500 + +j8_par_executor: &j8_par_executor + executor: + name: java8-executor + #exec_resource_class: xlarge + parallelism: 4 + +j8_small_par_executor: &j8_small_par_executor + executor: + name: java8-executor + #exec_resource_class: xlarge + parallelism: 1 + +j8_small_executor: &j8_small_executor + executor: + name: java8-executor + exec_resource_class: medium + parallelism: 1 + +j8_medium_par_executor: &j8_medium_par_executor + executor: + name: java8-executor + #exec_resource_class: xlarge + parallelism: 1 + +j8_seq_executor: &j8_seq_executor + executor: + name: java8-executor + #exec_resource_class: xlarge + parallelism: 1 # sequential, single container tests: no parallelism benefits + +j11_par_executor: &j11_par_executor + executor: + name: java11-executor + #exec_resource_class: xlarge + parallelism: 4 + +j11_small_par_executor: &j11_small_par_executor + executor: + name: java11-executor + #exec_resource_class: xlarge + parallelism: 1 + +j11_small_executor: &j11_small_executor + executor: + name: java11-executor + #exec_resource_class: medium + parallelism: 1 + +j11_medium_par_executor: &j11_medium_par_executor + executor: + name: java11-executor + #exec_resource_class: xlarge + parallelism: 1 + +j11_seq_executor: &j11_seq_executor + executor: + name: java11-executor + #exec_resource_class: xlarge + parallelism: 1 # sequential, single container tests: no parallelism benefits + +j8_repeated_utest_executor: &j8_repeated_utest_executor + executor: + name: java8-executor + parallelism: 4 + +j8_repeated_dtest_executor: &j8_repeated_dtest_executor + executor: + name: java8-executor + parallelism: 4 + +j8_repeated_upgrade_dtest_executor: &j8_repeated_upgrade_dtest_executor + executor: + name: java8-executor + parallelism: 4 + +j8_repeated_jvm_upgrade_dtest_executor: &j8_repeated_jvm_upgrade_dtest_executor + executor: + name: java8-executor + parallelism: 4 + +j11_repeated_utest_executor: &j11_repeated_utest_executor + executor: + name: java11-executor + parallelism: 4 + +j11_repeated_dtest_executor: &j11_repeated_dtest_executor + executor: + name: java11-executor + parallelism: 4 + +j8_separate_jobs: &j8_separate_jobs + jobs: + - start_j8_build: + type: approval + - j8_build: + requires: + - start_j8_build + # Java 8 unit tests + - start_j8_unit_tests: + type: approval + - j8_unit_tests: + requires: + - start_j8_unit_tests + - j8_build + - start_j8_unit_tests_repeat: + type: approval + - j8_unit_tests_repeat: + requires: + - start_j8_unit_tests_repeat + - j8_build + - start_j8_jvm_dtests: + type: approval + - j8_jvm_dtests: + requires: + - start_j8_jvm_dtests + - j8_build + - start_j8_jvm_dtests_repeat: + type: approval + - j8_jvm_dtests_repeat: + requires: + - start_j8_jvm_dtests_repeat + - j8_build + - start_j8_cqlshlib_tests: + type: approval + - j8_cqlshlib_tests: + requires: + - start_j8_cqlshlib_tests + - j8_build + # Java 11 unit tests + - start_j11_unit_tests: + type: approval + - j11_unit_tests: + requires: + - start_j11_unit_tests + - j8_build + - start_j11_unit_tests_repeat: + type: approval + - j11_unit_tests_repeat: + requires: + - start_j11_unit_tests_repeat + - j8_build + # specialized unit tests (all run on request) + - start_j8_utests_long: + type: approval + - j8_utests_long: + requires: + - start_j8_utests_long + - j8_build + - start_j11_utests_long: + type: approval + - j11_utests_long: + requires: + - start_j11_utests_long + - j8_build + - start_j8_utests_long_repeat: + type: approval + - j8_utests_long_repeat: + requires: + - start_j8_utests_long_repeat + - j8_build + - start_j11_utests_long_repeat: + type: approval + - j11_utests_long_repeat: + requires: + - start_j11_utests_long_repeat + - j8_build + - start_j8_utests_cdc: + type: approval + - j8_utests_cdc: + requires: + - start_j8_utests_cdc + - j8_build + - start_j11_utests_cdc: + type: approval + - j11_utests_cdc: + requires: + - start_j11_utests_cdc + - j8_build + - start_j8_utests_cdc_repeat: + type: approval + - j8_utests_cdc_repeat: + requires: + - start_j8_utests_cdc_repeat + - j8_build + - start_j11_utests_cdc_repeat: + type: approval + - j11_utests_cdc_repeat: + requires: + - start_j11_utests_cdc_repeat + - j8_build + - start_j8_utests_compression: + type: approval + - j8_utests_compression: + requires: + - start_j8_utests_compression + - j8_build + - start_j11_utests_compression: + type: approval + - j11_utests_compression: + requires: + - start_j11_utests_compression + - j8_build + - start_j8_utests_compression_repeat: + type: approval + - j8_utests_compression_repeat: + requires: + - start_j8_utests_compression_repeat + - j8_build + - start_j11_utests_compression_repeat: + type: approval + - j11_utests_compression_repeat: + requires: + - start_j11_utests_compression_repeat + - j8_build + - start_j8_utests_stress: + type: approval + - j8_utests_stress: + requires: + - start_j8_utests_stress + - j8_build + - start_j11_utests_stress: + type: approval + - j11_utests_stress: + requires: + - start_j11_utests_stress + - j8_build + - start_j8_utests_stress_repeat: + type: approval + - j8_utests_stress_repeat: + requires: + - start_j8_utests_stress_repeat + - j8_build + - start_j11_utests_stress_repeat: + type: approval + - j11_utests_stress_repeat: + requires: + - start_j11_utests_stress_repeat + - j8_build + - start_j8_utests_fqltool: + type: approval + - j8_utests_fqltool: + requires: + - start_j8_utests_fqltool + - j8_build + - start_j11_utests_fqltool: + type: approval + - j11_utests_fqltool: + requires: + - start_j11_utests_fqltool + - j8_build + - start_j8_utests_fqltool_repeat: + type: approval + - j8_utests_fqltool_repeat: + requires: + - start_j8_utests_fqltool_repeat + - j8_build + - start_j11_utests_fqltool_repeat: + type: approval + - j11_utests_fqltool_repeat: + requires: + - start_j11_utests_fqltool_repeat + - j8_build + - start_j8_utests_system_keyspace_directory: + type: approval + - j8_utests_system_keyspace_directory: + requires: + - start_j8_utests_system_keyspace_directory + - j8_build + - start_j11_utests_system_keyspace_directory: + type: approval + - j11_utests_system_keyspace_directory: + requires: + - start_j11_utests_system_keyspace_directory + - j8_build + - start_j8_utests_system_keyspace_directory_repeat: + type: approval + - j8_utests_system_keyspace_directory_repeat: + requires: + - start_j8_utests_system_keyspace_directory_repeat + - j8_build + - start_j11_utests_system_keyspace_directory_repeat: + type: approval + - j11_utests_system_keyspace_directory_repeat: + requires: + - start_j11_utests_system_keyspace_directory_repeat + - j8_build + - start_j8_dtest_jars_build: + type: approval + - j8_dtest_jars_build: + requires: + - j8_build + - start_j8_dtest_jars_build + - start_jvm_upgrade_dtests: + type: approval + - j8_jvm_upgrade_dtests: + requires: + - start_jvm_upgrade_dtests + - j8_dtest_jars_build + - start_jvm_upgrade_dtests_repeat: + type: approval + - j8_jvm_upgrade_dtests_repeat: + requires: + - start_jvm_upgrade_dtests_repeat + - j8_dtest_jars_build + # Java 8 dtests + - start_j8_dtests: + type: approval + - j8_dtests: + requires: + - start_j8_dtests + - j8_build + - start_j8_dtests_repeat: + type: approval + - j8_dtests_repeat: + requires: + - start_j8_dtests_repeat + - j8_build + - start_j8_dtests_vnode: + type: approval + - j8_dtests_vnode: + requires: + - start_j8_dtests_vnode + - j8_build + - start_j8_dtests_vnode_repeat: + type: approval + - j8_dtests_vnode_repeat: + requires: + - start_j8_dtests_vnode_repeat + - j8_build + # Java 8 off-heap dtests + - start_j8_dtests_offheap: + type: approval + - j8_dtests_offheap: + requires: + - start_j8_dtests_offheap + - j8_build + - start_j8_dtests_offheap_repeat: + type: approval + - j8_dtests_offheap_repeat: + requires: + - start_j8_dtests_offheap_repeat + - j8_build + # Java 8 large DTests + - start_j8_dtests_large: + type: approval + - j8_dtests_large: + requires: + - start_j8_dtests_large + - j8_build + - start_j8_dtests_large_repeat: + type: approval + - j8_dtests_large_repeat: + requires: + - start_j8_dtests_large_repeat + - j8_build + - start_j8_dtests_large_vnode: + type: approval + - j8_dtests_large_vnode: + requires: + - start_j8_dtests_large_vnode + - j8_build + - start_j8_dtests_large_vnode_repeat: + type: approval + - j8_dtests_large_vnode_repeat: + requires: + - start_j8_dtests_large_vnode_repeat + - j8_build + # Java 11 dtests + - start_j11_dtests: + type: approval + - j11_dtests: + requires: + - start_j11_dtests + - j8_build + - start_j11_dtests_repeat: + type: approval + - j11_dtests_repeat: + requires: + - start_j11_dtests_repeat + - j8_build + - start_j11_dtests_vnode: + type: approval + - j11_dtests_vnode: + requires: + - start_j11_dtests_vnode + - j8_build + - start_j11_dtests_vnode_repeat: + type: approval + - j11_dtests_vnode_repeat: + requires: + - start_j11_dtests_vnode_repeat + - j8_build + # Java 11 off-heap dtests + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j8_build + - start_j11_dtests_offheap_repeat: + type: approval + - j11_dtests_offheap_repeat: + requires: + - start_j11_dtests_offheap_repeat + - j8_build + # Java 11 Python large DTests + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j8_build + - start_j11_dtests_large_repeat: + type: approval + - j11_dtests_large_repeat: + requires: + - start_j11_dtests_large_repeat + - j8_build + - start_j11_dtests_large_vnode: + type: approval + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large_vnode + - j8_build + - j11_dtests_large_vnode_repeat: + requires: + - start_j11_dtests_large_repeat + - j8_build + # Java 8 upgrade tests + - start_upgrade_tests: + type: approval + - j8_upgrade_dtests: + requires: + - start_upgrade_tests + - j8_build + - start_j8_upgrade_dtests_repeat: + type: approval + - j8_upgrade_dtests_repeat: + requires: + - start_j8_upgrade_dtests_repeat + - j8_build + # Java 8 cqlsh dtests + - start_j8_cqlsh_tests: + type: approval + - j8_cqlsh-dtests-py2-with-vnodes: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py3_vnode: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py38_vnode: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py311_vnode: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh-dtests-py2-no-vnodes: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py3: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py38: + requires: + - start_j8_cqlsh_tests + - j8_build + - j8_cqlsh_dtests_py311: + requires: + - start_j8_cqlsh_tests + - j8_build + # Java 8 cqlsh offheap dtests offheap + - start_j8_cqlsh_tests_offheap: + type: approval + - j8_cqlsh-dtests-py2-offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - j8_cqlsh_dtests_py3_offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - j8_cqlsh_dtests_py38_offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + - j8_cqlsh_dtests_py311_offheap: + requires: + - start_j8_cqlsh_tests_offheap + - j8_build + # Java 11 cqlsh dtests + - start_j11_cqlsh_tests: + type: approval + - j11_cqlsh-dtests-py2-with-vnodes: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py3_vnode: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py38_vnode: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py311_vnode: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh-dtests-py2-no-vnodes: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py3: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py38: + requires: + - start_j11_cqlsh_tests + - j8_build + - j11_cqlsh_dtests_py311: + requires: + - start_j11_cqlsh_tests + - j8_build + # Java 11 cqlsh dtests off-heap + - start_j11_cqlsh_tests_offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh_tests_offheap + - j8_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh_tests_offheap + - j8_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh_tests_offheap + - j8_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh_tests_offheap + - j8_build + # Java 8 repeated utest + - start_j8_repeated_ant_test: + type: approval + - j8_repeated_ant_test: + requires: + - start_j8_repeated_ant_test + - j8_build + # Java 11 repeated utest + - start_j11_repeated_ant_test: + type: approval + - j11_repeated_ant_test: + requires: + - start_j11_repeated_ant_test + - j8_build + +j8_pre-commit_jobs: &j8_pre-commit_jobs + jobs: + - start_pre-commit_tests: + type: approval + - j8_build: + requires: + - start_pre-commit_tests + # Java 8 unit tests + - j8_unit_tests: + requires: + - j8_build + - j8_unit_tests_repeat: + requires: + - j8_build + - j8_jvm_dtests: + requires: + - j8_build + - j8_jvm_dtests_repeat: + requires: + - j8_build + - j8_cqlshlib_tests: + requires: + - j8_build + # Java 11 unit tests + - j11_unit_tests: + requires: + - j8_build + - j11_unit_tests_repeat: + requires: + - j8_build + # specialized unit tests (all run on request) + - start_utests_long: + type: approval + - j8_utests_long: + requires: + - start_utests_long + - j8_build + - j11_utests_long: + requires: + - start_utests_long + - j8_build + - j8_utests_long_repeat: + requires: + - start_utests_long + - j8_build + - j11_utests_long_repeat: + requires: + - start_utests_long + - j8_build + - start_utests_cdc: + type: approval + - j8_utests_cdc: + requires: + - start_utests_cdc + - j8_build + - j11_utests_cdc: + requires: + - start_utests_cdc + - j8_build + - j8_utests_cdc_repeat: + requires: + - start_utests_cdc + - j8_build + - j11_utests_cdc_repeat: + requires: + - start_utests_cdc + - j8_build + - start_utests_compression: + type: approval + - j8_utests_compression: + requires: + - start_utests_compression + - j8_build + - j11_utests_compression: + requires: + - start_utests_compression + - j8_build + - j8_utests_compression_repeat: + requires: + - start_utests_compression + - j8_build + - j11_utests_compression_repeat: + requires: + - start_utests_compression + - j8_build + - start_utests_stress: + type: approval + - j8_utests_stress: + requires: + - start_utests_stress + - j8_build + - j11_utests_stress: + requires: + - start_utests_stress + - j8_build + - j8_utests_stress_repeat: + requires: + - start_utests_stress + - j8_build + - j11_utests_stress_repeat: + requires: + - start_utests_stress + - j8_build + - start_utests_fqltool: + type: approval + - j8_utests_fqltool: + requires: + - start_utests_fqltool + - j8_build + - j11_utests_fqltool: + requires: + - start_utests_fqltool + - j8_build + - j8_utests_fqltool_repeat: + requires: + - start_utests_fqltool + - j8_build + - j11_utests_fqltool_repeat: + requires: + - start_utests_fqltool + - j8_build + - start_utests_system_keyspace_directory: + type: approval + - j8_utests_system_keyspace_directory: + requires: + - j8_build + - j11_utests_system_keyspace_directory: + requires: + - start_utests_system_keyspace_directory + - j8_build + - j8_utests_system_keyspace_directory_repeat: + requires: + - j8_build + - j11_utests_system_keyspace_directory_repeat: + requires: + - start_utests_system_keyspace_directory + - j8_build + - start_jvm_upgrade_dtests: + type: approval + - j8_dtest_jars_build: + requires: + - j8_build + - start_jvm_upgrade_dtests + - j8_jvm_upgrade_dtests: + requires: + - j8_dtest_jars_build + - j8_jvm_upgrade_dtests_repeat: + requires: + - j8_dtest_jars_build + # Java 8 dtests + - j8_dtests: + requires: + - j8_build + - j8_dtests_repeat: + requires: + - j8_build + - j8_dtests_vnode: + requires: + - j8_build + - j8_dtests_vnode_repeat: + requires: + - j8_build + - start_j8_dtests_offheap: + type: approval + - j8_dtests_offheap: + requires: + - start_j8_dtests_offheap + - j8_build + - j8_dtests_offheap_repeat: + requires: + - start_j8_dtests_offheap + - j8_build + # Java 11 dtests + - j11_dtests: + requires: + - j8_build + - j11_dtests_repeat: + requires: + - j8_build + - j11_dtests_vnode: + requires: + - j8_build + - j11_dtests_vnode_repeat: + requires: + - j8_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j8_build + - j11_dtests_offheap_repeat: + requires: + - start_j11_dtests_offheap + - j8_build + # Java 8 large DTests + - start_j8_dtests_large: + type: approval + - j8_dtests_large: + requires: + - start_j8_dtests_large + - j8_build + - j8_dtests_large_repeat: + requires: + - start_j8_dtests_large + - j8_build + - j8_dtests_large_vnode: + requires: + - start_j8_dtests_large + - j8_build + - j8_dtests_large_vnode_repeat: + requires: + - start_j8_dtests_large + - j8_build + # Java 11 large DTests + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j8_build + - j11_dtests_large_repeat: + requires: + - start_j11_dtests_large + - j8_build + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large + - j8_build + - j11_dtests_large_vnode_repeat: + requires: + - start_j11_dtests_large + - j8_build + # Java 8 upgrade tests (on request) + - start_upgrade_tests: + type: approval + - j8_upgrade_dtests: + requires: + - j8_build + - start_upgrade_tests + - j8_upgrade_dtests_repeat: + requires: + - j8_build + - start_upgrade_tests + # Java 8 cqlsh dtests + - j8_cqlsh-dtests-py2-with-vnodes: + requires: + - j8_build + - j8_cqlsh_dtests_py3_vnode: + requires: + - j8_build + - j8_cqlsh_dtests_py38_vnode: + requires: + - j8_build + - j8_cqlsh_dtests_py311_vnode: + requires: + - j8_build + - j8_cqlsh-dtests-py2-no-vnodes: + requires: + - j8_build + - j8_cqlsh_dtests_py3: + requires: + - j8_build + - j8_cqlsh_dtests_py38: + requires: + - j8_build + - j8_cqlsh_dtests_py311: + requires: + - j8_build + # Java 8 cqlsh dtests offheap + - start_j8_cqlsh_dtests_offheap: + type: approval + - j8_cqlsh-dtests-py2-offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j8_cqlsh_dtests_py3_offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j8_cqlsh_dtests_py38_offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + - j8_cqlsh_dtests_py311_offheap: + requires: + - start_j8_cqlsh_dtests_offheap + - j8_build + # Java 11 cqlsh dtests + - j11_cqlsh-dtests-py2-with-vnodes: + requires: + - j8_build + - j11_cqlsh_dtests_py3_vnode: + requires: + - j8_build + - j11_cqlsh_dtests_py38_vnode: + requires: + - j8_build + - j11_cqlsh_dtests_py311_vnode: + requires: + - j8_build + - j11_cqlsh-dtests-py2-no-vnodes: + requires: + - j8_build + - j11_cqlsh_dtests_py3: + requires: + - j8_build + - j11_cqlsh_dtests_py38: + requires: + - j8_build + - j11_cqlsh_dtests_py311: + requires: + - j8_build + # Java 11 cqlsh dtests off-heap + - start_j11_cqlsh-dtests-offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j8_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j8_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j8_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j8_build + +j11_separate_jobs: &j11_separate_jobs + jobs: + - start_j11_build: + type: approval + - j11_build: + requires: + - start_j11_build + # Java 11 unit tests + - start_j11_unit_tests: + type: approval + - j11_unit_tests: + requires: + - start_j11_unit_tests + - j11_build + - start_j11_unit_tests_repeat: + type: approval + - j11_unit_tests_repeat: + requires: + - start_j11_unit_tests_repeat + - j11_build + - start_j11_jvm_dtests: + type: approval + - j11_jvm_dtests: + requires: + - start_j11_jvm_dtests + - j11_build + - start_j11_jvm_dtests_repeat: + type: approval + - j11_jvm_dtests_repeat: + requires: + - start_j11_jvm_dtests_repeat + - j11_build + - start_j11_cqlshlib_tests: + type: approval + - j11_cqlshlib_tests: + requires: + - start_j11_cqlshlib_tests + - j11_build + # Java 11 dtests + - start_j11_dtests: + type: approval + - j11_dtests: + requires: + - start_j11_dtests + - j11_build + - start_j11_dtests_vnode: + type: approval + - j11_dtests_vnode: + requires: + - start_j11_dtests_vnode + - j11_build + - start_j11_dtests_repeat: + type: approval + - j11_dtests_repeat: + requires: + - start_j11_dtests_repeat + - j11_build + - start_j11_dtests_vnode_repeat: + type: approval + - j11_dtests_vnode_repeat: + requires: + - start_j11_dtests_vnode_repeat + - j11_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j11_build + - start_j11_dtests_offheap_repeat: + type: approval + - j11_dtests_offheap_repeat: + requires: + - start_j11_dtests_offheap_repeat + - j11_build + - start_j11_cqlsh_tests: + type: approval + - j11_cqlsh-dtests-py2-with-vnodes: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py3_vnode: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py38_vnode: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py311_vnode: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh-dtests-py2-no-vnodes: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py3: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py38: + requires: + - start_j11_cqlsh_tests + - j11_build + - j11_cqlsh_dtests_py311: + requires: + - start_j11_cqlsh_tests + - j11_build + - start_j11_cqlsh-dtests-offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + # Python large DTests + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j11_build + - start_j11_dtests_large_repeat: + type: approval + - j11_dtests_large_repeat: + requires: + - start_j11_dtests_large_repeat + - j11_build + - start_j11_dtests_large_vnode: + type: approval + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large_vnode + - j11_build + - start_j11_dtests_large_vnode_repeat: + type: approval + - j11_dtests_large_vnode_repeat: + requires: + - start_j11_dtests_large_vnode_repeat + - j11_build + # specialized unit tests (all run on request) + - start_j11_utests_long: + type: approval + - j11_utests_long: + requires: + - start_j11_utests_long + - j11_build + - start_j11_utests_long_repeat: + type: approval + - j11_utests_long_repeat: + requires: + - start_j11_utests_long_repeat + - j11_build + - start_j11_utests_cdc: + type: approval + - j11_utests_cdc: + requires: + - start_j11_utests_cdc + - j11_build + - start_j11_utests_cdc_repeat: + type: approval + - j11_utests_cdc_repeat: + requires: + - start_j11_utests_cdc_repeat + - j11_build + - start_j11_utests_compression: + type: approval + - j11_utests_compression: + requires: + - start_j11_utests_compression + - j11_build + - start_j11_utests_compression_repeat: + type: approval + - j11_utests_compression_repeat: + requires: + - start_j11_utests_compression_repeat + - j11_build + - start_j11_utests_stress: + type: approval + - j11_utests_stress: + requires: + - start_j11_utests_stress + - j11_build + - start_j11_utests_stress_repeat: + type: approval + - j11_utests_stress_repeat: + requires: + - start_j11_utests_stress_repeat + - j11_build + - start_j11_utests_fqltool: + type: approval + - j11_utests_fqltool: + requires: + - start_j11_utests_fqltool + - j11_build + - start_j11_utests_fqltool_repeat: + type: approval + - j11_utests_fqltool_repeat: + requires: + - start_j11_utests_fqltool_repeat + - j11_build + - start_j11_utests_system_keyspace_directory: + type: approval + - j11_utests_system_keyspace_directory: + requires: + - start_j11_utests_system_keyspace_directory + - j11_build + - start_j11_utests_system_keyspace_directory_repeat: + type: approval + - j11_utests_system_keyspace_directory_repeat: + requires: + - start_j11_utests_system_keyspace_directory_repeat + - j11_build + # Java 11 repeated utest + - start_j11_repeated_ant_test: + type: approval + - j11_repeated_ant_test: + requires: + - start_j11_repeated_ant_test + - j11_build + +j11_pre-commit_jobs: &j11_pre-commit_jobs + jobs: + - start_pre-commit_tests: + type: approval + - j11_build: + requires: + - start_pre-commit_tests + - j11_unit_tests: + requires: + - j11_build + - j11_unit_tests_repeat: + requires: + - j11_build + - j11_jvm_dtests: + requires: + - j11_build + - j11_jvm_dtests_repeat: + requires: + - j11_build + - j11_cqlshlib_tests: + requires: + - j11_build + - j11_jvm_dtests: + requires: + - j11_build + - j11_cqlshlib_tests: + requires: + - j11_build + - j11_dtests: + requires: + - j11_build + - j11_dtests_repeat: + requires: + - j11_build + - j11_dtests_vnode: + requires: + - j11_build + - j11_dtests_vnode_repeat: + requires: + - j11_build + - start_j11_dtests_offheap: + type: approval + - j11_dtests_offheap: + requires: + - start_j11_dtests_offheap + - j11_build + - start_j11_dtests_offheap_repeat: + type: approval + - j11_dtests_offheap_repeat: + requires: + - start_j11_dtests_offheap_repeat + - j11_build + - j11_cqlsh-dtests-py2-with-vnodes: + requires: + - j11_build + - j11_cqlsh_dtests_py3_vnode: + requires: + - j11_build + - j11_cqlsh_dtests_py38_vnode: + requires: + - j11_build + - j11_cqlsh_dtests_py311_vnode: + requires: + - j11_build + - j11_cqlsh-dtests-py2-no-vnodes: + requires: + - j11_build + - j11_cqlsh_dtests_py3: + requires: + - j11_build + - j11_cqlsh_dtests_py38: + requires: + - j11_build + - j11_cqlsh_dtests_py311: + requires: + - j11_build + - start_j11_cqlsh-dtests-offheap: + type: approval + - j11_cqlsh-dtests-py2-offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py3_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py38_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + - j11_cqlsh_dtests_py311_offheap: + requires: + - start_j11_cqlsh-dtests-offheap + - j11_build + # Large Python DTests + - start_j11_dtests_large: + type: approval + - j11_dtests_large: + requires: + - start_j11_dtests_large + - j11_build + - j11_dtests_large_repeat: + requires: + - start_j11_dtests_large + - j11_build + - j11_dtests_large_vnode: + requires: + - start_j11_dtests_large + - j11_build + - j11_dtests_large_vnode_repeat: + requires: + - start_j11_dtests_large + - j11_build + # specialized unit tests (all run on request) + - start_utests_long: + type: approval + - j11_utests_long: + requires: + - start_utests_long + - j11_build + - j11_utests_long_repeat: + requires: + - start_utests_long + - j11_build + - start_utests_cdc: + type: approval + - j11_utests_cdc: + requires: + - start_utests_cdc + - j11_build + - j11_utests_cdc_repeat: + requires: + - start_utests_cdc + - j11_build + - start_utests_compression: + type: approval + - j11_utests_compression: + requires: + - start_utests_compression + - j11_build + - j11_utests_compression_repeat: + requires: + - start_utests_compression + - j11_build + - start_utests_stress: + type: approval + - j11_utests_stress: + requires: + - start_utests_stress + - j11_build + - j11_utests_stress_repeat: + requires: + - start_utests_stress + - j11_build + - start_utests_fqltool: + type: approval + - j11_utests_fqltool: + requires: + - start_utests_fqltool + - j11_build + - j11_utests_fqltool_repeat: + requires: + - start_utests_fqltool + - j11_build + - start_utests_system_keyspace_directory: + type: approval + - j11_utests_system_keyspace_directory: + requires: + - start_utests_system_keyspace_directory + - j11_build + - j11_utests_system_keyspace_directory_repeat: + requires: + - start_utests_system_keyspace_directory + - j11_build + +workflows: + version: 2 + java8_separate_tests: *j8_separate_jobs + java8_pre-commit_tests: *j8_pre-commit_jobs + java11_separate_tests: *j11_separate_jobs + java11_pre-commit_tests: *j11_pre-commit_jobs + +executors: + java8-executor: + parameters: + exec_resource_class: + type: string + default: medium + docker: + - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:latest + resource_class: << parameters.exec_resource_class >> + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + environment: + <<: *default_env_vars + JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + + java11-executor: + parameters: + exec_resource_class: + type: string + default: medium + docker: + - image: apache/cassandra-testing-ubuntu2004-java11:latest + resource_class: << parameters.exec_resource_class >> + working_directory: ~/ + shell: /bin/bash -eo pipefail -l + environment: + <<: *default_env_vars + JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64 + CASSANDRA_USE_JDK11: true + +build_common: &build_common + parallelism: 1 # This job doesn't benefit from parallelism + steps: + - log_environment + - clone_cassandra + - build_cassandra + - run_eclipse_warnings + - persist_to_workspace: + root: /home/cassandra + paths: + - cassandra + - .m2 + +jobs: + j8_build: + executor: java8-executor + <<: *build_common + + j11_build: + executor: java11-executor + <<: *build_common + + j8_dtest_jars_build: + executor: java8-executor + parallelism: 1 + steps: + - attach_workspace: + at: /home/cassandra + - build_cassandra_dtest_jars + - persist_to_workspace: + root: /home/cassandra + paths: + - dtest_jars + + j8_unit_tests: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - create_junit_containers + - log_environment + - run_parallel_junit_tests + + j8_jvm_dtests: + <<: *j8_small_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - create_junit_containers: + classlistprefix: distributed + extra_filters: "| grep -v upgrade" + - log_environment + - run_parallel_junit_tests: + classlistprefix: distributed + target: "testclasslist" + + j11_jvm_dtests: + <<: *j11_small_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - create_junit_containers: + classlistprefix: distributed + extra_filters: "| grep -v upgrade" + - log_environment + - run_parallel_junit_tests: + classlistprefix: distributed + target: "testclasslist" + + j8_jvm_upgrade_dtests: + <<: *j8_medium_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - create_junit_containers: + classlistprefix: distributed + extra_filters: "| grep upgrade" + - log_environment + - run_parallel_junit_tests: + classlistprefix: distributed + target: "testclasslist" + + j11_unit_tests: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - create_junit_containers + - log_environment + - run_parallel_junit_tests + + j8_cqlshlib_tests: + <<: *j8_small_executor + steps: + - attach_workspace: + at: /home/cassandra + - run_cqlshlib_tests + + j11_cqlshlib_tests: + <<: *j11_small_executor + steps: + - attach_workspace: + at: /home/cassandra + - run_cqlshlib_tests + + j8_utests_long: + <<: *j8_seq_executor + steps: + - attach_workspace: + at: /home/cassandra + - run_junit_tests: + target: long-test + + j11_utests_long: + <<: *j11_seq_executor + steps: + - attach_workspace: + at: /home/cassandra + - run_junit_tests: + target: long-test + + j8_utests_cdc: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - create_junit_containers + - log_environment + - run_parallel_junit_tests: + target: testclasslist-cdc + + j11_utests_cdc: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - create_junit_containers + - log_environment + - run_parallel_junit_tests: + target: testclasslist-cdc + + j8_utests_compression: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - create_junit_containers + - log_environment + - run_parallel_junit_tests: + target: testclasslist-compression + + j11_utests_compression: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - create_junit_containers + - log_environment + - run_parallel_junit_tests: + target: testclasslist-compression + + j8_utests_stress: + <<: *j8_seq_executor + steps: + - attach_workspace: + at: /home/cassandra + - run_junit_tests: + target: stress-test + + j11_utests_stress: + <<: *j11_seq_executor + steps: + - attach_workspace: + at: /home/cassandra + - run_junit_tests: + target: stress-test + + j8_utests_fqltool: + <<: *j8_seq_executor + steps: + - attach_workspace: + at: /home/cassandra + - run_junit_tests: + target: fqltool-test + + j11_utests_fqltool: + <<: *j11_seq_executor + steps: + - attach_workspace: + at: /home/cassandra + - run_junit_tests: + target: fqltool-test + + j8_utests_system_keyspace_directory: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - create_junit_containers + - log_environment + - run_parallel_junit_tests: + target: testclasslist-system-keyspace-directory + + j11_utests_system_keyspace_directory: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - create_junit_containers + - log_environment + - run_parallel_junit_tests: + target: testclasslist-system-keyspace-directory + + j8_dtests_vnode: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j8_with_vnodes + run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql'" + - run_dtests: + file_tag: j8_with_vnodes + pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' + + j8_dtests_offheap: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j8_dtests_offheap + run_dtests_extra_args: "--use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k not cql'" + - run_dtests: + file_tag: j8_dtests_offheap + pytest_extra_args: '--use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests' + + j11_dtests_vnode: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j11_with_vnodes + run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k not cql'" + - run_dtests: + file_tag: j11_with_vnodes + pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' + + j11_dtests_offheap: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j11_dtests_offheap + run_dtests_extra_args: "--use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k not cql'" + - run_dtests: + file_tag: j11_dtests_offheap + pytest_extra_args: '--use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests' + + j8_dtests: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j8_without_vnodes + run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k not cql'" + - run_dtests: + file_tag: j8_without_vnodes + pytest_extra_args: '--skip-resource-intensive-tests' + + j11_dtests: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j11_without_vnodes + run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k not cql'" + - run_dtests: + file_tag: j11_without_vnodes + pytest_extra_args: '--skip-resource-intensive-tests' + + j8_dtests_large_vnode: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j8_large_with_vnodes + run_dtests_extra_args: '--use-vnodes --only-resource-intensive-tests --force-resource-intensive-tests' + - run_dtests: + file_tag: j8_large_with_vnodes + pytest_extra_args: '--use-vnodes --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests' + + j8_dtests_large: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j8_large_without_vnodes + run_dtests_extra_args: '--only-resource-intensive-tests --force-resource-intensive-tests' + - run_dtests: + file_tag: j8_large_without_vnodes + pytest_extra_args: '--num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests' + + j11_dtests_large_vnode: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j11_large_with_vnodes + run_dtests_extra_args: '--use-vnodes --only-resource-intensive-tests --force-resource-intensive-tests' + - run_dtests: + file_tag: j11_large_with_vnodes + pytest_extra_args: '--use-vnodes --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests' + + j11_dtests_large: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j11_large_without_vnodes + run_dtests_extra_args: '--only-resource-intensive-tests --force-resource-intensive-tests' + - run_dtests: + file_tag: j11_large_without_vnodes + pytest_extra_args: '--num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests' + + j8_upgrade_dtests: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j8_upgradetests_without_vnodes + run_dtests_extra_args: '--execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all' + - run_dtests: + file_tag: j8_upgradetests_without_vnodes + pytest_extra_args: '--execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all' + + j8_cqlsh-dtests-py2-with-vnodes: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j8_with_vnodes + run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" + - run_dtests: + file_tag: j8_with_vnodes + pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' + + j8_cqlsh-dtests-py2-offheap: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j8_dtests_offheap + run_dtests_extra_args: "--use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql'" + - run_dtests: + file_tag: j8_dtests_offheap + pytest_extra_args: '--use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' + + j8_cqlsh_dtests_py3_vnode: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j8_with_vnodes + run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" + - run_dtests: + file_tag: j8_with_vnodes + pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' + + j8_cqlsh_dtests_py3_offheap: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j8_dtests_offheap + run_dtests_extra_args: "--use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql'" + - run_dtests: + file_tag: j8_dtests_offheap + pytest_extra_args: '--use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' + + j8_cqlsh_dtests_py38_vnode: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv: + python_version: '3.8' + - create_dtest_containers: + file_tag: j8_with_vnodes + run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" + python_version: '3.8' + - run_dtests: + file_tag: j8_with_vnodes + pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8' + python_version: '3.8' + + j8_cqlsh_dtests_py311_vnode: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv: + python_version: '3.11' + - create_dtest_containers: + file_tag: j8_with_vnodes + run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" + python_version: '3.11' + - run_dtests: + file_tag: j8_with_vnodes + pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.11' + python_version: '3.11' + + j8_cqlsh_dtests_py38_offheap: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv: + python_version: '3.8' + - create_dtest_containers: + file_tag: j8_dtests_offheap + run_dtests_extra_args: "--use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql'" + python_version: '3.8' + - run_dtests: + file_tag: j8_dtests_offheap + pytest_extra_args: '--use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8' + python_version: '3.8' + + j8_cqlsh_dtests_py311_offheap: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv: + python_version: '3.11' + - create_dtest_containers: + file_tag: j8_dtests_offheap + run_dtests_extra_args: "--use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql'" + python_version: '3.11' + - run_dtests: + file_tag: j8_dtests_offheap + pytest_extra_args: '--use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.11' + python_version: '3.11' + + j8_cqlsh-dtests-py2-no-vnodes: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j8_without_vnodes + run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" + - run_dtests: + file_tag: j8_without_vnodes + pytest_extra_args: '--skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' + + j8_cqlsh_dtests_py3: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j8_without_vnodes + run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" + - run_dtests: + file_tag: j8_without_vnodes + pytest_extra_args: '--skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' + + j8_cqlsh_dtests_py38: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv: + python_version: '3.8' + - create_dtest_containers: + file_tag: j8_without_vnodes + run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" + python_version: '3.8' + - run_dtests: + file_tag: j8_without_vnodes + pytest_extra_args: '--skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8' + python_version: '3.8' + + j8_cqlsh_dtests_py311: + <<: *j8_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv: + python_version: '3.11' + - create_dtest_containers: + file_tag: j8_without_vnodes + run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" + python_version: '3.11' + - run_dtests: + file_tag: j8_without_vnodes + pytest_extra_args: '--skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.11' + python_version: '3.11' + + j11_cqlsh-dtests-py2-with-vnodes: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j11_with_vnodes + run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" + - run_dtests: + file_tag: j11_with_vnodes + pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' + + j11_cqlsh-dtests-py2-offheap: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j11_dtests_offheap + run_dtests_extra_args: "--use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql'" + - run_dtests: + file_tag: j11_dtests_offheap + pytest_extra_args: '--use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' + + j11_cqlsh_dtests_py3_vnode: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j11_with_vnodes + run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" + - run_dtests: + file_tag: j11_with_vnodes + pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' + + j11_cqlsh_dtests_py3_offheap: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j11_dtests_offheap + run_dtests_extra_args: "--use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql'" + - run_dtests: + file_tag: j11_dtests_offheap + pytest_extra_args: '--use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' + + j11_cqlsh_dtests_py38_vnode: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv: + python_version: '3.8' + - create_dtest_containers: + file_tag: j11_with_vnodes + run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" + python_version: '3.8' + - run_dtests: + file_tag: j11_with_vnodes + pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8' + python_version: '3.8' + + j11_cqlsh_dtests_py311_vnode: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv: + python_version: '3.11' + - create_dtest_containers: + file_tag: j11_with_vnodes + run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'" + python_version: '3.11' + - run_dtests: + file_tag: j11_with_vnodes + pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.11' + python_version: '3.11' + + j11_cqlsh_dtests_py38_offheap: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv: + python_version: '3.8' + - create_dtest_containers: + file_tag: j11_dtests_offheap + run_dtests_extra_args: "--use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql'" + python_version: '3.8' + - run_dtests: + file_tag: j11_dtests_offheap + pytest_extra_args: '--use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8' + python_version: '3.8' + + j11_cqlsh_dtests_py311_offheap: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv: + python_version: '3.11' + - create_dtest_containers: + file_tag: j11_dtests_offheap + run_dtests_extra_args: "--use-vnodes --use-off-heap-memtables --skip-resource-intensive-tests --pytest-options '-k cql'" + python_version: '3.11' + - run_dtests: + file_tag: j11_dtests_offheap + pytest_extra_args: '--use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.11' + python_version: '3.11' + + j11_cqlsh-dtests-py2-no-vnodes: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j11_without_vnodes + run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" + - run_dtests: + file_tag: j11_without_vnodes + pytest_extra_args: '--skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' + + j11_cqlsh_dtests_py3: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - create_dtest_containers: + file_tag: j11_without_vnodes + run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" + - run_dtests: + file_tag: j11_without_vnodes + pytest_extra_args: '--skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' + + j11_cqlsh_dtests_py38: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv: + python_version: '3.8' + - create_dtest_containers: + file_tag: j11_without_vnodes + run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" + python_version: '3.8' + - run_dtests: + file_tag: j11_without_vnodes + pytest_extra_args: '--skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8' + python_version: '3.8' + + j11_cqlsh_dtests_py311: + <<: *j11_par_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv: + python_version: '3.11' + - create_dtest_containers: + file_tag: j11_without_vnodes + run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'" + python_version: '3.11' + - run_dtests: + file_tag: j11_without_vnodes + pytest_extra_args: '--skip-resource-intensive-tests' + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.11' + python_version: '3.11' + + j8_unit_tests_repeat: + <<: *j8_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_unit_tests_repeat + + j11_unit_tests_repeat: + <<: *j11_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_unit_tests_repeat + + j8_utests_cdc_repeat: + <<: *j8_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_utests_cdc_repeat + + j11_utests_cdc_repeat: + <<: *j11_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_utests_cdc_repeat + + j8_utests_compression_repeat: + <<: *j8_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_utests_compression_repeat + + j11_utests_compression_repeat: + <<: *j11_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_utests_compression_repeat + + j8_utests_system_keyspace_directory_repeat: + <<: *j8_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_utests_system_keyspace_directory_repeat + + j11_utests_system_keyspace_directory_repeat: + <<: *j11_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_utests_system_keyspace_directory_repeat + + j8_utests_fqltool_repeat: + <<: *j8_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_utests_fqltool_repeat + + j11_utests_fqltool_repeat: + <<: *j11_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_utests_fqltool_repeat + + j8_utests_long_repeat: + <<: *j8_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_utests_long_repeat + + j11_utests_long_repeat: + <<: *j11_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_utests_long_repeat + + j8_utests_stress_repeat: + <<: *j8_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_utests_stress_repeat + + j11_utests_stress_repeat: + <<: *j11_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_utests_stress_repeat + + j8_jvm_dtests_repeat: + <<: *j8_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_jvm_dtests_repeat + + + j8_dtests_large_repeat: + <<: *j8_repeated_dtest_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - run_repeated_dtest: + tests: ${REPEATED_LARGE_DTESTS} + vnodes: "false" + upgrade: "false" + count: ${REPEATED_LARGE_DTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + extra_dtest_args: "--only-resource-intensive-tests --force-resource-intensive-tests" + + j8_dtests_large_vnode_repeat: + <<: *j8_repeated_dtest_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - run_repeated_dtest: + tests: ${REPEATED_LARGE_DTESTS} + vnodes: "true" + upgrade: "false" + count: ${REPEATED_LARGE_DTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + extra_dtest_args: "--only-resource-intensive-tests --force-resource-intensive-tests" + + j11_dtests_large_repeat: + <<: *j11_repeated_dtest_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - run_repeated_dtest: + tests: ${REPEATED_LARGE_DTESTS} + vnodes: "false" + upgrade: "false" + count: ${REPEATED_LARGE_DTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + extra_dtest_args: "--only-resource-intensive-tests --force-resource-intensive-tests" + + j11_dtests_large_vnode_repeat: + <<: *j11_repeated_dtest_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - run_repeated_dtest: + tests: ${REPEATED_LARGE_DTESTS} + vnodes: "true" + upgrade: "false" + count: ${REPEATED_LARGE_DTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + extra_dtest_args: "--only-resource-intensive-tests --force-resource-intensive-tests" + + j8_jvm_upgrade_dtests_repeat: + <<: *j8_repeated_jvm_upgrade_dtest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_jvm_upgrade_dtests_repeat + + j11_jvm_dtests_repeat: + <<: *j11_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_jvm_dtests_repeat + + j8_repeated_ant_test: + <<: *j8_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_repeated_utest: + target: ${REPEATED_ANT_TEST_TARGET} + class: ${REPEATED_ANT_TEST_CLASS} + methods: ${REPEATED_ANT_TEST_METHODS} + count: ${REPEATED_ANT_TEST_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + j11_repeated_ant_test: + <<: *j11_repeated_utest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - run_repeated_utest: + target: ${REPEATED_ANT_TEST_TARGET} + class: ${REPEATED_ANT_TEST_CLASS} + methods: ${REPEATED_ANT_TEST_METHODS} + count: ${REPEATED_ANT_TEST_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + j8_dtests_repeat: + <<: *j8_repeated_dtest_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - run_repeated_dtest: + tests: ${REPEATED_DTESTS} + vnodes: "false" + upgrade: "false" + count: ${REPEATED_DTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + j8_dtests_vnode_repeat: + <<: *j8_repeated_dtest_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - run_repeated_dtest: + tests: ${REPEATED_DTESTS} + vnodes: "true" + upgrade: "false" + count: ${REPEATED_DTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + j8_dtests_offheap_repeat: + <<: *j8_repeated_dtest_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - run_repeated_dtest: + tests: ${REPEATED_DTESTS} + vnodes: "true" + upgrade: "false" + count: ${REPEATED_DTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + extra_dtest_args: "--use-off-heap-memtables --skip-resource-intensive-tests" + + j11_dtests_repeat: + <<: *j11_repeated_dtest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - clone_dtest + - create_venv + - run_repeated_dtest: + tests: ${REPEATED_DTESTS} + vnodes: "false" + upgrade: "false" + count: ${REPEATED_DTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + j11_dtests_vnode_repeat: + <<: *j11_repeated_dtest_executor + steps: + - attach_workspace: + at: /home/cassandra + - log_environment + - clone_dtest + - create_venv + - run_repeated_dtest: + tests: ${REPEATED_DTESTS} + vnodes: "true" + upgrade: "false" + count: ${REPEATED_DTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + j11_dtests_offheap_repeat: + <<: *j11_repeated_dtest_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - run_repeated_dtest: + tests: ${REPEATED_DTESTS} + vnodes: "true" + upgrade: "false" + count: ${REPEATED_DTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + extra_dtest_args: "--use-off-heap-memtables --skip-resource-intensive-tests" + + j8_upgrade_dtests_repeat: + <<: *j8_repeated_upgrade_dtest_executor + steps: + - attach_workspace: + at: /home/cassandra + - clone_dtest + - create_venv + - run_repeated_dtest: + tests: ${REPEATED_UPGRADE_DTESTS} + vnodes: "false" + upgrade: "true" + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + count: ${REPEATED_UPGRADE_DTESTS_COUNT} + +commands: + log_environment: + steps: + - run: + name: Log Environment Information + command: | + echo '*** id ***' + id + echo '*** cat /proc/cpuinfo ***' + cat /proc/cpuinfo + echo '*** free -m ***' + free -m + echo '*** df -m ***' + df -m + echo '*** ifconfig -a ***' + ifconfig -a + echo '*** uname -a ***' + uname -a + echo '*** mount ***' + mount + echo '*** env ***' + env + echo '*** java ***' + which java + java -version + + clone_cassandra: + steps: + - run: + name: Clone Cassandra Repository (via git) + command: | + git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH https://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra + + clone_dtest: + steps: + - run: + name: Clone Cassandra dtest Repository (via git) + command: | + git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest + + build_cassandra: + steps: + - run: + name: Build Cassandra + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. + for x in $(seq 1 3); do + ${ANT_HOME}/bin/ant clean realclean jar + RETURN="$?" + if [ "${RETURN}" -eq "0" ]; then + break + fi + done + # Exit, if we didn't build successfully + if [ "${RETURN}" -ne "0" ]; then + echo "Build failed with exit code: ${RETURN}" + exit ${RETURN} + fi + no_output_timeout: 15m + + build_cassandra_dtest_jars: + steps: + - run: + name: Build Cassandra DTest jars + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + mkdir ~/dtest_jars + git remote add apache https://github.com/apache/cassandra.git + for branch in cassandra-2.2 cassandra-3.0 cassandra-3.11 cassandra-4.0 trunk; do + # check out the correct cassandra version: + git remote set-branches --add apache '$branch' + git fetch --depth 1 apache $branch + git checkout $branch + git clean -fd + # Loop to prevent failure due to maven-ant-tasks not downloading a jar.. + for x in $(seq 1 3); do + ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar + RETURN="$?" + if [ "${RETURN}" -eq "0" ]; then + cp build/dtest*.jar ~/dtest_jars + break + fi + done + # Exit, if we didn't build successfully + if [ "${RETURN}" -ne "0" ]; then + echo "Build failed with exit code: ${RETURN}" + exit ${RETURN} + fi + done + # and build the dtest-jar for the branch under test + ${ANT_HOME}/bin/ant realclean + git checkout origin/$CIRCLE_BRANCH + git clean -fd + for x in $(seq 1 3); do + ${ANT_HOME}/bin/ant realclean; ${ANT_HOME}/bin/ant jar dtest-jar + RETURN="$?" + if [ "${RETURN}" -eq "0" ]; then + cp build/dtest*.jar ~/dtest_jars + break + fi + done + # Exit, if we didn't build successfully + if [ "${RETURN}" -ne "0" ]; then + echo "Build failed with exit code: ${RETURN}" + exit ${RETURN} + fi + ls -l ~/dtest_jars + no_output_timeout: 15m + + run_eclipse_warnings: + steps: + - run: + name: Run eclipse-warnings + command: | + export PATH=$JAVA_HOME/bin:$PATH + cd ~/cassandra + ant eclipse-warnings + + create_junit_containers: + parameters: + classlistprefix: + type: string + default: unit + extra_filters: + type: string + default: "" + steps: + - run: + name: Determine <> Tests to Run + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + rm -fr ~/cassandra-dtest/upgrade_tests + echo "***java tests***" + + # get all of our unit test filenames + set -eo pipefail && circleci tests glob "$HOME/cassandra/test/<>/**/*.java" > /tmp/all_java_unit_tests.txt + + # split up the unit tests into groups based on the number of containers we have + set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt + set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/<>/;;g" | grep "Test\.java$" <> > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt" + cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt + + no_output_timeout: 15m + + run_junit_tests: + parameters: + target: + type: string + no_output_timeout: + type: string + default: 15m + steps: + - run: + name: Run Unit Tests (<>) + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + ant <> -Dno-build-test=true + no_output_timeout: <> + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + + run_cqlshlib_tests: + parameters: + no_output_timeout: + type: string + default: 15m + steps: + - run: + name: Run cqlshlib Unit Tests + command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra/pylib + ./cassandra-cqlsh-tests.sh .. + no_output_timeout: <> + - store_test_results: + path: /tmp/cassandra/pylib + + run_parallel_junit_tests: + parameters: + target: + type: string + default: testclasslist + no_output_timeout: + type: string + default: 15m + classlistprefix: + type: string + default: unit + steps: + - run: + name: Run Unit Tests (<>) + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + test_timeout=$(grep 'name="test.<>.timeout"' build.xml | awk -F'"' '{print $4}' || true) + if [ -z "$test_timeout" ]; then + test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}') + fi + ant <> -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=<> -Dno-build-test=true + no_output_timeout: <> + - store_test_results: + path: /tmp/cassandra/build/test/output/ + - store_artifacts: + path: /tmp/cassandra/build/test/output + destination: junitxml + - store_artifacts: + path: /tmp/cassandra/build/test/logs + destination: logs + + create_venv: + parameters: + python_version: + type: enum + default: "3.6" + enum: ["3.6", "3.7", "3.8", "3.11"] + steps: + - run: + name: Configure virtualenv and python Dependencies + command: | + # note, this should be super quick as all dependencies should be pre-installed in the docker image + # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated + # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and + # rebuild the docker image! (it automatically pulls the latest requirements.txt on build) + source ~/env<>/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt + pip3 uninstall -y cqlsh + pip3 freeze + + create_dtest_containers: + parameters: + file_tag: + type: string + run_dtests_extra_args: + type: string + default: '' + extra_env_args: + type: string + default: '' + tests_filter_pattern: + type: string + default: '' + python_version: + type: enum + default: "3.6" + enum: ["3.6", "3.7", "3.8", "3.11"] + steps: + - run: + name: Determine Tests to Run (<>) + no_output_timeout: 5m + command: | + # reminder: this code (along with all the steps) is independently executed on every circle container + # so the goal here is to get the circleci script to return the tests *this* container will run + # which we do via the `circleci` cli tool. + + cd cassandra-dtest + source ~/env<>/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + if [ -n '<>' ]; then + export <> + fi + + echo "***Collected DTests (<>)***" + set -eo pipefail && ./run_dtests.py <> --dtest-print-tests-only --dtest-print-tests-output=/tmp/all_dtest_tests_<>_raw --cassandra-dir=../cassandra + if [ -z '<>' ]; then + mv /tmp/all_dtest_tests_<>_raw /tmp/all_dtest_tests_<> + else + grep -e '<>' /tmp/all_dtest_tests_<>_raw > /tmp/all_dtest_tests_<> || { echo "Filter did not match any tests! Exiting build."; exit 0; } + fi + set -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_<> > /tmp/split_dtest_tests_<>.txt + cat /tmp/split_dtest_tests_<>.txt | tr '\n' ' ' > /tmp/split_dtest_tests_<>_final.txt + cat /tmp/split_dtest_tests_<>_final.txt + + run_dtests: + parameters: + file_tag: + type: string + pytest_extra_args: + type: string + default: '' + extra_env_args: + type: string + default: '' + python_version: + type: enum + default: "3.6" + enum: ["3.6", "3.7", "3.8", "3.11"] + steps: + - run: + name: Run dtests (<>) + no_output_timeout: 15m + command: | + echo "cat /tmp/split_dtest_tests_<>_final.txt" + cat /tmp/split_dtest_tests_<>_final.txt + + source ~/env<>/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + if [ -n '<>' ]; then + export <> + fi + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + export SPLIT_TESTS=`cat /tmp/split_dtest_tests_<>_final.txt` + if [ ! -z "$SPLIT_TESTS" ]; then + set -o pipefail && cd ~/cassandra-dtest && pytest <> --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_<>.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt + else + echo "Tune your parallelism, there are more containers than test classes. Nothing to do in this container" + (exit 1) + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest_<> + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_<>_logs + + run_unit_tests_repeat: + steps: + - run_repeated_utests: + target: testsome + tests: ${REPEATED_UTESTS} + count: ${REPEATED_UTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + run_utests_cdc_repeat: + steps: + - run_repeated_utests: + target: test-cdc + tests: ${REPEATED_UTESTS} + count: ${REPEATED_UTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + run_utests_compression_repeat: + steps: + - run_repeated_utests: + target: test-compression + tests: ${REPEATED_UTESTS} + count: ${REPEATED_UTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + run_utests_system_keyspace_directory_repeat: + steps: + - run_repeated_utests: + target: test-system-keyspace-directory + tests: ${REPEATED_UTESTS} + count: ${REPEATED_UTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + run_utests_long_repeat: + steps: + - run_repeated_utests: + target: long-testsome + tests: ${REPEATED_UTESTS_LONG} + count: ${REPEATED_UTESTS_LONG_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + run_utests_fqltool_repeat: + steps: + - run_repeated_utests: + target: fqltool-test + tests: ${REPEATED_UTESTS_FQLTOOL} + count: ${REPEATED_UTESTS_FQLTOOL_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + run_utests_stress_repeat: + steps: + - run_repeated_utests: + target: stress-test-some + tests: ${REPEATED_UTESTS_STRESS} + count: ${REPEATED_UTESTS_STRESS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + run_jvm_dtests_repeat: + steps: + - run_repeated_utests: + target: test-jvm-dtest-some + tests: ${REPEATED_JVM_DTESTS} + count: ${REPEATED_JVM_DTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + run_jvm_upgrade_dtests_repeat: + steps: + - run_repeated_utests: + target: test-jvm-dtest-some + tests: ${REPEATED_JVM_UPGRADE_DTESTS} + count: ${REPEATED_JVM_UPGRADE_DTESTS_COUNT} + stop_on_failure: ${REPEATED_TESTS_STOP_ON_FAILURE} + + run_repeated_utests: + parameters: + target: + type: string + tests: + type: string + count: + type: string + stop_on_failure: + type: string + steps: + - run: + name: Repeatedly run new or modifed JUnit tests + no_output_timeout: 15m + command: | + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + # Calculate the number of test iterations to be run by the current parallel runner. + count=$((<> / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (<> % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + # Put manually specified tests and automatically detected tests together, removing duplicates + tests=$(echo <> | sed -e "s///" | sed -e "s/ //" | tr "," "\n" | tr " " "\n" | sort -n | uniq -u) + echo "Tests to be repeated: ${tests}" + + # Prepare the testtag for the target, used by the test macro in build.xml to group the output files + target=<> + testtag="" + if [[ $target == "test-cdc" ]]; then + testtag="cdc" + elif [[ $target == "test-compression" ]]; then + testtag="compression" + elif [[ $target == "test-system-keyspace-directory" ]]; then + testtag="system_keyspace_directory" + fi + + # Run each test class as many times as requested. + exit_code="$?" + for test in $tests; do + + # Split class and method names from the test name + if [[ $test =~ "#" ]]; then + class=${test%"#"*} + method=${test#*"#"} + else + class=$test + method="" + fi + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name_arg="-Dtest.name=${class##*.}" + else + name_arg="-Dtest.name=$class" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [[ $method == "" ]]; then + methods_arg="" + else + methods_arg="-Dtest.methods=$method" + fi + + for i in $(seq -w 1 $count); do + echo "Running test $test, iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && \ + ant <> $name_arg $methods_arg -Dno-build-test=true | \ + tee stdout.txt \ + ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utests/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/${test}.txt + + # move the XML output files + source=build/test/output/${testtag} + dest=/tmp/results/repeated_utests/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs/${testtag} + dest=/tmp/results/repeated_utests/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ <> = true ]] && (( $exit_code > 0 )); then + break + fi + done + done + (exit ${exit_code}) + - store_test_results: + path: /tmp/results/repeated_utests/output + - store_artifacts: + path: /tmp/results/repeated_utests/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utests/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utests/logs + destination: logs + + run_repeated_utest: + parameters: + target: + type: string + class: + type: string + methods: + type: string + count: + type: string + stop_on_failure: + type: string + steps: + - run: + name: Run repeated JUnit test + no_output_timeout: 15m + command: | + if [ "<>" == "" ]; then + echo "Repeated utest class name hasn't been defined, exiting without running any test" + elif [ "<>" == "" ]; then + echo "Repeated utest count hasn't been defined, exiting without running any test" + elif [ "<>" -le 0 ]; then + echo "Repeated utest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((<> / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (<> % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running <> <> <> <> times" + + set -x + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra + if [ -d ~/dtest_jars ]; then + cp ~/dtest_jars/dtest* /tmp/cassandra/build/ + fi + + target=<> + class_path=<> + class_name="${class_path##*.}" + + # Prepare the -Dtest.name argument. + # It can be the fully qualified class name or the short class name, depending on the target. + if [[ $target == "test" || \ + $target == "test-cdc" || \ + $target == "test-compression" || \ + $target == "test-system-keyspace-directory" || \ + $target == "fqltool-test" || \ + $target == "long-test" || \ + $target == "stress-test" ]]; then + name="-Dtest.name=$class_name" + else + name="-Dtest.name=$class_path" + fi + + # Prepare the -Dtest.methods argument, which is optional + if [ "<>" == "" ]; then + methods="" + else + methods="-Dtest.methods=<>" + fi + + # Run the test target as many times as requested collecting the exit code, + # stopping the iteration only if stop_on_failure is set. + exit_code="$?" + for i in $(seq -w 1 $count); do + + echo "Running test iteration $i of $count" + + # run the test + status="passes" + if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then + status="fails" + exit_code=1 + fi + + # move the stdout output file + dest=/tmp/results/repeated_utest/stdout/${status}/${i} + mkdir -p $dest + mv stdout.txt $dest/<>-<>.txt + + # move the XML output files + source=build/test/output + dest=/tmp/results/repeated_utest/output/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # move the log files + source=build/test/logs + dest=/tmp/results/repeated_utest/logs/${status}/${i} + mkdir -p $dest + if [[ -d $source && -n "$(ls $source)" ]]; then + mv $source/* $dest/ + fi + + # maybe stop iterations on test failure + if [[ <> = true ]] && (( $exit_code > 0 )); then + break + fi + done + + (exit ${exit_code}) + fi + fi + - store_test_results: + path: /tmp/results/repeated_utest/output + - store_artifacts: + path: /tmp/results/repeated_utest/stdout + destination: stdout + - store_artifacts: + path: /tmp/results/repeated_utest/output + destination: junitxml + - store_artifacts: + path: /tmp/results/repeated_utest/logs + destination: logs + + run_repeated_dtest: + parameters: + tests: + type: string + vnodes: + type: string + upgrade: + type: string + count: + type: string + stop_on_failure: + type: string + extra_dtest_args: + type: string + default: "" + steps: + - run: + name: Run repeated Python dtest + no_output_timeout: 15m + command: | + if [ "<>" == "" ]; then + echo "Repeated dtest name hasn't been defined, exiting without running any test" + elif [ "<>" == "" ]; then + echo "Repeated dtest count hasn't been defined, exiting without running any test" + elif [ "<>" -le 0 ]; then + echo "Repeated dtest count is lesser or equals than zero, exiting without running any test" + else + + # Calculate the number of test iterations to be run by the current parallel runner. + # Since we are running the same test multiple times there is no need to use `circleci tests split`. + count=$((<> / CIRCLE_NODE_TOTAL)) + if (($CIRCLE_NODE_INDEX < (<> % CIRCLE_NODE_TOTAL))); then + count=$((count+1)) + fi + + if (($count <= 0)); then + echo "No tests to run in this runner" + else + echo "Running <> $count times" + + source ~/env3.6/bin/activate + export PATH=$JAVA_HOME/bin:$PATH + + java -version + cd ~/cassandra-dtest + mkdir -p /tmp/dtest + + echo "env: $(env)" + echo "** done env" + mkdir -p /tmp/results/dtests + + tests_arg=$(echo <> | sed -e "s/,/ /g") + + stop_on_failure_arg="" + if <>; then + stop_on_failure_arg="-x" + fi + + vnodes_args="" + if <>; then + vnodes_args="--use-vnodes --num-tokens=16" + fi + + upgrade_arg="" + if <>; then + upgrade_arg="--execute-upgrade-tests --upgrade-target-version-only --upgrade-version-selection all" + fi + + # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee + set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir <> $tests_arg | tee /tmp/dtest/stdout.txt + fi + fi + - store_test_results: + path: /tmp/results + - store_artifacts: + path: /tmp/dtest + destination: dtest + - store_artifacts: + path: ~/cassandra-dtest/logs + destination: dtest_logs diff --git a/.circleci/config-2_1.yml.mid_res.patch b/.circleci/config_template.yml.PAID.patch similarity index 51% rename from .circleci/config-2_1.yml.mid_res.patch rename to .circleci/config_template.yml.PAID.patch index 43a684651a60..a39b6ec0671f 100644 --- a/.circleci/config-2_1.yml.mid_res.patch +++ b/.circleci/config_template.yml.PAID.patch @@ -1,6 +1,6 @@ ---- config-2_1.yml 2022-05-30 12:01:09.000000000 -0400 -+++ config-2_1.yml.MIDRES 2022-05-30 12:02:51.000000000 -0400 -@@ -101,14 +101,14 @@ +--- config-2_1.yml 2023-02-07 21:22:17.000000000 -0500 ++++ config-2_1.yml.MIDRES 2023-02-08 10:15:45.000000000 -0500 +@@ -145,14 +145,14 @@ j8_par_executor: &j8_par_executor executor: name: java8-executor @@ -19,13 +19,13 @@ j8_small_executor: &j8_small_executor executor: -@@ -119,26 +119,38 @@ +@@ -163,26 +163,38 @@ j8_medium_par_executor: &j8_medium_par_executor executor: name: java8-executor - #exec_resource_class: xlarge - parallelism: 1 -+ exec_resource_class: large ++ exec_resource_class: xlarge + parallelism: 4 + +j8_large_par_executor: &j8_large_par_executor @@ -65,16 +65,28 @@ j11_small_executor: &j11_small_executor executor: -@@ -146,35 +158,47 @@ - #exec_resource_class: medium - parallelism: 1 - +@@ -193,44 +205,56 @@ + j11_medium_par_executor: &j11_medium_par_executor + executor: + name: java11-executor +- #exec_resource_class: xlarge +- parallelism: 1 ++ exec_resource_class: xlarge ++ parallelism: 4 ++ +j11_large_par_executor: &j11_large_par_executor + executor: + name: java11-executor + exec_resource_class: large + parallelism: 50 -+ + + j11_seq_executor: &j11_seq_executor + executor: + name: java11-executor +- #exec_resource_class: xlarge ++ exec_resource_class: medium + parallelism: 1 # sequential, single container tests: no parallelism benefits + j8_repeated_utest_executor: &j8_repeated_utest_executor executor: name: java8-executor @@ -117,54 +129,108 @@ + exec_resource_class: large + parallelism: 25 - j8_with_dtests_jobs: &j8_with_dtests_jobs + j8_separate_jobs: &j8_separate_jobs jobs: -@@ -869,7 +893,7 @@ +@@ -1700,7 +1724,7 @@ target: testclasslist-system-keyspace-directory - j8_dtests-with-vnodes: + j8_dtests_vnode: - <<: *j8_par_executor + <<: *j8_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -883,7 +907,7 @@ +@@ -1714,7 +1738,7 @@ pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' - j11_dtests-with-vnodes: + j8_dtests_offheap: +- <<: *j8_par_executor ++ <<: *j8_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -1728,7 +1752,7 @@ + pytest_extra_args: '--use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests' + + j11_dtests_vnode: - <<: *j11_par_executor + <<: *j11_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -898,7 +922,7 @@ +@@ -1743,7 +1767,7 @@ pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests' - j8_dtests-no-vnodes: + j11_dtests_offheap: +- <<: *j11_par_executor ++ <<: *j11_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -1758,7 +1782,7 @@ + pytest_extra_args: '--use-vnodes --num-tokens=16 --use-off-heap-memtables --skip-resource-intensive-tests' + + j8_dtests: - <<: *j8_par_executor + <<: *j8_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -912,7 +936,7 @@ +@@ -1772,7 +1796,7 @@ pytest_extra_args: '--skip-resource-intensive-tests' - j11_dtests-no-vnodes: + j11_dtests: - <<: *j11_par_executor + <<: *j11_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -927,7 +951,7 @@ +@@ -1787,7 +1811,7 @@ pytest_extra_args: '--skip-resource-intensive-tests' - j8_upgradetests-no-vnodes: + j8_dtests_large_vnode: +- <<: *j8_par_executor ++ <<: *j8_medium_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -1801,7 +1825,7 @@ + pytest_extra_args: '--use-vnodes --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests' + + j8_dtests_large: +- <<: *j8_par_executor ++ <<: *j8_medium_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -1815,7 +1839,7 @@ + pytest_extra_args: '--num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests' + + j11_dtests_large_vnode: +- <<: *j11_par_executor ++ <<: *j11_medium_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -1829,7 +1853,7 @@ + pytest_extra_args: '--use-vnodes --num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests' + + j11_dtests_large: +- <<: *j11_par_executor ++ <<: *j11_medium_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -1843,7 +1867,7 @@ + pytest_extra_args: '--num-tokens=16 --only-resource-intensive-tests --force-resource-intensive-tests' + + j8_upgrade_dtests: - <<: *j8_par_executor + <<: *j8_very_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -941,7 +965,7 @@ +@@ -1857,7 +1881,7 @@ pytest_extra_args: '--execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all' j8_cqlsh-dtests-py2-with-vnodes: @@ -173,106 +239,214 @@ steps: - attach_workspace: at: /home/cassandra -@@ -956,7 +980,7 @@ +@@ -1872,7 +1896,7 @@ + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' + + j8_cqlsh-dtests-py2-offheap: +- <<: *j8_par_executor ++ <<: *j8_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -1887,7 +1911,7 @@ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' - j8_cqlsh-dtests-py3-with-vnodes: + j8_cqlsh_dtests_py3_vnode: - <<: *j8_par_executor + <<: *j8_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -971,7 +995,7 @@ +@@ -1902,7 +1926,7 @@ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' - j8_cqlsh-dtests-py38-with-vnodes: + j8_cqlsh_dtests_py3_offheap: - <<: *j8_par_executor + <<: *j8_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -989,7 +1013,7 @@ +@@ -1917,7 +1941,7 @@ + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' + + j8_cqlsh_dtests_py38_vnode: +- <<: *j8_par_executor ++ <<: *j8_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -1935,7 +1959,7 @@ python_version: '3.8' + j8_cqlsh_dtests_py311_vnode: +- <<: *j8_par_executor ++ <<: *j8_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -1953,7 +1977,7 @@ + python_version: '3.11' + + j8_cqlsh_dtests_py38_offheap: +- <<: *j8_par_executor ++ <<: *j8_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -1971,7 +1995,7 @@ + python_version: '3.8' + + j8_cqlsh_dtests_py311_offheap: +- <<: *j8_par_executor ++ <<: *j8_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -1989,7 +2013,7 @@ + python_version: '3.11' + j8_cqlsh-dtests-py2-no-vnodes: - <<: *j8_par_executor + <<: *j8_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -1004,7 +1028,7 @@ +@@ -2004,7 +2028,7 @@ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' - j8_cqlsh-dtests-py3-no-vnodes: + j8_cqlsh_dtests_py3: - <<: *j8_par_executor + <<: *j8_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -1019,7 +1043,7 @@ +@@ -2019,7 +2043,7 @@ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' - j8_cqlsh-dtests-py38-no-vnodes: + j8_cqlsh_dtests_py38: - <<: *j8_par_executor + <<: *j8_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -1037,7 +1061,7 @@ +@@ -2037,7 +2061,7 @@ python_version: '3.8' + j8_cqlsh_dtests_py311: +- <<: *j8_par_executor ++ <<: *j8_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -2055,7 +2079,7 @@ + python_version: '3.11' + j11_cqlsh-dtests-py2-with-vnodes: - <<: *j11_par_executor + <<: *j11_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -1052,7 +1076,7 @@ +@@ -2070,7 +2094,7 @@ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' - j11_cqlsh-dtests-py3-with-vnodes: + j11_cqlsh-dtests-py2-offheap: +- <<: *j11_par_executor ++ <<: *j11_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -2085,7 +2109,7 @@ + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' + + j11_cqlsh_dtests_py3_vnode: +- <<: *j11_par_executor ++ <<: *j11_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -2100,7 +2124,7 @@ + extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' + + j11_cqlsh_dtests_py3_offheap: - <<: *j11_par_executor + <<: *j11_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -1067,7 +1091,7 @@ +@@ -2115,7 +2139,7 @@ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' - j11_cqlsh-dtests-py38-with-vnodes: + j11_cqlsh_dtests_py38_vnode: +- <<: *j11_par_executor ++ <<: *j11_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -2133,7 +2157,7 @@ + python_version: '3.8' + + j11_cqlsh_dtests_py311_vnode: +- <<: *j11_par_executor ++ <<: *j11_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -2151,7 +2175,7 @@ + python_version: '3.11' + + j11_cqlsh_dtests_py38_offheap: - <<: *j11_par_executor + <<: *j11_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -1085,7 +1109,7 @@ +@@ -2169,7 +2193,7 @@ python_version: '3.8' + j11_cqlsh_dtests_py311_offheap: +- <<: *j11_par_executor ++ <<: *j11_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -2187,7 +2211,7 @@ + python_version: '3.11' + j11_cqlsh-dtests-py2-no-vnodes: - <<: *j11_par_executor + <<: *j11_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -1100,7 +1124,7 @@ +@@ -2202,7 +2226,7 @@ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7' - j11_cqlsh-dtests-py3-no-vnodes: + j11_cqlsh_dtests_py3: - <<: *j11_par_executor + <<: *j11_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -1115,7 +1139,7 @@ +@@ -2217,7 +2241,7 @@ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6' - j11_cqlsh-dtests-py38-no-vnodes: + j11_cqlsh_dtests_py38: +- <<: *j11_par_executor ++ <<: *j11_large_par_executor + steps: + - attach_workspace: + at: /home/cassandra +@@ -2235,7 +2259,7 @@ + python_version: '3.8' + + j11_cqlsh_dtests_py311: - <<: *j11_par_executor + <<: *j11_large_par_executor steps: - attach_workspace: at: /home/cassandra -@@ -1757,3 +1781,4 @@ +@@ -3336,3 +3360,4 @@ - store_artifacts: path: ~/cassandra-dtest/logs destination: dtest_logs diff --git a/.circleci/generate.sh b/.circleci/generate.sh index 12e56ec95a04..1e5742d5c66c 100755 --- a/.circleci/generate.sh +++ b/.circleci/generate.sh @@ -1,4 +1,4 @@ -#!/bin/sh +#!/bin/bash # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -18,6 +18,8 @@ # BASEDIR=`dirname $0` +BASE_BRANCH=cassandra-4.0 +set -e die () { @@ -28,48 +30,65 @@ die () print_help() { - echo "Usage: $0 [-l|-m|-h|-f|-e]" - echo " -a Generate the default config.yml using low resources and the three templates" - echo " (config.yml.LOWRES, config.yml.MIDRES and config.yml.HIGHRES). Use this for" - echo " permanent changes in config-2_1.yml that will be committed to the main repo." - echo " -l Generate config.yml using low resources" - echo " -m Generate config.yml using mid resources" - echo " -h Generate config.yml using high resources" + echo "Usage: $0 [-f|-p|-a|-e|-i|-b|-s]" + echo " -a Generate the config.yml, config.yml.FREE and config.yml.PAID expanded configuration" + echo " files from the main config_template.yml reusable configuration file." + echo " Use this for permanent changes in config that will be committed to the main repo." + echo " -f Generate config.yml for tests compatible with the CircleCI free tier resources" + echo " -p Generate config.yml for tests compatible with the CircleCI paid tier resources" + echo " -b Specify the base git branch for comparison when determining changed tests to" + echo " repeat. Defaults to ${BASE_BRANCH}. Note that this option is not used when" + echo " the '-a' option is specified." + echo " -s Skip automatic detection of changed tests. Useful when you need to repeat a few ones," + echo " or when there are too many changed tests for CircleCI." echo " -e Environment variables to be used in the generated config.yml, e.g.:" echo " -e DTEST_BRANCH=CASSANDRA-8272" echo " -e DTEST_REPO=https://github.com/adelapena/cassandra-dtest.git" - echo " -e REPEATED_UTEST_TARGET=testsome" - echo " -e REPEATED_UTEST_CLASS=org.apache.cassandra.cql3.ViewTest" - echo " -e REPEATED_UTEST_METHODS=testCompoundPartitionKey,testStaticTable" - echo " -e REPEATED_UTEST_COUNT=100" - echo " -e REPEATED_UTEST_STOP_ON_FAILURE=false" - echo " -e REPEATED_DTEST_NAME=cqlsh_tests/test_cqlsh.py::TestCqlshSmoke" - echo " -e REPEATED_DTEST_VNODES=false" - echo " -e REPEATED_DTEST_COUNT=100" - echo " -e REPEATED_DTEST_STOP_ON_FAILURE=false" + echo " -e REPEATED_TESTS_STOP_ON_FAILURE=false" + echo " -e REPEATED_UTESTS=org.apache.cassandra.cql3.ViewTest#testCountersTable" + echo " -e REPEATED_UTESTS_COUNT=500" + echo " -e REPEATED_UTESTS_FQLTOOL=org.apache.cassandra.fqltool.FQLCompareTest" + echo " -e REPEATED_UTESTS_FQLTOOL_COUNT=500" + echo " -e REPEATED_UTESTS_LONG=org.apache.cassandra.db.commitlog.CommitLogStressTest" + echo " -e REPEATED_UTESTS_LONG_COUNT=100" + echo " -e REPEATED_UTESTS_STRESS=org.apache.cassandra.stress.generate.DistributionGaussianTest" + echo " -e REPEATED_UTESTS_STRESS_COUNT=500" + echo " -e REPEATED_JVM_DTESTS=org.apache.cassandra.distributed.test.PagingTest" + echo " -e REPEATED_JVM_DTESTS_COUNT=500" + echo " -e REPEATED_JVM_UPGRADE_DTESTS=org.apache.cassandra.distributed.upgrade.GroupByTest" + echo " -e REPEATED_JVM_UPGRADE_DTESTS_COUNT=500" + echo " -e REPEATED_DTESTS=cdc_test.py cqlsh_tests/test_cqlsh.py::TestCqlshSmoke" + echo " -e REPEATED_DTESTS_COUNT=500" + echo " -e REPEATED_LARGE_DTESTS=replace_address_test.py::TestReplaceAddress::test_replace_stopped_node" + echo " -e REPEATED_LARGE_DTESTS=100" + echo " -e REPEATED_UPGRADE_DTESTS=upgrade_tests/cql_tests.py upgrade_tests/paging_test.py" + echo " -e REPEATED_UPGRADE_DTESTS_COUNT=25" + echo " -e REPEATED_ANT_TEST_TARGET=testsome" + echo " -e REPEATED_ANT_TEST_CLASS=org.apache.cassandra.cql3.ViewTest" + echo " -e REPEATED_ANT_TEST_METHODS=testCompoundPartitionKey,testStaticTable" + echo " -e REPEATED_ANT_TEST_COUNT=500" echo " For the complete list of environment variables, please check the" - echo " list of examples in config-2_1.yml and/or the documentation." + echo " list of examples in config_template.yml and/or the documentation." echo " If you want to specify multiple environment variables simply add" - echo " multiple -e options. The flags -l/-m/-h should be used when using -e." - echo " -f Stop checking that the environment variables are known" + echo " multiple -e options. The flags -f/-p should be used when using -e." + echo " -i Ignore unknown environment variables" } all=false -lowres=false -midres=false -highres=false +free=false +paid=false env_vars="" has_env_vars=false check_env_vars=true -while getopts "e:almhf" opt; do +detect_changed_tests=true +while getopts "e:afpib:s" opt; do case $opt in a ) all=true + detect_changed_tests=false ;; - l ) lowres=true + f ) free=true ;; - m ) midres=true - ;; - h ) highres=true + p ) paid=true ;; e ) if (!($has_env_vars)); then env_vars="$OPTARG" @@ -78,7 +97,11 @@ while getopts "e:almhf" opt; do fi has_env_vars=true ;; - f ) check_env_vars=false + b ) BASE_BRANCH="$OPTARG" + ;; + i ) check_env_vars=false + ;; + s ) detect_changed_tests=false ;; \?) die "Invalid option: -$OPTARG" ;; @@ -95,88 +118,208 @@ if $has_env_vars && $check_env_vars; then key=$(echo $entry | tr "=" "\n" | head -n 1) if [ "$key" != "DTEST_REPO" ] && [ "$key" != "DTEST_BRANCH" ] && - [ "$key" != "REPEATED_UTEST_TARGET" ] && - [ "$key" != "REPEATED_UTEST_CLASS" ] && - [ "$key" != "REPEATED_UTEST_METHODS" ] && - [ "$key" != "REPEATED_UTEST_COUNT" ] && - [ "$key" != "REPEATED_UTEST_STOP_ON_FAILURE" ] && - [ "$key" != "REPEATED_DTEST_NAME" ] && - [ "$key" != "REPEATED_DTEST_VNODES" ] && - [ "$key" != "REPEATED_DTEST_COUNT" ] && - [ "$key" != "REPEATED_DTEST_STOP_ON_FAILURE" ] && - [ "$key" != "REPEATED_UPGRADE_DTEST_NAME" ] && - [ "$key" != "REPEATED_UPGRADE_DTEST_COUNT" ] && - [ "$key" != "REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE" ] && - [ "$key" != "REPEATED_JVM_UPGRADE_DTEST_CLASS" ] && - [ "$key" != "REPEATED_JVM_UPGRADE_DTEST_METHODS" ] && - [ "$key" != "REPEATED_JVM_UPGRADE_DTEST_COUNT" ] && - [ "$key" != "REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE" ]; then + [ "$key" != "REPEATED_TESTS_STOP_ON_FAILURE" ] && + [ "$key" != "REPEATED_UTESTS" ] && + [ "$key" != "REPEATED_UTESTS_COUNT" ] && + [ "$key" != "REPEATED_UTESTS_FQLTOOL" ] && + [ "$key" != "REPEATED_UTESTS_FQLTOOL_COUNT" ] && + [ "$key" != "REPEATED_UTESTS_LONG" ] && + [ "$key" != "REPEATED_UTESTS_LONG_COUNT" ] && + [ "$key" != "REPEATED_UTESTS_STRESS" ] && + [ "$key" != "REPEATED_UTESTS_STRESS_COUNT" ] && + [ "$key" != "REPEATED_JVM_DTESTS" ] && + [ "$key" != "REPEATED_JVM_DTESTS_COUNT" ] && + [ "$key" != "REPEATED_JVM_UPGRADE_DTESTS" ] && + [ "$key" != "REPEATED_JVM_UPGRADE_DTESTS_COUNT" ] && + [ "$key" != "REPEATED_DTESTS" ] && + [ "$key" != "REPEATED_DTESTS_COUNT" ] && + [ "$key" != "REPEATED_LARGE_DTESTS" ] && + [ "$key" != "REPEATED_LARGE_DTESTS_COUNT" ] && + [ "$key" != "REPEATED_UPGRADE_DTESTS" ] && + [ "$key" != "REPEATED_UPGRADE_DTESTS_COUNT" ] && + [ "$key" != "REPEATED_ANT_TEST_TARGET" ] && + [ "$key" != "REPEATED_ANT_TEST_CLASS" ] && + [ "$key" != "REPEATED_ANT_TEST_METHODS" ] && + [ "$key" != "REPEATED_ANT_TEST_COUNT" ]; then die "Unrecognised environment variable name: $key" fi done fi -if $lowres; then - ($all || $midres || $highres) && die "Cannot use option -l with options -a, -m or -h" - echo "Generating new config.yml file with low resources from config-2_1.yml" - circleci config process $BASEDIR/config-2_1.yml > $BASEDIR/config.yml.LOWRES.tmp - cat $BASEDIR/license.yml $BASEDIR/config.yml.LOWRES.tmp > $BASEDIR/config.yml - rm $BASEDIR/config.yml.LOWRES.tmp - -elif $midres; then - ($all || $lowres || $highres) && die "Cannot use option -m with options -a, -l or -h" - echo "Generating new config.yml file with middle resources from config-2_1.yml" - patch -o $BASEDIR/config-2_1.yml.MIDRES $BASEDIR/config-2_1.yml $BASEDIR/config-2_1.yml.mid_res.patch - circleci config process $BASEDIR/config-2_1.yml.MIDRES > $BASEDIR/config.yml.MIDRES.tmp - cat $BASEDIR/license.yml $BASEDIR/config.yml.MIDRES.tmp > $BASEDIR/config.yml - rm $BASEDIR/config-2_1.yml.MIDRES $BASEDIR/config.yml.MIDRES.tmp - -elif $highres; then - ($all || $lowres || $midres) && die "Cannot use option -h with options -a, -l or -m" - echo "Generating new config.yml file with high resources from config-2_1.yml" - patch -o $BASEDIR/config-2_1.yml.HIGHRES $BASEDIR/config-2_1.yml $BASEDIR/config-2_1.yml.high_res.patch - circleci config process $BASEDIR/config-2_1.yml.HIGHRES > $BASEDIR/config.yml.HIGHRES.tmp - cat $BASEDIR/license.yml $BASEDIR/config.yml.HIGHRES.tmp > $BASEDIR/config.yml - rm $BASEDIR/config-2_1.yml.HIGHRES $BASEDIR/config.yml.HIGHRES.tmp +if $free; then + ($all || $paid) && die "Cannot use option -f with options -a or -p" + echo "Generating new config.yml file for free tier from config_template.yml" + circleci config process $BASEDIR/config_template.yml > $BASEDIR/config.yml.FREE.tmp + cat $BASEDIR/license.yml $BASEDIR/config.yml.FREE.tmp > $BASEDIR/config.yml + rm $BASEDIR/config.yml.FREE.tmp -elif $all; then - ($lowres || $midres || $highres || $has_env_vars) && die "Cannot use option -a with options -l, -m, -h or -e" - echo "Generating new config.yml file with low resources and LOWRES/MIDRES/HIGHRES templates from config-2_1.yml" +elif $paid; then + ($all || $free) && die "Cannot use option -p with options -a or -f" + echo "Generating new config.yml file for paid tier from config_template.yml" + patch -o $BASEDIR/config_template.yml.PAID $BASEDIR/config_template.yml $BASEDIR/config_template.yml.PAID.patch + circleci config process $BASEDIR/config_template.yml.PAID > $BASEDIR/config.yml.PAID.tmp + cat $BASEDIR/license.yml $BASEDIR/config.yml.PAID.tmp > $BASEDIR/config.yml + rm $BASEDIR/config_template.yml.PAID $BASEDIR/config.yml.PAID.tmp - # setup lowres - circleci config process $BASEDIR/config-2_1.yml > $BASEDIR/config.yml.LOWRES.tmp - cat $BASEDIR/license.yml $BASEDIR/config.yml.LOWRES.tmp > $BASEDIR/config.yml.LOWRES - rm $BASEDIR/config.yml.LOWRES.tmp +elif $all; then + ($free || $paid || $has_env_vars) && die "Cannot use option -a with options -f, -p or -e" + echo "Generating new default config.yml file for free tier and FREE/PAID templates from config_template.yml." + echo "Make sure you commit the newly generated config.yml, config.yml.FREE and config.yml.PAID files" + echo "after running this command if you want them to persist." - # setup midres - patch -o $BASEDIR/config-2_1.yml.MIDRES $BASEDIR/config-2_1.yml $BASEDIR/config-2_1.yml.mid_res.patch - circleci config process $BASEDIR/config-2_1.yml.MIDRES > $BASEDIR/config.yml.MIDRES.tmp - cat $BASEDIR/license.yml $BASEDIR/config.yml.MIDRES.tmp > $BASEDIR/config.yml.MIDRES - rm $BASEDIR/config-2_1.yml.MIDRES $BASEDIR/config.yml.MIDRES.tmp + # setup config for free tier + circleci config process $BASEDIR/config_template.yml > $BASEDIR/config.yml.FREE.tmp + cat $BASEDIR/license.yml $BASEDIR/config.yml.FREE.tmp > $BASEDIR/config.yml.FREE + rm $BASEDIR/config.yml.FREE.tmp - # setup highres - patch -o $BASEDIR/config-2_1.yml.HIGHRES $BASEDIR/config-2_1.yml $BASEDIR/config-2_1.yml.high_res.patch - circleci config process $BASEDIR/config-2_1.yml.HIGHRES > $BASEDIR/config.yml.HIGHRES.tmp - cat $BASEDIR/license.yml $BASEDIR/config.yml.HIGHRES.tmp > $BASEDIR/config.yml.HIGHRES - rm $BASEDIR/config-2_1.yml.HIGHRES $BASEDIR/config.yml.HIGHRES.tmp + # setup config for paid tier + patch -o $BASEDIR/config_template.yml.PAID $BASEDIR/config_template.yml $BASEDIR/config_template.yml.PAID.patch + circleci config process $BASEDIR/config_template.yml.PAID > $BASEDIR/config.yml.PAID.tmp + cat $BASEDIR/license.yml $BASEDIR/config.yml.PAID.tmp > $BASEDIR/config.yml.PAID + rm $BASEDIR/config_template.yml.PAID $BASEDIR/config.yml.PAID.tmp - # copy lower into config.yml to make sure this gets updated - cp $BASEDIR/config.yml.LOWRES $BASEDIR/config.yml + # copy free tier into config.yml to make sure this gets updated + cp $BASEDIR/config.yml.FREE $BASEDIR/config.yml elif (!($has_env_vars)); then print_help + exit 0 +fi + +# add new or modified tests to the sets of tests to be repeated +if $detect_changed_tests; then + # Sanity check that the referenced branch exists + if ! git show ${BASE_BRANCH} -- >&/dev/null; then + echo -e "\n\nUnknown base branch: ${BASE_BRANCH}. Unable to detect changed tests.\n" + echo "Please use the '-b' option to choose an existing branch name" + echo "(e.g. origin/${BASE_BRANCH}, apache/${BASE_BRANCH}, etc.)." + exit 2 + fi + + add_diff_tests () + { + dir="${BASEDIR}/../${2}" + diff=$(git --no-pager diff --name-only --diff-filter=AMR ${BASE_BRANCH}...HEAD ${dir}) + tests=$( echo "$diff" \ + | grep "Test\\.java" \ + | sed -e "s/\\.java//" \ + | sed -e "s,^${2},," \ + | tr '/' '.' \ + | grep ${3} )\ + || : # avoid execution interruptions due to grep return codes and set -e + for test in $tests; do + echo " $test" + has_env_vars=true + if echo "$env_vars" | grep -q "${1}="; then + env_vars=$(echo "$env_vars" | sed -e "s/${1}=/${1}=${test},/") + elif [ -z "$env_vars" ]; then + env_vars="${1}=${test}" + else + env_vars="$env_vars|${1}=${test}" + fi + done + } + + echo + echo "Detecting new or modified tests with git diff --diff-filter=AMR ${BASE_BRANCH}...HEAD:" + add_diff_tests "REPEATED_UTESTS" "test/unit/" "org.apache.cassandra" + add_diff_tests "REPEATED_UTESTS_LONG" "test/long/" "org.apache.cassandra" + add_diff_tests "REPEATED_UTESTS_STRESS" "tools/stress/test/unit/" "org.apache.cassandra.stress" + add_diff_tests "REPEATED_UTESTS_FQLTOOL" "tools/fqltool/test/unit/" "org.apache.cassandra.fqltool" + add_diff_tests "REPEATED_JVM_DTESTS" "test/distributed/" "org.apache.cassandra.distributed.test" + add_diff_tests "REPEATED_JVM_UPGRADE_DTESTS" "test/distributed/" "org.apache.cassandra.distributed.upgrade" fi # replace environment variables if $has_env_vars; then + echo + echo "Setting environment variables:" IFS='=' echo "$env_vars" | tr '|' '\n' | while read entry; do set -- $entry key=$1 val=$2 - echo "Setting environment variable $key: $val" + echo " $key: $val" sed -i.bak "s|- $key:.*|- $key: $val|" $BASEDIR/config.yml done unset IFS fi +# Define function to remove unneeded jobs. +# The first argument is the file name, and the second arguemnt is the job name. +delete_job() +{ + delete_yaml_block() + { + sed -Ei.bak "/^ - ${2}/,/^ [^[:space:]]+|^ [^[:space:]]+/{//!d;}" "$1" + sed -Ei.bak "/^ - ${2}/d" "$1" + } + file="$BASEDIR/$1" + delete_yaml_block "$file" "${2}" + delete_yaml_block "$file" "start_${2}" +} + +# Define function to remove any unneeded repeated jobs. +# The first and only argument is the file name. +delete_repeated_jobs() +{ + if (! (echo "$env_vars" | grep -q "REPEATED_UTESTS=" )); then + delete_job "$1" "j8_unit_tests_repeat" + delete_job "$1" "j11_unit_tests_repeat" + delete_job "$1" "j8_utests_cdc_repeat" + delete_job "$1" "j11_utests_cdc_repeat" + delete_job "$1" "j8_utests_compression_repeat" + delete_job "$1" "j11_utests_compression_repeat" + delete_job "$1" "j8_utests_system_keyspace_directory_repeat" + delete_job "$1" "j11_utests_system_keyspace_directory_repeat" + fi + if (! (echo "$env_vars" | grep -q "REPEATED_UTESTS_LONG=")); then + delete_job "$1" "j8_utests_long_repeat" + delete_job "$1" "j11_utests_long_repeat" + fi + if (! (echo "$env_vars" | grep -q "REPEATED_UTESTS_STRESS=")); then + delete_job "$1" "j8_utests_stress_repeat" + delete_job "$1" "j11_utests_stress_repeat" + fi + if (! (echo "$env_vars" | grep -q "REPEATED_UTESTS_FQLTOOL=")); then + delete_job "$1" "j8_utests_fqltool_repeat" + delete_job "$1" "j11_utests_fqltool_repeat" + fi + if (! (echo "$env_vars" | grep -q "REPEATED_JVM_DTESTS=")); then + delete_job "$1" "j8_jvm_dtests_repeat" + delete_job "$1" "j8_jvm_dtests_vnode_repeat" + delete_job "$1" "j11_jvm_dtests_repeat" + delete_job "$1" "j11_jvm_dtests_vnode_repeat" + fi + if (! (echo "$env_vars" | grep -q "REPEATED_JVM_UPGRADE_DTESTS=")); then + delete_job "$1" "start_jvm_upgrade_dtests_repeat" + delete_job "$1" "j8_jvm_upgrade_dtests_repeat" + fi + if (! (echo "$env_vars" | grep -q "REPEATED_DTESTS=")); then + delete_job "$1" "j8_dtests_repeat" + delete_job "$1" "j8_dtests_vnode_repeat" + delete_job "$1" "j8_dtests_offheap_repeat" + delete_job "$1" "j11_dtests_repeat" + delete_job "$1" "j11_dtests_vnode_repeat" + delete_job "$1" "j11_dtests_offheap_repeat" + fi + if (! (echo "$env_vars" | grep -q "REPEATED_LARGE_DTESTS=")); then + delete_job "$1" "j8_dtests_large_repeat" + delete_job "$1" "j8_dtests_large_vnode_repeat" + delete_job "$1" "j11_dtests_large_repeat" + delete_job "$1" "j11_dtests_large_vnode_repeat" + fi + if (! (echo "$env_vars" | grep -q "REPEATED_UPGRADE_DTESTS=")); then + delete_job "$1" "j8_upgrade_dtests_repeat" + fi + if (! (echo "$env_vars" | grep -q "REPEATED_ANT_TEST_CLASS=")); then + delete_job "$1" "j8_repeated_ant_test" + delete_job "$1" "j11_repeated_ant_test" + fi +} + +delete_repeated_jobs "config.yml" +if $all; then + delete_repeated_jobs "config.yml.FREE" + delete_repeated_jobs "config.yml.PAID" +fi diff --git a/.circleci/readme.md b/.circleci/readme.md index ab7d5a9483cf..59bd98def31a 100644 --- a/.circleci/readme.md +++ b/.circleci/readme.md @@ -22,97 +22,167 @@ This directory contains the configuration for CircleCI continous integration platform. The file `config.yml` is the configuration file that is read by CircleCI. This file is -automatically generated by the `generate.sh` script from the `config-2_1.yml` file. +automatically generated by the `generate.sh` script from the `config_template.yml` file. The provided `config.yml` file uses low resources so users of the CircleCI free tier can -use it. Additionally, there are three versions of this file using different resources so +use it. Additionally, there are two versions of this file using different resources so users who have access to premium CircleCI resources can use larger instances and more -parallelism. These files are `config.yml.LOWRES`, `config.yml.MIDRES` and `config.yml.HIGHRES`. -The default `config.yml` file is just a copy of `config.yml.LOWRES`. +parallelism. These files are `config.yml.FREE` and `config.yml.PAID`. +The default `config.yml` file is just a copy of `config.yml.FREE`. ## Switching to higher resource settings -This directory contains generated files for low, middle and high resource settings. -Switch between them by copying the correct file to `config.yml` and committing the result: +This directory contains generated files for free and paid resource settings. +Switch between them by copying the correct file to `config.yml` and committing the result. +For example, for using higher resources benefiting from CircleCI's paid tier: -`cp .circleci/config.yml.HIGHRES .circleci/config.yml` +`cp .circleci/config.yml.PAID .circleci/config.yml` -Alternatively, you can run the `generate.sh` script with the flags `-l`/`-m`/`-h` -to regenerate the `config.yml` file from `config-2_1.yml` using LOWRES/MIDRES/HIGHRES. -This script validates and applies any changes to the `config-2_1.yml` file, and it +And for using lower resources comaptible with CircleCI's free tier: + +`cp .circleci/config.yml.FREE .circleci/config.yml` + +Alternatively, you can run the `generate.sh` script with the flags `-f`/`-p` +to regenerate the `config.yml` file from `config_template.yml` using free or paid resources. +This script validates and applies any changes to the `config_template.yml` file, and it requires the [CircleCI CLI](https://circleci.com/docs/2.0/local-cli/#install) to be installed. ## Setting environment variables -Both `config-2_1.yml` and `config.yml` files contain a set of environment variables -defining things like what dtest repo and branch to use, what tests could be repeatedly +Both `config_template.yml` and `config.yml` files contain a set of environment variables +defining things like what dtest repo and branch to use, what tests could be repeatedly run, etc. -These environment variables can be directly edited in the `config.yml` file, although if -you do this you should take into account that the entire set of env vars is repeated on -every job. +These environment variables can be directly edited in the `config.yml` file, although if +you do this you should take into account that the entire set of env vars is repeated on +every job. -A probably better approach is editing them in `config-2_1.yml` and then regenerate the +A probably better approach is editing them in `config_template.yml` and then regenerate the `config.yml` file using the `generate.sh` script. You can also directly pass environment -variable values to the `generate.sh` script with the `-e` flag. For example, to set the +variable values to the `generate.sh` script with the `-e` flag. For example, to set the dtest repo and branch with MIDRES config you can run: ``` -generate.sh -m \ +generate.sh -p \ -e DTEST_REPO=https://github.com/adelapena/cassandra-dtest.git \ - -e DTEST_BRANCH=CASSANDRA-8272 + -e DTEST_BRANCH=CASSANDRA-8272 + ``` -Or you can set the test multiplexer for repeating a specific test with HIGHRES: +## Running tests in a loop +Running the `generate.sh` script with use `git diff` to find the new or modified tests. +The script will then create jobs to run each of these new or modified tests for a certain +number of times, to verify that they are stable. You can use environment variables to +specify the number of iterations of each type of test: +``` +generate.sh -p \ + -e REPEATED_UTESTS_COUNT=500 \ + -e REPEATED_UTESTS_FQLTOOL_COUNT=500 \ + -e REPEATED_UTESTS_LONG_COUNT=100 \ + -e REPEATED_UTESTS_STRESS_COUNT=500 \ + -e REPEATED_SIMULATOR_DTESTS_COUNT=500 \ + -e REPEATED_JVM_DTESTS_COUNT=500 \ + -e REPEATED_JVM_UPGRADE_DTESTS_COUNT=500 \ + -e REPEATED_DTESTS_COUNT=500 \ + -e REPEATED_LARGE_DTESTS_COUNT=100 \ + -e REPEATED_UPGRADE_DTESTS_COUNT=25 \ + -e REPEATED_ANT_TEST_COUNT=500 +``` +You can also specify whether the iteration should fail on the first test failure: +``` +generate.sh -p -e REPEATED_TESTS_STOP_ON_FAILURE=false +``` +In addition to the automatically detected tests, it's also possible to provide lists of +specific tests to be repeated: +``` +generate.sh -p \ + -e REPEATED_UTESTS=org.apache.cassandra.cql3.ViewTest,org.apache.cassandra.db.CellTest \ + -e REPEATED_UTESTS_FQLTOOL=org.apache.cassandra.fqltool.FQLCompareTest \ + -e REPEATED_UTESTS_LONG=org.apache.cassandra.io.sstable.CQLSSTableWriterLongTest#testWideRow \ + -e REPEATED_UTESTS_STRESS=org.apache.cassandra.stress.generate.DistributionGaussianTest \ + -e REPEATED_DTESTS=cql_test.py,consistency_test.py::TestAvailability::test_simple_strategy \ + -e REPEATED_LARGE_DTESTS=replace_address_test.py::TestReplaceAddress::test_replace_stopped_node \ + -e REPEATED_JVM_DTESTS=org.apache.cassandra.distributed.test.PagingTest#testPaging \ + -e REPEATED_UPGRADE_DTESTS=upgrade_tests/cql_tests.py \ + -e REPEATED_JVM_UPGRADE_DTESTS=org.apache.cassandra.distributed.upgrade.GroupByTest +``` +If you need to repeat a particular set of tests and avoid automatic detection of changed tests +use the -s command line option. +For particular Ant test targets that are not included in the regular test suites, you can +use the `run_repeated_utest` job: +``` +generate.sh -p \ + -e REPEATED_ANT_TEST_TARGET=test-cdc \ + -e REPEATED_ANT_TEST_CLASS=org.apache.cassandra.cql3.ViewTest \ + -e REPEATED_ANT_TEST_METHODS=testCompoundPartitionKey,testStaticTable \ + -e REPEATED_ANT_TEST_COUNT=500 +``` +Putting all together, you can have runs as complex as: ``` -generate.sh -h \ - -e REPEATED_UTEST_TARGET=testsome \ - -e REPEATED_UTEST_CLASS=org.apache.cassandra.cql3.ViewTest \ - -e REPEATED_UTEST_METHODS=testCompoundPartitionKey,testStaticTable \ - -e REPEATED_UTEST_COUNT=100 +generate.sh -p \ + -e REPEATED_TESTS_STOP_ON_FAILURE=true \ + -e REPEATED_UTESTS=org.apache.cassandra.cql3.ViewTest,org.apache.cassandra.db.CellTest \ + -e REPEATED_UTESTS_COUNT=500 \ + -e REPEATED_UTESTS_FQLTOOL=org.apache.cassandra.fqltool.FQLCompareTest \ + -e REPEATED_UTESTS_FQLTOOL_COUNT=500 \ + -e REPEATED_UTESTS_LONG=org.apache.cassandra.io.sstable.CQLSSTableWriterLongTest#testWideRow \ + -e REPEATED_UTESTS_LONG_COUNT=100 \ + -e REPEATED_UTESTS_STRESS=org.apache.cassandra.stress.generate.DistributionGaussianTest \ + -e REPEATED_UTESTS_STRESS_COUNT=500 \ + -e REPEATED_DTESTS=cql_test.py,consistency_test.py::TestAvailability::test_simple_strategy \ + -e REPEATED_DTESTS_COUNT=500 \ + -e REPEATED_LARGE_DTESTS=replace_address_test.py,materialized_views_test.py \ + -e REPEATED_LARGE_DTESTS_COUNT=100 \ + -e REPEATED_JVM_DTESTS=org.apache.cassandra.distributed.test.PagingTest#testPaging \ + -e REPEATED_JVM_DTESTS_COUNT=500 \ + -e REPEATED_UPGRADE_DTESTS=upgrade_tests/cql_tests.py \ + -e REPEATED_UPGRADE_DTESTS_COUNT=25 \ + -e REPEATED_JVM_UPGRADE_DTESTS=org.apache.cassandra.distributed.upgrade.GroupByTest \ + -e REPEATED_JVM_UPGRADE_DTESTS_COUNT=500 \ + -e REPEATED_ANT_TEST_TARGET=test-cdc \ + -e REPEATED_ANT_TEST_CLASS=org.apache.cassandra.cql3.ViewTest \ + -e REPEATED_ANT_TEST_METHODS=testCompoundPartitionKey,testStaticTable \ + -e REPEATED_ANT_TEST_COUNT=500 ``` +WARNING: The Python large DTests test_network_topology_strategy and test_network_topology_strategy_each_quorum require +XLarge containers in order to be run in a loop. Otherwise, you would see NO HOST AVAILABLE if you run them with the +default medium config and Large containers. + ## Updating the config For configuration changes meant to be permanent in the Apache repo you should never edit -the `config.yml` file manually. Instead, you should edit the `config-2_1.yml` file and then -regenerate the `config.yml`, `config.yml.LOWRES`, `config.yml.MIDRES` and `config.yml.HIGHRES` -files by runnining the `generate.sh` script without any flags. For using this script you +the `config.yml` file manually. Instead, you should edit the `config_template.yml` file and then +regenerate the `config.yml`, `config.yml.FREE` and `config.yml.PAID` +files by runnining the `generate.sh` script with `-a` flag. For using this script you need to install the [CircleCI CLI](https://circleci.com/docs/2.0/local-cli/#install). - + As for temporal changes done while working in a patch, such as pointing to you dtest repo or -running a test repeatedly, you can either directly edit `config.yml` or edit `config-2_1.yml` -and then regenerate `config.yml` with the `generate.sh` script using a `-l`/`-m`/`-h` flag. +running a test repeatedly, you can either directly edit `config.yml` or edit `config_template.yml` +and then regenerate `config.yml` with the `generate.sh` script using a `-f`/`-p` flag. When this flag is used only the `config.yml` will be generated. Please note that any previous swapping or edition of the generated files will be overriden -by running `generate.sh` without arguments, returning `config.yml` to the default LOWRES. So if -you previously swapped your `config.yml` to MIDRES or HIGHRES you would need to either swap it -again or use the `-l`/`-m`/`-h` script flags. +by running `generate.sh` with `-a` argument, returning `config.yml` to the default FREE. So if +you previously swapped your `config.yml` to MIDRES you would need to either swap it +again or use the `-f`/`-p` script flags. Read below for details how to generate the files manually without the `generate.sh` script: -1. make your edits to config-2_1.yml - let it stay at lowres settings -1. generate a valid LOWRES file: - `circleci config process config-2_1.yml > config.yml.LOWRES` -1. add the Apache license header to the newly created LOWRES file: - `cat license.yml config.yml.LOWRES > config.yml.LOWRES.new && mv config.yml.LOWRES.new config.yml.LOWRES` -1. then apply the highres patch to config-2_1.yml; - `patch -o config-2_1.yml.HIGHRES config-2_1.yml config-2_1.yml.high_res.patch` - (this creates a new file `config-2_1.yml.HIGHRES` instead of in-place patching - config-2_1.yml) - Note that if the patch no longer applies to `config-2_1.yml` a new patch file - is needed, do this by manually making `config-2_1.yml` high resource and create +1. make your edits to config_template.yml - let it stay at free tier settings +2. generate a valid FREE file: + `circleci config process config_template.yml > config.yml.FREE` +3. add the Apache license header to the newly created FREE file: + `cat license.yml config.yml.FREE > config.yml.FREE.new && mv config.yml.FREE.new config.yml.FREE` +4. then apply the PAID patch to config_template.yml; + `patch -o config_template.yml.PAID config_template.yml config_template.yml.PAID.patch` + (this creates a new file `config_template.yml.PAID` instead of in-place patching + config_template.yml). + Note that if the patch no longer applies to `config_template.yml` a new patch file + is needed, do this by manually creating and editing `config_template.yml.PAID`, and create the patch file based on the diff (don't commit it though). -1. generate the HIGHRES file: - `circleci config process config-2_1.yml.HIGHRES > config.yml.HIGHRES` -1. remove the temporary patched HIGHRES file: `rm config-2_1.yml.HIGHRES` -1. add the Apache license header to the newly created HIGHRES file: - `cat license.yml config.yml.HIGHRES > config.yml.HIGHRES.new && mv config.yml.HIGHRES.new config.yml.HIGHRES` -1. repeat the last steps to generate the MIDRES file: - ``` - patch -o config-2_1.yml.MIDRES config-2_1.yml config-2_1.yml.mid_res.patch - circleci config process config-2_1.yml.MIDRES > config.yml.MIDRES - rm config-2_1.yml.MIDRES - cat license.yml config.yml.MIDRES > config.yml.MIDRES.new && mv config.yml.MIDRES.new config.yml.MIDRES - ``` - +5. generate the PAID file: + `circleci config process config_template.yml.PAID > config.yml.PAID` +6. remove the temporary patched PAID file: `rm config_template.yml.PAID` +7. add the Apache license header to the newly created PAID file: + `cat license.yml config.yml.PAID > config.yml.PAID.new && mv config.yml.PAID.new config.yml.PAID` +8. finally, remember to update the config.yml diff --git a/.gitignore b/.gitignore index 666e69cc2929..0eb712d0b459 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,7 @@ .gitignore # C* +build build/ src/gen-java/ src/resources/org/apache/cassandra/config/ diff --git a/CHANGES.txt b/CHANGES.txt index ed5036505a33..0ed4cd11ab2a 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -11,7 +11,117 @@ Merged from 5.0: * Add support for UUID based sstable generation identifiers (CASSANDRA-17048) * Fix a race condition where a keyspace can be oopened while it is being removed (CASSANDRA-17658) + +4.0.12 + * Migrate Python optparse to argparse (CASSANDRA-17914) +Merged from 3.11: + * Moved jflex from runtime to build dependencies (CASSANDRA-18664) + + +4.0.11 + * Revert CASSANDRA-16718 (CASSANDRA-18560) + * Upgrade snappy to 1.1.10.1 (CASSANDRA-18608) + * Fix assertion error when describing mv as table (CASSANDRA-18596) + * Track the amount of read data per row (CASSANDRA-18513) + * Fix Down nodes counter in nodetool describecluster (CASSANDRA-18512) + * Remove unnecessary shuffling of GossipDigests in Gossiper#makeRandomGossipDigest (CASSANDRA-18546) +Merged from 3.11: + * Fix CAST function for float to decimal (CASSANDRA-18647) + * Suppress CVE-2022-45688 (CASSANDRA-18643) + * Remove unrepaired SSTables from garbage collection when only_purge_repaired_tombstones is true (CASSANDRA-14204) + * Wait for live endpoints in gossip waiting to settle (CASSANDRA-18543) + * Fix error message handling when trying to use CLUSTERING ORDER with non-clustering column (CASSANDRA-17818 + * Add keyspace and table name to exception message during ColumnSubselection deserialization (CASSANDRA-18346) +Merged from 3.0: + * Suppress CVE-2023-34462 (CASSANDRA-18649) + * Add support for AWS Ec2 IMDSv2 (CASSANDRA-16555) + * Suppress CVE-2023-35116 (CASSANDRA-18630) + * Pass taskId from CompactionTask to system.compaction_history (CASSANDRA-12183) + * Backport CASSANDRA-10508: Remove hard-coded SSL cipher suites (CASSANDRA-18575) + * Suppress CVE-2023-2976 (CASSANDRA-18562) + * Remove dh_python use in Debian packaging (CASSANDRA-18558) + + +4.0.10 + * Improve nodetool enable{audit,fullquery}log (CASSANDRA-18550) + * Report network cache info in nodetool (CASSANDRa-18400) + * Partial compaction can resurrect deleted data (CASSANDRA-18507) + * Allow internal address to change with reconnecting snitches (CASSANDRA-16718) + * Fix quoting in toCqlString methods of UDTs and aggregates (CASSANDRA-17918) + * NPE when deserializing malformed collections from client (CASSANDRA-18505) + * Improve 'Not enough space for compaction' logging messages (CASSANDRA-18260) + * Incremental repairs fail on mixed IPv4/v6 addresses serializing SyncRequest (CASSANDRA-18474) + * Deadlock updating sstable metadata if disk boundaries need reloading (CASSANDRA-18443) + * Fix nested selection of reversed collections (CASSANDRA-17913) +Merged from 3.11: + * Remove unnecessary String.format invocation in QueryProcessor when getting a prepared statement from cache (CASSANDRA-17202) + * Fix the capital P usage in the CQL parser (CASSANDRA-17919) +Merged from 3.0: + * Pass down all contact points to driver for cassandra-stress (CASSANDRA-18025) + * Validate the existence of a datacenter in nodetool rebuild (CASSANDRA-14319) + * Suppress CVE-2023-2251 (CASSANDRA-18497) + * Do not remove SSTables when cause of FSReadError is OutOfMemoryError while using best_effort disk failure policy (CASSANDRA-18336) + * Do not remove truncated_at entry in system.local while dropping an index (CASSANDRA-18105) + + +4.0.9 + * Update zstd-jni library to version 1.5.5 (CASSANDRA-18429) + * Backport CASSANDRA-17205 to 4.0 branch - Remove self-reference in SSTableTidier (CASSANDRA-18332) + * Avoid loading the preferred IP for BulkLoader streaming (CASSANDRA-18370) + * Fix BufferPool incorrect memoryInUse when putUnusedPortion is used (CASSANDRA-18311) + * Improve memtable allocator accounting when updating AtomicBTreePartition (CASSANDRA-18125) + * Update zstd-jni to version 1.5.4-1 (CASSANDRA-18259) + * Split and order IDEA workspace template VM_PARAMETERS (CASSANDRA-18242) +Merged from 3.11: + * Fix sstable_count metric missing from tablestats json/yaml output (CASSANDRA-18448) + * Suppress CVE-2022-45688 (CASSANDRA-18389) + * Fix Splitter sometimes creating more splits than requested (CASSANDRA-18013) +Merged from 3.0: + * Save host id to system.local and flush immediately after startup (CASSANDRA-18153) + * Fix the ordering of sstables when running sstableupgrade tool (CASSANDRA-18143) + * Fix default file system error handler for disk_failure_policy die (CASSANDRA-18294) + + +4.0.8 + * Log warning message on aggregation queries without key or on multiple keys (CASSANDRA-18219) + * Fix the output of FQL dump tool to properly separate entries (CASSANDRA-18215) + * Add cache type information for maximum memory usage warning message (CASSANDRA-18184) + * Fix NPE in fqltool dump on null value (CASSANDRA-18113) + * Improve unit tests performance (CASSANDRA-17427) + * Connect to listen address when own broadcast address is requested (CASSANDRA-18200) + * Add safeguard so cleanup fails when node has pending ranges (CASSANDRA-16418) + * Fix legacy clustering serialization for paging with compact storage (CASSANDRA-17507) + * Add support for python 3.11 (CASSANDRA-18088) + * Fix formatting of duration in cqlsh (CASSANDRA-18141) + * Fix sstable loading of keyspaces named snapshots or backups (CASSANDRA-14013) + * Avoid ConcurrentModificationException in STCS/DTCS/TWCS.getSSTables (CASSANDRA-17977) + * Restore internode custom tracing on 4.0's new messaging system (CASSANDRA-17981) + * Harden parsing of boolean values in CQL in PropertyDefinitions (CASSANDRA-17878) + * Fix error message about type hints (CASSANDRA-17915) + * Fix possible race condition on repair snapshots (CASSANDRA-17955) + * Fix ASM bytecode version inconsistency (CASSANDRA-17873) +Merged from 3.11: + * Fix Splitter sometimes creating more splits than requested (CASSANDRA-18013) +Merged from 3.0: + * Introduce check for names of test classes (CASSANDRA-17964) + * Suppress CVE-2022-41915 (CASSANDRA-18147) + * Suppress CVE-2021-1471, CVE-2021-3064, CVE-2021-4235 (CASSANDRA-18149) + * Switch to snakeyaml's SafeConstructor (CASSANDRA-18150) + * Expand build.dir property in rat targets (CASSANDRA-18183) + * Suppress CVE-2022-41881 (CASSANDRA-18148) + * Default role is created with zero timestamp (CASSANDRA-12525) + * Suppress CVE-2021-37533 (CASSANDRA-18146) + * Add to the IntelliJ Git Window issue navigation links to Cassandra's Jira (CASSANDRA-18126) + * Avoid anticompaction mixing data from two different time windows with TWCS (CASSANDRA-17970) + * Do not spam the logs with MigrationCoordinator not being able to pull schemas (CASSANDRA-18096) + * Fix incorrect resource name in LIST PERMISSION output (CASSANDRA-17848) + * Suppress CVE-2022-41854 and similar (CASSANDRA-18083) + * Fix running Ant rat targets without git (CASSANDRA-17974) + + 4.0.7 + * Remove empty cq4 files in log directory to not fail the startup of BinLog (CASSANDRA-17933) + * Fix multiple BufferPool bugs (CASSANDRA-16681) * Fix StorageService.getNativeaddress handling of IPv6 addresses (CASSANDRA-17945) * Mitigate direct buffer memory OOM on replacements (CASSANDRA-17895) * Fix repair failure on assertion if two peers have overlapping mismatching ranges (CASSANDRA-17900) @@ -20,14 +130,18 @@ Merged from 5.0: * Avoid getting hanging repairs due to repair message timeouts (CASSANDRA-17613) * Prevent infinite loop in repair coordinator on FailSession (CASSANDRA-17834) Merged from 3.11: + * Suppress CVE-2022-42003 and CVE-2022-42004 (CASSANDRA-17966) * Make LongBufferPoolTest insensitive to timing (CASSANDRA-16681) * Suppress CVE-2022-25857 and other snakeyaml CVEs (CASSANDRA-17907) * Fix potential IndexOutOfBoundsException in PagingState in mixed mode clusters (CASSANDRA-17840) Merged from 3.0: + * Harden JMX by resolving beanshooter issues (CASSANDRA-17921) + * Suppress CVE-2019-2684 (CASSANDRA-17965) * Fix auto-completing "WITH" when creating a materialized view (CASSANDRA-17879) * Fix scrubber falling into infinite loop when the last partition is broken (CASSANDRA-17862) * Fix resetting schema (CASSANDRA-17819) + 4.0.6 * Fix race condition on updating cdc size and advancing to next segment (CASSANDRA-17792) * Add 'noboolean' rpm build for older distros like CentOS7 (CASSANDRA-17765) @@ -44,6 +158,7 @@ Merged from 3.0: * Improve libjemalloc resolution in bin/cassandra (CASSANDRA-15767) * Fix restarting of services on gossipping-only member (CASSANDRA-17752) + 4.0.5 * Utilise BTree improvements to reduce garbage and improve throughput (CASSANDRA-15511) * Make sure existing delayed tasks in StreamTransferTask cannot prevent clean shutdown (CASSANDRA-17706) @@ -62,6 +177,7 @@ Merged from 3.0: * Suppress CVE-2022-24823 (CASSANDRA-17633) * fsync TOC and digest files (CASSANDRA-10709) + 4.0.4 * Optimise BTree build,update and transform operations (CASSANDRA-15510) * Clean up schema migration coordinator and tests (CASSANDRA-17533) @@ -104,11 +220,13 @@ Merged from 3.0: * Lazy transaction log replica creation allows incorrect replica content divergence during anticompaction (CASSANDRA-17273) * LeveledCompactionStrategy disk space check improvements (CASSANDRA-17272) + 4.0.3 * Deprecate otc_coalescing_strategy, otc_coalescing_window_us, otc_coalescing_enough_coalesced_messages, otc_backlog_expiration_interval_ms (CASSANDRA-17377) * Improve start up processing of Incremental Repair information read from system.repairs (CASSANDRA-17342) + 4.0.2 * Extend operator control over the UDF threading model for CVE-2021-44521 (CASSANDRA-17352) * Full Java 11 support (CASSANDRA-16894) @@ -168,6 +286,7 @@ Merged from 3.0: * Make the addition of regular column to COMPACT tables throw an InvalidRequestException (CASSANDRA-14564) * Race in CompactionExecutorTest (CASSANDRA-17239) + 4.0.1 * Tolerate missing DNS entry when completing a host replacement (CASSANDRA-16873) * Harden PrunableArrayQueue against Pruner implementations that might throw exceptions (CASSANDRA-16866) @@ -204,6 +323,7 @@ Merged from 3.0: * Ensure java executable is on the path (CASSANDRA-14325) * Clean transaction log leftovers at the beginning of sstablelevelreset and sstableofflinerelevel (CASSANDRA-12519) + 4.0.0 * Avoid signaling DigestResolver until the minimum number of responses are guaranteed to be visible (CASSANDRA-16807) * Fix pre-4.0 FWD_FRM parameter serializer (CASSANDRA-16808) @@ -213,6 +333,7 @@ Merged from 3.0: Merged from 3.11: Merged from 3.0: + 4.0-rc2 * Improved password obfuscation (CASSANDRA-16801) * Avoid memoizing the wrong min cluster version during upgrades (CASSANDRA-16759) @@ -248,6 +369,7 @@ Merged from 3.0: * Prevent loss of commit log data when moving sstables between nodes (CASSANDRA-16619) * Fix materialized view builders inserting truncated data (CASSANDRA-16567) + 4.0-rc1 * Allow for setting buffer max capacity to increase it dynamically as needed (CASSANDRA-16524) * Harden internode message resource limit accounting against serialization failures (CASSANDRA-16616) @@ -329,6 +451,7 @@ Merged from 2.2: * Fix centos packaging for arm64, >=4.0 rpm's now require python3 (CASSANDRA-16477) * Make TokenMetadata's ring version increments atomic (CASSANDRA-16286) + 4.0-beta4 * DROP COMPACT STORAGE should invalidate prepared statements still using CompactTableMetadata (CASSANDRA-16361) * Update default num_tokens to 16 and allocate_tokens_for_local_replication_factor to 3 (CASSANDRA-13701) @@ -376,6 +499,7 @@ Merged from 3.0: Merged from 2.2: * Fix the histogram merge of the table metrics (CASSANDRA-16259) + 4.0-beta3 * Segregate Network and Chunk Cache BufferPools and Recirculate Partially Freed Chunks (CASSANDRA-15229) * Fail truncation requests when they fail on a replica (CASSANDRA-16208) @@ -418,6 +542,7 @@ Merged from 2.2: * Automatically drop compact storage on tables for which it is safe (CASSANDRA-16048) * Fixed NullPointerException for COMPACT STORAGE tables with null clustering (CASSANDRA-16241) + 4.0-beta2 * Add addition incremental repair visibility to nodetool repair_admin (CASSANDRA-14939) * Always access system properties and environment variables via the new CassandraRelevantProperties and CassandraRelevantEnv classes (CASSANDRA-15876) @@ -456,6 +581,7 @@ Merged from 2.2: Merged from 2.1: * Only allow strings to be passed to JMX authentication (CASSANDRA-16077) + 4.0-beta1 * Remove BackPressureStrategy (CASSANDRA-15375) * Improve messaging on indexing frozen collections (CASSANDRA-15908) @@ -545,6 +671,7 @@ Merged from 2.2: Merged from 2.1: * Fix writing of snapshot manifest when the table has table-backed secondary indexes (CASSANDRA-10968) + 4.0-alpha4 * Add client request size server metrics (CASSANDRA-15704) * Add additional logging around FileUtils and compaction leftover cleanup (CASSANDRA-15705) @@ -601,6 +728,7 @@ Merged from 2.1: * Fix Red Hat init script on newer systemd versions (CASSANDRA-15273) * Allow EXTRA_CLASSPATH to work on tar/source installations (CASSANDRA-15567) + 4.0-alpha3 * Restore monotonic read consistency guarantees for blocking read repair (CASSANDRA-14740) * Separate exceptions for CAS write timeout exceptions caused by contention and unkown result (CASSANDRA-15350) @@ -652,6 +780,7 @@ Merged from 2.2: * In-JVM DTest: Support NodeTool in dtest (CASSANDRA-15429) * Added data modeling documentation (CASSANDRA-15443) + 4.0-alpha2 * Fix SASI non-literal string comparisons (range operators) (CASSANDRA-15169) * Upgrade Guava to 27, and to java-driver 3.6.0 (from 3.4.0-SNAPSHOT) (CASSANDRA-14655) @@ -1043,6 +1172,7 @@ Merged from 3.0: * Multi-version in-JVM dtests (CASSANDRA-14937) * Allow instance class loaders to be garbage collected for inJVM dtest (CASSANDRA-15170) + 3.11.6 * Fix bad UDT sstable metadata serialization headers written by C* 3.0 on upgrade and in sstablescrub (CASSANDRA-15035) * Fix nodetool compactionstats showing extra pending task for TWCS - patch implemented (CASSANDRA-15409) @@ -1073,6 +1203,7 @@ Merged from 2.2: * In-JVM DTest: Support NodeTool in dtest (CASSANDRA-15429) * Fix NativeLibrary.tryOpenDirectory callers for Windows (CASSANDRA-15426) + 3.11.5 * Fix cassandra-env.sh to use $CASSANDRA_CONF to find cassandra-jaas.config (CASSANDRA-14305) * Fixed nodetool cfstats printing index name twice (CASSANDRA-14903) @@ -2209,7 +2340,6 @@ Merged from 2.1: * Sane default (200Mbps) for inter-DC streaming througput (CASSANDRA-8708) - 3.2 * Make sure tokens don't exist in several data directories (CASSANDRA-6696) * Add requireAuthorization method to IAuthorizer (CASSANDRA-10852) @@ -7242,7 +7372,6 @@ Merged from 0.8: - Similarly, merged batch_insert_super into batch_insert. - 0.4.0 beta * On-disk data format has changed to allow billions of keys/rows per node instead of only millions @@ -7279,7 +7408,6 @@ Merged from 0.8: * Rename configuration "table" to "keyspace" * Moved to crash-only design; no more shutdown (just kill the process) * Lots of bug fixes - Full list of issues resolved in 0.4 is at https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&&pid=12310865&fixfor=12313862&resolution=1&sorter/field=issuekey&sorter/order=DESC @@ -7310,3 +7438,5 @@ Full list of issues resolved in 0.4 is at https://issues.apache.org/jira/secure/ * Combined blocking and non-blocking versions of insert APIs * Added FlushPeriodInMinutes configuration parameter to force flushing of infrequently-updated ColumnFamilies + + diff --git a/NEWS.txt b/NEWS.txt index 3c67ccfbc13b..ec843e7aa62f 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -51,6 +51,30 @@ restore snapshots created with the previous major version using the 'sstableloader' tool. You can upgrade the file format of your snapshots using the provided 'sstableupgrade' tool. +4.0.11 +====== + +Upgrading +--------- + - Please beware that if you use Ec2Snitch or Ec2MultiRegionSnitch, by default it will + communicate with AWS IMDS of version 2. This change is transparent, there does not need + to be done anything upon upgrade. Furthermore, IMDS of version 2 can be configured to be required in AWS EC2 console. + Consult cassandra-rackdc.properties for more details. (CASSANDRA-16555) + +4.0.8 +===== + +New features +------------ + +Upgrading +--------- + - All previous versions of 4.x contained a mistake on the implementation of the old CQL native protocol v3. That + mistake produced issues when paging over tables with compact storage and a single clustering column during rolling + upgrades involving 3.x and 4.x nodes. The fix for that issue makes it can now appear during rolling upgrades from + 4.0.0-4.0.7. If that is your case, please use protocol v4 or higher in your driver. See CASSANDRA-17507 for further + details. + 4.0.6 ===== diff --git a/NOTICE.txt b/NOTICE.txt index 0f8294d6f015..fd185210450f 100644 --- a/NOTICE.txt +++ b/NOTICE.txt @@ -1,5 +1,5 @@ Apache Cassandra -Copyright 2009-2022 The Apache Software Foundation +Copyright 2009- The Apache Software Foundation This product includes software developed by The Apache Software Foundation (http://www.apache.org/). diff --git a/bin/cqlsh.py b/bin/cqlsh.py index 6ff6e333f0a2..d2ef7ee6a3b5 100755 --- a/bin/cqlsh.py +++ b/bin/cqlsh.py @@ -22,7 +22,7 @@ import codecs import csv import getpass -import optparse +import argparse import os import platform import re @@ -196,60 +196,61 @@ def find_zip(libprefix): host (and optional port number) are given on the command line, they take precedence over any defaults.""" % globals() -parser = optparse.OptionParser(description=description, epilog=epilog, - usage="Usage: %prog [options] [host [port]]", - version='cqlsh ' + version) -parser.add_option("-C", "--color", action='store_true', dest='color', - help='Always use color output') -parser.add_option("--no-color", action='store_false', dest='color', - help='Never use color output') -parser.add_option("--browser", dest='browser', help="""The browser to use to display CQL help, where BROWSER can be: +parser = argparse.ArgumentParser(description=description, epilog=epilog, + usage="Usage: %(prog)s [options] [host [port]]", + prog='cqlsh') +parser.add_argument('-v', '--version', action='version', version='cqlsh ' + version) +parser.add_argument("-C", "--color", action='store_true', dest='color', + help='Always use color output') +parser.add_argument("--no-color", action='store_false', dest='color', + help='Never use color output') +parser.add_argument("--browser", dest='browser', help="""The browser to use to display CQL help, where BROWSER can be: - one of the supported browsers in https://docs.python.org/3/library/webbrowser.html. - browser path followed by %s, example: /usr/bin/google-chrome-stable %s""") -parser.add_option('--ssl', action='store_true', help='Use SSL', default=False) -parser.add_option("-u", "--username", help="Authenticate as user.") -parser.add_option("-p", "--password", help="Authenticate using password.") -parser.add_option('-k', '--keyspace', help='Authenticate to the given keyspace.') -parser.add_option('-b', '--secure-connect-bundle', - help="Connect using secure connect bundle. If this option is specified host, port settings are ignored.") -parser.add_option("-f", "--file", help="Execute commands from FILE, then exit") -parser.add_option('--debug', action='store_true', - help='Show additional debugging information') -parser.add_option('--coverage', action='store_true', - help='Collect coverage data') -parser.add_option("--encoding", help="Specify a non-default encoding for output." - + " (Default: %s)" % (UTF8,)) -parser.add_option("--cqlshrc", help="Specify an alternative cqlshrc file location.") -parser.add_option('--cqlversion', default=None, - help='Specify a particular CQL version, ' - 'by default the highest version supported by the server will be used.' - ' Examples: "3.0.3", "3.1.0"') -parser.add_option("--protocol-version", type="int", default=None, - help='Specify a specific protcol version otherwise the client will default and downgrade as necessary') - -parser.add_option("-e", "--execute", help='Execute the statement and quit.') -parser.add_option("--connect-timeout", default=DEFAULT_CONNECT_TIMEOUT_SECONDS, dest='connect_timeout', - help='Specify the connection timeout in seconds (default: %default seconds).') -parser.add_option("--request-timeout", default=DEFAULT_REQUEST_TIMEOUT_SECONDS, dest='request_timeout', - help='Specify the default request timeout in seconds (default: %default seconds).') -parser.add_option("--consistency-level", dest='consistency_level', - help='Specify the initial consistency level.') -parser.add_option("--serial-consistency-level", dest='serial_consistency_level', - help='Specify the initial serial consistency level.') -parser.add_option("-t", "--tty", action='store_true', dest='tty', - help='Force tty mode (command prompt).') -parser.add_option("--no-file-io", action='store_true', dest='no_file_io', - help='Disable cqlsh commands that perform file I/O.') -parser.add_option('--disable-history', action='store_true', help='Disable saving of history', default=False) - -optvalues = optparse.Values() -(options, arguments) = parser.parse_args(sys.argv[1:], values=optvalues) +parser.add_argument('--ssl', action='store_true', help='Use SSL', default=False) +parser.add_argument("-u", "--username", help="Authenticate as user.") +parser.add_argument("-p", "--password", help="Authenticate using password.") +parser.add_argument('-k', '--keyspace', help='Authenticate to the given keyspace.') +parser.add_argument('-b', '--secure-connect-bundle', + help="Connect using secure connect bundle. If this option is specified host, port settings are ignored.") +parser.add_argument("-f", "--file", help="Execute commands from FILE, then exit") +parser.add_argument('--debug', action='store_true', + help='Show additional debugging information') +parser.add_argument('--coverage', action='store_true', + help='Collect coverage data') +parser.add_argument("--encoding", help="Specify a non-default encoding for output." + + " (Default: %s)" % (UTF8,)) +parser.add_argument("--cqlshrc", help="Specify an alternative cqlshrc file location.") +parser.add_argument('--cqlversion', default=None, + help='Specify a particular CQL version, ' + 'by default the highest version supported by the server will be used.' + ' Examples: "3.0.3", "3.1.0"') +parser.add_argument("--protocol-version", type=int, default=None, + help='Specify a specific protcol version otherwise the client will default and downgrade as necessary') + +parser.add_argument("-e", "--execute", help='Execute the statement and quit.') +parser.add_argument("--connect-timeout", default=DEFAULT_CONNECT_TIMEOUT_SECONDS, dest='connect_timeout', + help='Specify the connection timeout in seconds (default: %default seconds).') +parser.add_argument("--request-timeout", default=DEFAULT_REQUEST_TIMEOUT_SECONDS, dest='request_timeout', + help='Specify the default request timeout in seconds (default: %default seconds).') +parser.add_argument("--consistency-level", dest='consistency_level', + help='Specify the initial consistency level.') +parser.add_argument("--serial-consistency-level", dest='serial_consistency_level', + help='Specify the initial serial consistency level.') +parser.add_argument("-t", "--tty", action='store_true', dest='tty', + help='Force tty mode (command prompt).') +parser.add_argument("--no-file-io", action='store_true', dest='no_file_io', + help='Disable cqlsh commands that perform file I/O.') +parser.add_argument('--disable-history', action='store_true', help='Disable saving of history', default=False) + +cfarguments, args = parser.parse_known_args() + # BEGIN history/config definition HISTORY_DIR = os.path.expanduser(os.path.join('~', '.cassandra')) -if hasattr(options, 'cqlshrc'): - CONFIG_FILE = options.cqlshrc +if cfarguments.cqlshrc is not None: + CONFIG_FILE = cfarguments.cqlshrc if not os.path.exists(CONFIG_FILE): print('\nWarning: Specified cqlshrc location `%s` does not exist. Using `%s` instead.\n' % (CONFIG_FILE, HISTORY_DIR)) CONFIG_FILE = os.path.join(HISTORY_DIR, 'cqlshrc') @@ -2181,53 +2182,54 @@ def read_options(cmdlineargs, environment): rawconfigs = configparser.RawConfigParser() rawconfigs.read(CONFIG_FILE) - optvalues = optparse.Values() - optvalues.username = option_with_default(configs.get, 'authentication', 'username') - optvalues.password = option_with_default(rawconfigs.get, 'authentication', 'password') - optvalues.keyspace = option_with_default(configs.get, 'authentication', 'keyspace') - optvalues.secure_connect_bundle = option_with_default(configs.get, 'connection', 'secure_connect_bundle') - optvalues.browser = option_with_default(configs.get, 'ui', 'browser', None) - optvalues.completekey = option_with_default(configs.get, 'ui', 'completekey', + argvalues = argparse.Namespace() + + argvalues.username = option_with_default(configs.get, 'authentication', 'username') + argvalues.password = option_with_default(rawconfigs.get, 'authentication', 'password') + argvalues.keyspace = option_with_default(configs.get, 'authentication', 'keyspace') + argvalues.secure_connect_bundle = option_with_default(configs.get, 'connection', 'secure_connect_bundle') + argvalues.browser = option_with_default(configs.get, 'ui', 'browser', None) + argvalues.completekey = option_with_default(configs.get, 'ui', 'completekey', DEFAULT_COMPLETEKEY) - optvalues.color = option_with_default(configs.getboolean, 'ui', 'color') - optvalues.time_format = raw_option_with_default(configs, 'ui', 'time_format', + argvalues.color = option_with_default(configs.getboolean, 'ui', 'color') + argvalues.time_format = raw_option_with_default(configs, 'ui', 'time_format', DEFAULT_TIMESTAMP_FORMAT) - optvalues.nanotime_format = raw_option_with_default(configs, 'ui', 'nanotime_format', + argvalues.nanotime_format = raw_option_with_default(configs, 'ui', 'nanotime_format', DEFAULT_NANOTIME_FORMAT) - optvalues.date_format = raw_option_with_default(configs, 'ui', 'date_format', + argvalues.date_format = raw_option_with_default(configs, 'ui', 'date_format', DEFAULT_DATE_FORMAT) - optvalues.float_precision = option_with_default(configs.getint, 'ui', 'float_precision', + argvalues.float_precision = option_with_default(configs.getint, 'ui', 'float_precision', DEFAULT_FLOAT_PRECISION) - optvalues.double_precision = option_with_default(configs.getint, 'ui', 'double_precision', + argvalues.double_precision = option_with_default(configs.getint, 'ui', 'double_precision', DEFAULT_DOUBLE_PRECISION) - optvalues.field_size_limit = option_with_default(configs.getint, 'csv', 'field_size_limit', csv.field_size_limit()) - optvalues.max_trace_wait = option_with_default(configs.getfloat, 'tracing', 'max_trace_wait', + argvalues.field_size_limit = option_with_default(configs.getint, 'csv', 'field_size_limit', csv.field_size_limit()) + argvalues.max_trace_wait = option_with_default(configs.getfloat, 'tracing', 'max_trace_wait', DEFAULT_MAX_TRACE_WAIT) - optvalues.timezone = option_with_default(configs.get, 'ui', 'timezone', None) + argvalues.timezone = option_with_default(configs.get, 'ui', 'timezone', None) - optvalues.debug = False + argvalues.debug = False - optvalues.coverage = False + argvalues.coverage = False if 'CQLSH_COVERAGE' in environment.keys(): - optvalues.coverage = True + argvalues.coverage = True - optvalues.file = None - optvalues.ssl = option_with_default(configs.getboolean, 'connection', 'ssl', DEFAULT_SSL) - optvalues.encoding = option_with_default(configs.get, 'ui', 'encoding', UTF8) + argvalues.file = None + argvalues.ssl = option_with_default(configs.getboolean, 'connection', 'ssl', DEFAULT_SSL) + argvalues.encoding = option_with_default(configs.get, 'ui', 'encoding', UTF8) - optvalues.consistency_level = option_with_default(configs.get, 'cql', 'consistency_level', None) - optvalues.serial_consistency_level = option_with_default(configs.get, 'cql', 'serial_consistency_level', 'SERIAL') + argvalues.consistency_level = option_with_default(configs.get, 'cql', 'consistency_level', None) + argvalues.serial_consistency_level = option_with_default(configs.get, 'cql', 'serial_consistency_level', 'SERIAL') - optvalues.tty = option_with_default(configs.getboolean, 'ui', 'tty', sys.stdin.isatty()) - optvalues.protocol_version = option_with_default(configs.getint, 'protocol', 'version', None) - optvalues.cqlversion = option_with_default(configs.get, 'cql', 'version', None) - optvalues.connect_timeout = option_with_default(configs.getint, 'connection', 'timeout', DEFAULT_CONNECT_TIMEOUT_SECONDS) - optvalues.request_timeout = option_with_default(configs.getint, 'connection', 'request_timeout', DEFAULT_REQUEST_TIMEOUT_SECONDS) - optvalues.execute = None - optvalues.no_file_io = option_with_default(configs.getboolean, 'ui', 'no_file_io', DEFAULT_NO_FILE_IO) - optvalues.disable_history = option_with_default(configs.getboolean, 'history', 'disabled', False) + argvalues.tty = option_with_default(configs.getboolean, 'ui', 'tty', sys.stdin.isatty()) + argvalues.protocol_version = option_with_default(configs.getint, 'protocol', 'version', None) + argvalues.cqlversion = option_with_default(configs.get, 'cql', 'version', None) + argvalues.connect_timeout = option_with_default(configs.getint, 'connection', 'timeout', DEFAULT_CONNECT_TIMEOUT_SECONDS) + argvalues.request_timeout = option_with_default(configs.getint, 'connection', 'request_timeout', DEFAULT_REQUEST_TIMEOUT_SECONDS) + argvalues.execute = None + argvalues.no_file_io = option_with_default(configs.getboolean, 'ui', 'no_file_io', DEFAULT_NO_FILE_IO) + argvalues.disable_history = option_with_default(configs.getboolean, 'history', 'disabled', False) - (options, arguments) = parser.parse_args(cmdlineargs, values=optvalues) + options, arguments = parser.parse_known_args(cmdlineargs, argvalues) # Make sure some user values read from the command line are in unicode options.execute = maybe_ensure_text(options.execute) options.username = maybe_ensure_text(options.username) @@ -2286,8 +2288,8 @@ def read_options(cmdlineargs, environment): if options.execute and not options.execute.endswith(';'): options.execute += ';' - if optvalues.color in (True, False): - options.color = optvalues.color + if argvalues.color in (True, False): + options.color = argvalues.color else: if options.file is not None: options.color = False diff --git a/build.xml b/build.xml index 7b504193d1b4..f309fc77d26b 100644 --- a/build.xml +++ b/build.xml @@ -33,7 +33,7 @@ - + @@ -63,6 +63,7 @@ + @@ -114,12 +115,18 @@ + + + + + + @@ -163,6 +170,10 @@ + + + + - + @@ -307,7 +320,7 @@ @@ -510,10 +523,10 @@ - + - + @@ -557,8 +570,8 @@ - - + + @@ -988,7 +1001,7 @@ - + @@ -998,7 +1011,7 @@ ant stress-test-some -Dtest.name=org.apache.cassandra.stress.generate.DistributionGaussianTest ant stress-test-some -Dtest.name=org.apache.cassandra.stress.generate.DistributionGaussianTest -Dtest.methods=simpleGaussian --> - + @@ -1043,7 +1056,7 @@ - + @@ -1105,7 +1118,7 @@ - + @@ -1299,8 +1312,20 @@ - - + + + + + @@ -1331,12 +1358,26 @@ + + + + + + + + + + + + + + @@ -1425,7 +1466,7 @@ more aggressively rather than waiting. See CASSANDRA-14922 for more details. --> - + @@ -1440,6 +1481,9 @@ + + + @@ -1482,7 +1526,7 @@ - + @@ -1660,7 +1704,7 @@ - + @@ -1676,7 +1720,7 @@ ant long-testsome -Dtest.name=org.apache.cassandra.cql3.ManyRowsTest ant long-testsome -Dtest.name=org.apache.cassandra.cql3.ManyRowsTest -Dtest.methods=testLargeCount --> - + @@ -1689,7 +1733,7 @@ ant burn-testsome -Dtest.name=org.apache.cassandra.utils.memory.LongBufferPoolTest ant burn-testsome -Dtest.name=org.apache.cassandra.utils.memory.LongBufferPoolTest -Dtest.methods=testPoolAllocateWithRecyclePartially --> - + @@ -1697,7 +1741,7 @@ - + @@ -1706,7 +1750,7 @@ - + @@ -1714,7 +1758,7 @@ - + @@ -1722,7 +1766,7 @@ - + @@ -1730,7 +1774,7 @@ - + @@ -1754,39 +1798,39 @@ - + - + - + - + - + - + @@ -1794,14 +1838,14 @@ - + - + @@ -1847,7 +1891,7 @@ ant cql-test-some -Dtest.name=ListsTest ant cql-test-some -Dtest.name=ListsTest -Dtest.methods=testPrecisionTime_getNext_simple --> - + @@ -1959,7 +2003,7 @@ - + @@ -1979,42 +2023,42 @@ - + - + - + - + - + - + @@ -2046,7 +2090,7 @@ - + @@ -2055,7 +2099,7 @@ - + @@ -2067,7 +2111,7 @@ - + @@ -2099,6 +2143,8 @@ + + @@ -2110,7 +2156,7 @@ - + @@ -2137,14 +2183,14 @@ - ]]> @@ -2157,6 +2203,7 @@ + Apache Cassandra ${eclipse.project.name} diff --git a/conf/cassandra-rackdc.properties b/conf/cassandra-rackdc.properties index cc472b4e611d..2d17808d30f1 100644 --- a/conf/cassandra-rackdc.properties +++ b/conf/cassandra-rackdc.properties @@ -37,3 +37,9 @@ rack=rack1 # rack name is the region plus the availability zone letter. # Examples: us-west-1a => dc: us-west-1, rack: us-west-1a; us-west-2b => dc: us-west-2, rack: us-west-2b; # ec2_naming_scheme=standard +# +# Type of AWS IMDS, value might be either "v1" or "v2". Defaults to "v2". +# ec2_metadata_type=v2 +# If AWS IMDS of v2 is configured, ec2_metadata_token_ttl_seconds says how many seconds a token will be valid until +# it is refreshed. Defaults to 21600. Can not be smaller than 30 and bigger than 21600. Has to be an integer. +# ec2_metadata_token_ttl_seconds=21600 diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml index a8a723822ffc..01db4ea6a2d3 100644 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@ -1348,6 +1348,8 @@ audit_logging_options: # max_log_size: 17179869184 # 16 GiB ## archive command is "/path/to/script.sh %path" where %path is replaced with the file being rolled: # archive_command: + ## note that enabling this allows anyone with JMX/nodetool access to run local shell commands as the user running cassandra + # allow_nodetool_archive_command: false # max_archive_retries: 10 # validate tombstones on reads and compaction diff --git a/debian/changelog b/debian/changelog index 441d0b0d33e3..932737c72b0d 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,8 +1,32 @@ -cassandra (4.0.7) UNRELEASED; urgency=medium +cassandra (4.0.11) unstable; urgency=medium * New release - -- Mick Semb Wever Thu, 18 Aug 2022 20:30:16 +0200 + -- Stefan Miklosovic Wed, 12 Jul 2023 21:46:36 +0200 + +cassandra (4.0.10) unstable; urgency=medium + + * New release + + -- Mick Semb Wever Thu, 25 May 2023 14:44:07 +0200 + +cassandra (4.0.9) unstable; urgency=medium + + * New release + + -- Stefan Miklosovic Tue, 11 Apr 2023 09:31:33 +0200 + +cassandra (4.0.8) unstable; urgency=medium + + * New release + + -- Stefan Miklosovic Tue, 07 Feb 2023 22:33:43 +0100 + +cassandra (4.0.7) unstable; urgency=medium + + * New release + + -- Mick Semb Wever Wed, 19 Oct 2022 10:57:06 +0200 cassandra (4.0.6) unstable; urgency=medium diff --git a/debian/rules b/debian/rules index ea2281b2e95a..3d1babe916e7 100755 --- a/debian/rules +++ b/debian/rules @@ -41,7 +41,6 @@ install: build dh_testroot dh_installdirs dh_install - dh_python3 # Copy in the jar and symlink to something stable dh_install build/apache-cassandra-$(VERSION).jar \ diff --git a/doc/modules/cassandra/pages/data_modeling/data_modeling_queries.adoc b/doc/modules/cassandra/pages/data_modeling/data_modeling_queries.adoc index 21f980181223..ed40fb831f1f 100644 --- a/doc/modules/cassandra/pages/data_modeling/data_modeling_queries.adoc +++ b/doc/modules/cassandra/pages/data_modeling/data_modeling_queries.adoc @@ -28,7 +28,7 @@ here, however, you’ll want to think not only from the customer perspective in terms of how the data is written, but also in terms of how the data will be queried by downstream use cases. -You natural tendency as might be to focus first on designing the tables +Your natural tendency might be to focus first on designing the tables to store reservation and guest records, and only then start thinking about the queries that would access them. You may have felt a similar tension already when discussing the shopping queries before, thinking diff --git a/doc/modules/cassandra/pages/data_modeling/data_modeling_rdbms.adoc b/doc/modules/cassandra/pages/data_modeling/data_modeling_rdbms.adoc index b478df14a1c4..2acd6cc2bfbf 100644 --- a/doc/modules/cassandra/pages/data_modeling/data_modeling_rdbms.adoc +++ b/doc/modules/cassandra/pages/data_modeling/data_modeling_rdbms.adoc @@ -17,7 +17,7 @@ image::data_modeling_hotel_relational.png[image] == Design Differences Between RDBMS and Cassandra Let’s take a minute to highlight some of the key differences in doing -ata modeling for Cassandra versus a relational database. +data modeling for Cassandra versus a relational database. === No joins diff --git a/doc/modules/cassandra/pages/data_modeling/data_modeling_refining.adoc b/doc/modules/cassandra/pages/data_modeling/data_modeling_refining.adoc index 09f16da094b6..045a80c4fd0b 100644 --- a/doc/modules/cassandra/pages/data_modeling/data_modeling_refining.adoc +++ b/doc/modules/cassandra/pages/data_modeling/data_modeling_refining.adoc @@ -1,4 +1,5 @@ = Evaluating and Refining Data Models +:stem: latexmath Once you’ve created a physical model, there are some steps you’ll want to take to evaluate and refine table designs to help ensure optimal diff --git a/doc/modules/cassandra/pages/getting_started/production.adoc b/doc/modules/cassandra/pages/getting_started/production.adoc index de7fb54234ac..93b26084b257 100644 --- a/doc/modules/cassandra/pages/getting_started/production.adoc +++ b/doc/modules/cassandra/pages/getting_started/production.adoc @@ -51,7 +51,7 @@ appropriate number of replicates, to ensure even token allocation. Read ahead is an operating system feature that attempts to keep as much data as possible loaded in the page cache. Spinning disks can have long seek times causing high latency, so additional -throughout on reads using page cache can improve performance. +throughput on reads using page cache can improve performance. By leveraging read ahead, the OS can pull additional data into memory without the cost of additional seeks. This method works well when the available RAM is greater than the size of the diff --git a/doc/scripts/gen-nodetool-docs.py b/doc/scripts/gen-nodetool-docs.py index 1903dca8763c..d27e213b424c 100644 --- a/doc/scripts/gen-nodetool-docs.py +++ b/doc/scripts/gen-nodetool-docs.py @@ -24,6 +24,10 @@ import subprocess from subprocess import PIPE from subprocess import Popen +from itertools import islice +from threading import Thread + +batch_size = 3 if(os.environ.get("SKIP_NODETOOL") == "1"): sys.exit(0) @@ -36,6 +40,18 @@ command_re = re.compile("( )([_a-z]+)") commandADOCContent = "== {0}\n\n== Usage\n[source,plaintext]\n----\ninclude::example$TEXT/NODETOOL/{0}.txt[]\n----\n" +# https://docs.python.org/3/library/itertools.html#itertools-recipes +def batched(iterable, n): + "Batch data into tuples of length n. The last batch may be shorter." + # batched('ABCDEFG', 3) --> ABC DEF G + if n < 1: + raise ValueError('n must be at least one') + it = iter(iterable) + batch = tuple(islice(it, n)) + while (batch): + yield batch + batch = tuple(islice(it, n)) + # create the documentation directory if not os.path.exists(outdir): os.makedirs(outdir) @@ -78,6 +94,12 @@ def create_adoc(command): # create the command usage pages with open(helpfilename, "r+") as helpfile: - for commandLine in helpfile: - command = command_re.match(commandLine) - create_adoc(command) + for clis in batched(helpfile, batch_size): + threads = [] + for commandLine in clis: + command = command_re.match(commandLine) + t = Thread(target=create_adoc, args=[command]) + threads.append(t) + t.start() + for t in threads: + t.join() diff --git a/ide/idea/vcs.xml b/ide/idea/vcs.xml index 94a25f7f4cb4..81872fd3f150 100644 --- a/ide/idea/vcs.xml +++ b/ide/idea/vcs.xml @@ -3,4 +3,14 @@ + + + \ No newline at end of file diff --git a/ide/idea/workspace.xml b/ide/idea/workspace.xml index 2f25f78c221e..7bb85cd7a7fb 100644 --- a/ide/idea/workspace.xml +++ b/ide/idea/workspace.xml @@ -143,7 +143,14 @@