commit 79d38ce785a0fc3d265707e7690d11228fc289f1 Author: Sachin Jayant Joshi <44757683+sachin-j-joshi@users.noreply.github.com> Date: Thu Mar 12 19:18:37 2020 -0700 Issue 4609: ExtendedS3Storage.concat using appends when source segment is small (#4617) With this change , for smaller source segments, concat should read complete source segment and append it to target instead of using multi-part upload. For larger source segments continue using multi part upload method. Signed-off-by: Sachin Joshi commit 48bcf7b7922b4e0260daf81aa4055bc29774733f Author: Andrei Paduroiu Date: Thu Mar 12 14:45:51 2020 -0700 Issue 4618: (SegmentStore) Fixed a possible infinite loop in StorageWriter for empty appends. (#4619) Fixed a bug in SegmentAggregator where it could enter an infinite async loop if it receives an empty append. Signed-off-by: Andrei Paduroiu Co-authored-by: Raúl Gracia commit 6c6dca38aef681f1be57d9b81965c33f8b03d61f Author: Andrei Paduroiu Date: Wed Mar 11 17:27:23 2020 -0700 Issue 4610: (SegmentStore) Read Index data corruption during concurrent segment merge and cache eviction. (#4611) Fixed a bug where a concurrent segment merge and cache eviction could corrupt the Read Index. Signed-off-by: Andrei Paduroiu Co-authored-by: Raúl Gracia commit 3a94d10912af0fd714b01d79b9a216849e43c6b2 Author: Sandeep Date: Wed Mar 11 12:46:44 2020 +0530 Issue 4600: Fix Transaction related logs. (#4599) * Simplify debugging in case of deadline exceed exception on the controller client. * Reduce logging on the client. * Improve transaction related logs on the controller. * Ensure an empty response from SSS should not log a WARN message. Signed-off-by: Sandeep commit efe501a3e897e322d0561f4bbacfe00f7f3a3b29 Author: Raúl Gracia Date: Tue Mar 10 10:22:22 2020 +0100 Issue 4597: Optimization of String splits for Metrics (#4598) Signed-off-by: Raúl Gracia commit ec428ff982d31e08637b14fe1eddb8b118df6db8 Author: Sachin Jayant Joshi <44757683+sachin-j-joshi@users.noreply.github.com> Date: Mon Mar 9 18:20:30 2020 -0700 Issue 4601: (RollingStorage/ExtendedS3) S3 client consumes whole InputStream and causes java.lang.IllegalArgumentException (#4602) RollingStorage.write sends in BoundedInputStream to storage adapters to prevent reading more data. S3 client normally consumes whole InputStream and causes java.lang.IllegalArgumentException. Also, S3 client mocks simulate this InvalidRange behavior to match how the real system behaves. Signed-off-by: Sachin Joshi commit b57bf6c55b3712bceb5a3b938b3f23cba494e9e0 Author: Andrei Paduroiu Date: Mon Mar 9 16:45:27 2020 -0700 Issue 4590: (SegmentStore) Fixed Read Index concurrent Tier 2 read bug (Part 2) (#4596) Fixed a (second) bug in StreamSegmentReadIndex where it would have been possible for two concurrent Storage read requests to override one another, resulting in potential (in-memory) data loss should there be an append executed in between the two. Signed-off-by: Andrei Paduroiu commit 0e30ccfcdd107beb2f49d8e1623b7c915b974ac3 Author: Andrei Paduroiu Date: Fri Mar 6 07:37:06 2020 -0800 Issue 4594: Changed `pravegaVersion` to 0.8.0-SNAPSHOT in master. (#4595) Signed-off-by: Andrei Paduroiu commit 61a4b380a2efbac5296e8daafb98ec7ffcba7dad Author: Andrei Paduroiu Date: Wed Mar 4 19:45:45 2020 -0800 Issue 4591: Fixed ExtendedS3 exists() (#4593) This change uses getObjectMetadata instead of listObjects to implement exists. This fixes the problem where listObject based check behaved incorrectly when prefix instead of full path is provided as parameter. Signed-off-by: Sachin Joshi commit b919af736c3492bbb270d855b71410f62961c77d Author: Raúl Gracia Date: Wed Mar 4 21:15:36 2020 +0100 Issue 4590: (SegmentStore) Fixed Read Index concurrent Tier 2 read bug (#4592) Fixed a bug in StreamSegmentReadIndex where it would have been possible for two concurrent Storage read requests to override one another, resulting in potential (in-memory) data loss should there be an append executed in between the two. Signed-off-by: Andrei Paduroiu commit eafdda723d33cd30cc185e8294fe9c925d858484 Author: Kevin Han Date: Wed Mar 4 08:22:02 2020 -0800 Issue 4561 Fixed common.sh config.uri detection (#4589) Changed the approach to use "grep" to test substring. Signed-off-by: kevinhan88 commit 310f0c30c717d8ca4b758985f6b366cbf66f907f Author: Kevin Han Date: Mon Mar 2 13:45:33 2020 -0800 Issue 4561: Using S3 Rest Style Config URI for Extended S3 client construction (#4566) Using S3 Rest Style Config URI for Extended S3 client construction. Signed-off-by: kevinhan88 commit a3e03c69e0580597a338996046ea010bfc164900 Author: Kevin Han Date: Fri Feb 28 12:37:51 2020 -0800 Issue-4581: Replaced the unsafe while loop with iterator looping (#4582) Replaced the unsafe while loop in cleaning up testing registries with iterator, otherwise it may sometimes throw ConcurrentModificationException. Signed-off-by: kevinhan88 commit aee4d6c45c485493ab5c8a3932a57ad452b95015 Author: Andrei Paduroiu Date: Fri Feb 28 05:53:56 2020 -0800 Issue 4565: (SegmentStore) Improving Data Frames memory allocation (#4567) Changed the underlying buffer for DataFrame from a ByteArraySegment backed by a single byte array to a memory efficient composite array that only allocates memory when needed. Signed-off-by: Andrei Paduroiu commit 26d86eb6f1e13f3999b0df20d99f55950adc36d6 Author: Tom Kaitchuck Date: Thu Feb 27 19:51:35 2020 -0800 Issue 4578: Early wake reader in response to server reply. (#4579) Signed-off-by: Tom Kaitchuck commit a33969de73ac5d9a7da668c7a9bacdd578645339 Author: Andrei Paduroiu Date: Thu Feb 27 14:02:26 2020 -0800 Issue 4576: (SegmentStore) Fixed Cache Manager bugs (#4577) * Fixed a bug in CacheManager where it would incorrectly calculate the Newest Generation of a CacheStatus collection if at least one of them has no meaningful data. * Fixed a bug in CacheManager where it would not refresh the client status after an eviction which could lead to over-increasing the Oldest Generation, leading to excessive cache eviction. Signed-off-by: Andrei Paduroiu commit 5ccbaef7c5290ad9be670019ec5c87f32021dceb Author: Andrei Paduroiu Date: Wed Feb 26 14:58:46 2020 -0800 Issue 4574: (SegmentStore) Fixing logging verbosity in AttributeAggregator (#4575) Fixing logging verbosity when unable to persist root pointers in AttributeAggregator due to the segment having been deleted or merged into another. Signed-off-by: Andrei Paduroiu commit 7b6977bd6081658a3c26f932de8e37c54bfd9d63 Author: Kevin Han Date: Mon Feb 17 14:57:15 2020 -0800 Issue-4550: Renamed ExtendedS3 config item from root to prefix. (#4551) Renamed Pravega ExtendedS3 Adapter config item from "root" to "prefix". Signed-off-by: kevinhan88 commit 7cc2b8cb2ff3989ecf6d74e0978eb3e3c78c26e4 Author: Raúl Gracia Date: Fri Feb 14 15:55:17 2020 +0100 Issue 4558: Changed Default BK Digest type to CRC32C (#4559) Add configuration option for Bookkeeper digest type. Changed default Bookkeeper digest type to CRC32C. Signed-off-by: Raúl Gracia commit a199729b9537de100eca42fd80ab00e825b6f4b9 Author: Kevin Han Date: Thu Feb 13 15:44:27 2020 -0800 Issue 4149: Reduce metrics cache eviction to 3 minutes (#4117) Reduced the metrics cache eviction timeout to 3 minutes (from 30 minutes) Signed-off-by: kevinhan88 commit 9e9a53064e7ed50cd64ddce5091700e57491fe4f Author: bibin sebastian Date: Wed Feb 12 21:43:54 2020 +0530 Issue 4556: Enabling doclint on all modules (#4557) Enabling doclint on all modules. Signed-off-by: Bibin Sebastian commit 3b0068f8e1606993601f70b123587e1b63286f34 Author: bibin sebastian Date: Tue Feb 11 20:45:03 2020 +0530 Issue 4552: Reduce Javadoc errors (#4555) Issue 4552: Reduce Javadoc errors (#4355) Fixing Javadoc errors Signed-off-by: Bibin Sebastian commit fc2ce08987ad2035948b84706b4bb0ebb44d986d Author: anishakj <43978302+anishakj@users.noreply.github.com> Date: Tue Feb 11 15:41:48 2020 +0530 Issue 4338: Added new healthcheck for zookeeper (#4553) Added new healthcheck for zookeeper. Signed-off-by: anishakj commit 304ddb7ef558e9ab9df512098b689a693d02096b Author: Raúl Gracia Date: Fri Feb 7 16:32:43 2020 +0100 Issue 4509: Rack-aware Bookkeeper client config (#4511) Allow the Bookkeeper client in the Segment Store to be configured with rack-aware ensemble placement policy. Signed-off-by: Raúl Gracia commit 7f73a83a20e620ab3c5fde109698882abffd04ad Author: Tom Kaitchuck Date: Wed Feb 5 15:41:37 2020 -0800 Issue 4538: Adjust log level (#4539) Lower log level on segment truncated Signed-off-by: Tom Kaitchuck Co-authored-by: Raúl Gracia commit 4a312a8b994ff0c85f0045b68f2cb68af65923b7 Author: anishakj <43978302+anishakj@users.noreply.github.com> Date: Wed Feb 5 14:09:11 2020 +0530 Issue 4549: Removed the default git-clone depth (#4547) Signed-off-by: anishakj commit 8b9c69d765f9676b11969bb8e9d37c0f07c0edf3 Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Mon Feb 3 20:28:41 2020 +0530 Issue 4456: Consolidate get active txn calls during commit transactions for multiple transactions (#4499) Signed-off-by: Shivesh Ranjan commit 34f84bd11119f49494bb69c94c32967de57b8531 Author: Colin Hryniowski Date: Fri Jan 31 10:36:03 2020 -0800 Issue 4527: Include RequestIDs in the AppendProcessor error/warning logs. (#4537) Remove @slf4j annotation and replace the logger (as is done in PravegaRequestProcessor.java). As with PravegaRequestProcessor.java, supply the error/warn logs with a requestId, via the first parameter (the logs 'marker'). Signed-off-by: co-jo commit 84b060db2bac256472e27a4b8984707d86909a8c Author: Colin Hryniowski Date: Fri Jan 31 09:12:59 2020 -0800 Issue 4318: Replace `ByteArraySegment.subSegment()` with `ByteArraySegment.slice()`. (#4536) Replace subSegment(a,b) calls with the slice method. Signed-off-by: co-jo commit 4bf247ffb63dffaeeb30d5fcbd552363f420ff1e Author: Sandeep Date: Fri Jan 31 03:33:05 2020 +0530 Issue: Prevent readers from waiting indefinitely if there is a partial/no response from the Segment Store. (#4525) * Ensure Revisioned Stream resends the read request if the Segment store does not respond within 30 seconds after confirmation that the write-offset is greater than the current read-offset. * Ensure the Event Stream Reader resends the read request if the complete WireCommand frame is not received within 30 seconds. Co-authored-by: Tom Kaitchuck commit d513b903c01fc164cabc77ceccc7856e57e9bda7 Author: bibin sebastian Date: Thu Jan 30 15:06:09 2020 +0530 Issue 4532: Typo in Javadoc for io.pravega.client.stream.StreamConfiguration (#4533) Fixed method syntax of EventStreamReader#getCurrentTimeWindow(Stream) method in javadoc. Signed-off-by: Bibin Sebastian commit 758dde0aa41be92529b3e9197d98816e4952f101 Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Thu Jan 30 11:54:13 2020 +0530 Issue 4529: From Seal Stream workflow, call abort Txn even for transactions that are in Aborting state (#4530) Signed-off-by: Shivesh Ranjan commit 373f822d37b4baae2036766b18fdf9a4bcbcbe01 Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Wed Jan 29 17:42:58 2020 +0530 Issue 4478: Remove cached watermarking client for deleted stream (#4528) Signed-off-by: Shivesh Ranjan commit fe5daada018f15ddc494ef762512f4677b3194c3 Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Tue Jan 28 07:00:57 2020 +0530 Issue 4513, 4514: Readers from shutdown Controller's Event Processor's be present in readergroups after a failover #4523 The issue was observed on docker swarm system tests because readers in controller's event processors were not removed even as controller was shutdown. this meant that segments assigned to the reader were never reassigned and hence workflows were never executed. Signed-off-by: Shivesh Ranjan Co-authored-by: Raúl Gracia commit 97d28870edad884d9c20f78d53e762d1e47cd0ef Author: Ravi Sharda Date: Mon Jan 27 14:25:58 2020 +0530 Issue 4518: Upgrade Apache Bookkeeper Server version to 4.9.2 (#4519) Signed-off-by: Ravi Sharda commit e49091fbb8cee7ab5e4576314a896c1a6fd1f24d Author: Flavio Junqueira Date: Fri Jan 24 19:59:43 2020 +0100 Issue 4516: Enforce copyright through Checkstyle (#4517) Changes the checkstyle configuration to validate the copyright statement in license headers. Signed-off-by: Flavio Junqueira (fpj) commit c7ac009970787df2633163644691aa3d72fc1a4e Author: Flavio Junqueira Date: Fri Jan 24 18:55:36 2020 +0100 Issue 3508: Remove year from copyright in license header (#4515) Removes the year in the copyright statement of license headers. Signed-off-by: Flavio Junqueira (fpj) commit b8911f8f4c0b452d18cb34e3d6ce23a13d6eccee Author: Ravi Sharda Date: Fri Jan 24 16:34:47 2020 +0530 Issue 3696: Upgrades Bookeeper from 4.7.3 to 4.8.2 (#4420) * Upgrades the Bookeeper server from 4.7.3 to 4.8.2. Signed-off-by: Ravi Sharda commit 400c1aa9910b695cc62dfa2897c4ea8f85be79d8 Author: Sachin Jayant Joshi <44757683+sachin-j-joshi@users.noreply.github.com> Date: Thu Jan 23 07:55:08 2020 -0800 Issue 4505: Add missing metrics for ExtendedS3Storage. (#4510) Add missing metrics for ExtendedS3Storage. Signed-off-by: Sachin Joshi commit 9ccbb1e8494e7fbc33dc4681017bd5ca0286b24f Author: Tom Kaitchuck Date: Wed Jan 22 19:47:12 2020 -0800 Issue 4385: Limit characters that can appear in a readerId or a writerId (#4387) Signed-off-by: Tom Kaitchuck commit 182977031aaf305fce0157aa8158159c8748900d Author: Wenxiao Zhang Date: Wed Jan 22 06:39:51 2020 -0800 Issue 4490: Improve error logging on client (#4498) Signed-off-by: Wenxiao Zhang commit b2e0fa437042de35940210fe0731b3f4712d4c0f Author: Flavio Junqueira Date: Wed Jan 22 11:00:11 2020 +0100 Issue 4132: Update slack token and channel information (#4508) Add an entry to enable Travis notifications to pravega-io slack and remove stale email addresses. Signed-off-by: Flavio Junqueira (fpj) commit 40b38f44c1621c03c09b5ffbb1103eccedcb07f3 Author: Andrei Paduroiu Date: Tue Jan 21 14:47:49 2020 -0800 Issue 4495: Recording WARN and ERROR log entries as metrics. (#4496) Recording all WARNs and ERRORs as metrics with logging class name and exception name as tags. Signed-off-by: Andrei Paduroiu commit 4f578e270cd298587a9f867a84aa5c2b24a33359 Author: Andrei Paduroiu Date: Fri Jan 17 16:33:14 2020 -0800 Issue 4451: (SegmentStore) Enhancing Segment Attribute Tier 2 Sync (#4485) * Storing Segment Attribute BTreeIndex Root Pointer as a Core Attribute on the parent segment, every time the BTreeIndex is updated. * Split out the Tier 2 Attribute sync logic out of SegmentAggregator into its own class (AttributeAggregator). * Allowing certain UpdateAttributeOperations to be executed after the Segment is sealed. Signed-off-by: Andrei Paduroiu commit 43dd13dd65236f846b372f8b09eac3d1e94b0d04 Author: Sandeep Date: Sat Jan 18 04:57:32 2020 +0530 Issue 4457: Improve client javadoc and logging. (#4469) Improve client javadocs and logging. Signed-off-by: Sandeep Co-authored-by: Raúl Gracia commit dc5d2461580f0d15b1670af2deab43ef1b69ad52 Author: Andrei Paduroiu Date: Fri Jan 17 13:27:19 2020 -0800 Issue 4444: (SegmentStore) Fine-tuning Cache Throttler (#4447) The CacheThrottler now also accounts for in-flight appends when calculating Cache Utilization. Changed the OperationProcessor to fetch fewer items from its operation queue if the cache nears its utilization targets. Signed-off-by: Andrei Paduroiu commit 06867211913cceb9481c5662eb70103a4fd40f05 Author: Sandeep Date: Fri Jan 17 16:16:22 2020 +0530 Issue 4443: Use DelegationTokenProvider for BatchClient. (#4491) * Leverage DelegationTokenProvider to retrieve the delegation token on BatchClient. commit 8a206e1ed5b3a0604d318b2e2eed9a2b04b2e073 Author: Sandeep Date: Fri Jan 17 14:15:34 2020 +0530 Issue 4492: RevisionedStreamClient with response timeout log. (#4430) Ensure the RevisionedStreamClient logs a warning there is a delayed response from Segment store. Signed-off-by: Sandeep commit be42b9deaff1bac39cf4daa20834bbf282340718 Author: Andrei Paduroiu Date: Wed Jan 15 13:52:50 2020 -0800 Issue 4488: (SegmentStore) Added metrics for StorageWriter (#4489) Added metrics for StorageWriter. Signed-off-by: Andrei Paduroiu commit 94482e51e5324075d0be8d9c81a3871faaa5ef32 Author: Tom Kaitchuck Date: Tue Jan 14 01:22:57 2020 -0800 Issue 4416: Remove deprecated client APIs (#4417) Signed-off-by: Tom Kaitchuck commit 44780798accc3c429c560fde2e88ca4e867e7a07 Author: Raúl Gracia Date: Tue Jan 14 00:33:29 2020 +0100 Issue 4027: Add Metric for Container Recovery Time (#4028) Added a new metric to keep track of container recovery times. Signed-off-by: Raúl Gracia commit 75e1b1600b90601680645ead96f840f17d144ec8 Author: Sachin Jayant Joshi <44757683+sachin-j-joshi@users.noreply.github.com> Date: Mon Jan 13 12:56:33 2020 -0800 Issue 4453: Fix bug in FileSystemStorage.doRead (#4479) Fixed a bug in FileSystemStorage.doRead which could cause incorrect data to be read from Storage. Signed-off-by: Sachin Joshi commit e50f18276528fff0fb9b26da0708ad6666ea617d Author: Andrei Paduroiu Date: Mon Jan 13 01:06:21 2020 -0800 Issue 4452: (SegmentStore) Fixed Read Index Append Concurrency bug (#4461) Fixed a bug in StreamSegmentReadIndex where it would have been possible for an append data to get lost if the append method was invoked concurrently with a cache eviction (updateGenerations). Signed-off-by: Andrei Paduroiu commit b5ad15dd01ea8733a0a345d8605ff7a04b01ed29 Author: Andrei Paduroiu Date: Mon Jan 13 00:02:04 2020 -0800 Issue 4454: Reading BK Ledger Entries in batches (#4455) When performing a Ledger Read (Segment Container Recovery) the Ledger is now read in batches, as opposed from all at once. Signed-off-by: Andrei Paduroiu commit bac577a4051408d3f4ac07228fe733dd632c8876 Author: Raúl Gracia Date: Fri Jan 10 15:29:26 2020 +0100 Issue 4467: Wait for metrics to be available before asserting their content. (#4480) * Ensure that metrics are available in StreamMetricsTest.testTransactionMetrics before performing assertions. Signed-off-by: Raúl Gracia commit 8762b3913fa2066670291e91f79b7dd8167bd4cc Author: Tom Kaitchuck Date: Thu Jan 9 22:04:51 2020 -0800 Issue 1546: Remove thread from ControllerResolverFactory (#4179) * Use the executor from the controllerImpl to do dns lookups of the controller. * Use the provided thread pool Signed-off-by: Tom Kaitchuck commit 6222d9618febac4032abf62747544227d5087944 Author: Raúl Gracia Date: Wed Jan 8 12:56:34 2020 +0100 Issue 4460: BookieFailover Test sporadic failure (#4477) * Increased retry timeout in SegmentMetadataClientImpl to mitigate test failures due to slow Bookie restart and added read event assertions in BookieFailoverTest. Signed-off-by: Raúl Gracia commit b8c18b2ad8118d35f950da08b4edca6314b70941 Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Tue Jan 7 10:45:41 2020 +0530 Issue 2183: Rebalance readers in event processors when new controllers are added (#4472) Signed-off-by: Shivesh Ranjan commit 819d4e983013db8e3a15cb057651ee3552f8ec45 Author: Sandeep Date: Fri Dec 27 12:27:57 2019 +0530 Issue 4471: Fix controller client channel termination logic (#4470) * Fix a cause for the grpc/netty threads from not terminating. * Ensure channel is closed if there are exceptions while setting credentials. * Junit test. * Enable mock of final classes. Signed-off-by: Sandeep commit 9909d015f3f54693c136a0be83ea3330c5f7305d Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Thu Dec 26 15:28:32 2019 +0530 Issue 4465: Provide api to expose reader to segment-count distribution in readergroup (#4466) Signed-off-by: Shivesh Ranjan Co-authored-by: Sandeep commit f0f13143baeb1552f29f7b807ea88d594150f082 Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Thu Dec 26 11:17:16 2019 +0530 Issue 4445: Check to ensure a stream does not scale below minimum configured segments (#4450) Signed-off-by: Shivesh Ranjan Co-authored-by: Sandeep commit f4bdee2b88f875657dc85539e2ef6d624e5a0f42 Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Mon Dec 23 20:42:33 2019 +0530 Issue 4368: Sporadic failure in Watermark workflow test(#4468) Signed-off-by: Shivesh Ranjan commit 91b6dbd5ef768e0e0872baf5d4f7c55196918386 Author: Raúl Gracia Date: Wed Dec 18 18:04:44 2019 +0100 Issue 4426: Transaction performance metrics (#4437) * Added and refactored transaction-related metrics to understand better transaction performance. Signed-off-by: Raúl Gracia commit 290b6a0de32e432d112981321de6c9ea1a83a400 Author: anishakj <43978302+anishakj@users.noreply.github.com> Date: Tue Dec 17 14:42:24 2019 +0530 Issue 4458: Fixed docker swarm system test failure (#4462) Fixed a naming collision among two system tests that made Docker Swarm system tests to fail. Signed-off-by: anishakj commit e012ebb1b0df2c1c0f003b25fe1ceede97835a16 Author: Sandeep Date: Mon Dec 16 22:47:37 2019 +0530 Issue 4436: Ensure the refreshed token is used during a read retry. (#4435) Ensure the latest token is used during a retry on segment reader. Signed-off-by: Sandeep commit 5d0176839755e82f64329938466099e25088c3ad Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Sat Dec 14 21:39:14 2019 +0530 Issue 4448: Add withDeadlineAfter to each rpc call in controller client (#4449) Adds deadline to each GRPC call individually. Signed-off-by: Shivesh Ranjan commit 03d0cff98f8ddf44c584f01986a3394671b73096 Author: Andrei Paduroiu Date: Fri Dec 13 06:19:13 2019 -0800 Issue 4204: (SegmentStore) Fixed excessive cache eviction bug (#4439) Fixed a bug in CacheManager where it could excessively evict items from the cache as soon as they become eligible even if there was no cache pressure present. Signed-off-by: Andrei Paduroiu commit 1449fe95f8fa27b6d0dcc565f7bd4ae5c46ab80a Author: Andrei Paduroiu Date: Fri Dec 13 03:39:55 2019 -0800 Issue 4428: (SegmentStore) Throttling improvements (#4440) Removed CommitBacklog throttler and optimized BookKeeperLog.getQueueStatistics to avoid extra locking overhead. Signed-off-by: Andrei Paduroiu commit 4fda6c63e004695bd6551a08464e0a3659639ddc Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Thu Dec 12 11:24:25 2019 +0530 Issue 4403: Add deadline to all grpc calls to controller (#4421) Signed-off-by: Shivesh Ranjan commit 0f08670c962f1180658db83d8281256a53b94934 Author: Andrei Paduroiu Date: Wed Dec 11 11:50:22 2019 -0800 Issue 4044: (Segment Store) Direct Memory Cache (#4074) * Creates a block-based, direct memory cache implementation and integrates with the Segment Store. * Removes old cache implementations (RocksDB and InMemoryCache). Signed-off-by: Andrei Paduroiu commit fddfc458dc6fa4d365b353e20ba13228b72ecd8a Author: Sachin Jayant Joshi <44757683+sachin-j-joshi@users.noreply.github.com> Date: Mon Dec 9 15:10:19 2019 -0800 Issue 4412: Fix bug in ExtendedS3StorageConfig to correctly handle missing trailing '/' in root. (#4413) Fix bug in ExtendedS3StorageConfig to correctly handle missing trailing `/` in root. Signed-off-by: Sachin Joshi commit 1b79a08fc56b9142c61e0eae9826e8baf9334842 Author: Sachin Jayant Joshi <44757683+sachin-j-joshi@users.noreply.github.com> Date: Mon Dec 9 12:51:24 2019 -0800 Issue 3503: Disable HDFSIntegrationTest. (#4427) The HDFSIntegrationTest has been unstable and fails with intermittent failures. This causes builds with changes unrelated to HDFS to fail. Disabling HDFSIntegrationTest for a short while until it gets completely rewritten/replaced as part of PDP-34. Signed-off-by: Sachin Joshi commit fd88b66c05caa2ca9aaffacc2180d81d763cbb44 Author: Tom Kaitchuck Date: Sun Dec 8 18:01:51 2019 -0800 Issue 3679: Increase maximum event size limit. (#4370) - Increases the size limit for an event from 1MB to 8MB - Increases the wire command size limit from 8MB to 16MB Signed-off-by: Tom Kaitchuck commit b055a5d7945a9978f7448e5d849a975fffbacca0 Author: Tom Kaitchuck Date: Sun Dec 8 16:57:26 2019 -0800 Issue 4303: Avoid Netty Flush from the Client (#4294) - Avoids calling netty's channel.flush() - Avoids blocking on the futures returned from write() unless it is needed. Signed-off-by: Tom Kaitchuck commit 1b0cd39c8e638a6f86291c5011d4e98af5d73cec Author: Andrei Paduroiu Date: Fri Dec 6 11:56:05 2019 -0800 Issue 4393: Fixing error sequencing in AppendProcessor (#4418) Changed the AppendProcessor to send error messages back to the client only after all appends initiated prior to that error complete. Signed-off-by: Andrei Paduroiu commit ad9260d5d2e58e7262732818b8b2bf71ef325198 Author: Ravi Sharda Date: Mon Dec 2 23:51:29 2019 +0530 Issue 4409: Reset tokenRefreshFuture upon exception in obtaining a new delegation token (#4410) Fix a bug that prevents tokenRefreshFuture from getting reset. Signed-off-by: Ravi Sharda commit 5583375113d3a069a5fcad057d3b684f527f9076 Author: Andrei Paduroiu Date: Mon Dec 2 07:34:30 2019 -0800 Issue 4399: BookKeeper Write Throttling (#4402) Added a throttling mechanism in the Segment Store to prevent Bookkeeper from being overloaded. Signed-off-by: Andrei Paduroiu commit 072b2e0ea3324fe0347aff55be66bdf3780750e3 Author: Tom Kaitchuck Date: Wed Nov 27 18:56:33 2019 -0800 Issue 4147: Adjust txn ping to perform multiple pings per transaction (#4400) Attempt multiple pings before timing out a transaction. Signed-off-by: Tom Kaitchuck commit 32e7ec752ad0409b0800145ac38a2652584657e5 Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Tue Nov 26 19:43:59 2019 +0530 Issue 4396: Respond with meaningful status to ping transaction if transaction is already committed or aborted(#4397) Signed-off-by: Shivesh Ranjan commit 9d9591d764baafbcd3ec626743f78955991bada1 Author: Ravi Sharda Date: Mon Nov 25 22:51:16 2019 +0530 Issue 4390: Ensure retries by client when delegation token expires (#4391) Ensure that the client retries the request to segment store if the segment store throws an error indicating token expiry. Doing so will ensure that a new delegation token will be obtained after reauthentication and reauthorization with the Controller and used in the retried request. Signed-off-by: Ravi Sharda commit 51890c14ff8f761d7f6e190cad86f4b01a39464f Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Fri Nov 22 17:44:15 2019 +0530 Issue 4020: Expose additional apis on StreamMetadataStore interface for Pravega Cli (#4380) Added new calls to StreamMetadataStore to get additional metadata of a Stream. Signed-off-by: Shivesh Ranjan commit 4f7a8b89dbe598b08f32f31ac4c676a60eeba29f Author: Andrei Paduroiu Date: Thu Nov 21 10:37:49 2019 -0800 Issue 4369: (SegmentStore) Fix memory leak with Future Reads (#4373) Fixed a memory leak that can be triggered by abandoning Future Reads without any segment write activity. Signed-off-by: Andrei Paduroiu commit 502f35b7c31dfa6b02665f6b9984ced69369b31a Author: Andrey Kurilov Date: Thu Nov 21 17:20:58 2019 +0300 Issue 4374: Optimize the updateSegmentTags method (#4377) Optimize string manipulation in updateSegmentTags() function. Signed-off-by: Andrey Kurilov commit 83f5de24f4d9d972294e743be3639580053b2555 Author: Sandeep Date: Thu Nov 21 03:15:28 2019 +0530 Issue 4383: Fix exception handling during state synchronizer compaction. (#4382) Ensure we do not retry if the segment is already truncated while attempting to truncate a segment. Signed-off-by: Sandeep commit 3083b387f227d61e7dd5727db9b23d17d2976691 Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Wed Nov 20 13:31:10 2019 +0530 Issue 4357: Call noteWriterMark once per writer after committing a batch of transactions (#4378) Signed-off-by: Shivesh Ranjan commit 1de5073c4c0cc90f52a7e3151bef8722c0419636 Author: Sandeep Date: Wed Nov 20 11:08:49 2019 +0530 Issue 4372: Split controller thread pools to avoid potential deadlock. (#4381) * Split controller thread pools to avoid deadlock. * Ensure event writes are performed on a different executor. Signed-off-by: Sandeep commit db0f2b90c775eb001b52b5c1e19d191c41b79981 Author: Ravi Sharda Date: Thu Nov 14 12:20:20 2019 +0530 Issue 4352: Turn retrieve token operation of DelegationTokenProvider asynchronous (#4356) * Make the DelegationTokenProvider interface operation retrieveToken() asynchronous and make its invocations in client non-blocking, in order to avoid client-side deadlocks. Signed-off-by: Ravi Sharda commit 63af346ea8ba5d1fb62c328b79461d7f287330b1 Author: Andrei Paduroiu Date: Wed Nov 13 14:18:20 2019 -0800 Issue 4351: Removed assertion check from ConnectionTracker (#4360) Removed an assertion check that could incorrectly fire in concurrent tests. Signed-off-by: Andrei Paduroiu commit f617a01769a1dac640b04fcebbff6fcfceb4af01 Author: Tom Kaitchuck Date: Wed Nov 13 11:02:34 2019 -0800 Issue #4340: Remove warnings from missing annotations (#4341) Added missing override annotations. Signed-off-by: Tom Kaitchuck commit 719995ba614918a1a96eead79f9c4b10a6cccd89 Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Tue Nov 12 09:13:51 2019 +0530 Issue 4349: Complete partially created scopes before throwing ScopeExists (#4350) Signed-off-by: Shivesh Ranjan commit bc6a9e9edfc9fe04c2ba5081dd8dc69735cb1ae6 Author: Kevin Han Date: Mon Nov 11 09:32:09 2019 -0800 Issue 4260: No-Op Storage Adapter (#4302) - Created a thin No-Op (of payload) layer on top of SyncStorage, which doesn't change any existing functionality but skips the Storage payload writing (system segments still go through) - Introduced two new config properties for the No-Op layer - StorageLoader now detects the new config and load the new No-Op factory if necessary - Created a new unit test to prove that existing functionalities still work, and the new No-Op layer works as expected. Signed-off-by: kevinhan88 commit b01a0810b07da33286ea3a679726231ceca0de1f Author: Ravi Sharda Date: Fri Nov 8 11:23:00 2019 +0530 Issue 4334: Fix auth and related exception handling (#4336) Improve exception handling and logging in gRPC interface to distinguish clearly among conditions representing authentication, authorization and internal failures. Signed-off-by: Ravi Sharda commit 4dd4d240c1a931975e87f9712bd0a11513e9a210 Author: Andrei Paduroiu Date: Thu Nov 7 10:36:19 2019 -0800 Issue 4239: AppendProcessor pipelining (#4331) Changed AppendProcessor to pipeline requests through the underlying SegmentStore. Changed the AppendProcessor's TCP back-throttling logic to consider all connections when deciding which connection to pause or resume. Signed-off-by: Andrei Paduroiu commit c4e6dd85c1d8c662cbae953f68533c485d13fafc Author: Tom Kaitchuck Date: Wed Nov 6 12:36:20 2019 -0800 Issue 4274: Write padding more efficiently (#4275) Write padding more efficiently instead of one Long at a time. Signed-off-by: Tom Kaitchuck commit 91597f3c28b0c9d51ef7da2348585e3eb1b82370 Author: Andrei Paduroiu Date: Wed Nov 6 02:57:05 2019 -0800 Issue 4283: Fix SelfTester (#4284) Improvements in SelfTester tool. Signed-off-by: Andrei Paduroiu commit 26bfc25eca2e76e691bc32177ef2500cccb95879 Author: Raúl Gracia Date: Mon Oct 28 16:14:08 2019 +0100 Issue 4325: Sporadic OOMKilled in Segment Store under moderate workloads (#4326) Partial revert of new RocksDB config (leave only new parallelism settings). Signed-off-by: Raúl Gracia commit a83894b8b34fd1da0bf3e86ad8da8512cc8058f7 Author: Raúl Gracia Date: Fri Oct 25 04:45:32 2019 +0200 Issue 4319: Remove RocksDB option that induces link conflicts in Windows. (#4320) Removed RocksDB option that induces link conflicts in Windows. Signed-off-by: Raúl Gracia commit ef40b45b7d13e8411100c16fead2401fce234a58 Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Fri Oct 25 03:43:06 2019 +0530 Issue 4092: Change default store in InProcPravegaCluster to Pravega tables (#4293) Change default store in InProcPravegaCluster to Pravega tables instead of Zookeeper. Signed-off-by: Shivesh Ranjan commit e81f6abd3e65d0e86ce85cfc6fc313fdfcabb043 Author: Sandeep Date: Thu Oct 24 18:24:37 2019 +0530 Issue 4280: Reduce the amount of processing performed on EventLoop thread by SegmentOutputStream. (#4264) Reduce the amount of processing performed on the EventLoop thread by the Segment writer post receiving a WireCommand.DataAppended. Signed-off-by: Sandeep commit 7c0c026017f334fe6164a6d9559306667f675d88 Author: shivesh ranjan <13659022+shiveshr@users.noreply.github.com> Date: Wed Oct 23 19:48:22 2019 +0530 Issue 4310: Change Pravega version in master to 0.7.0-SNAPSHOT (#4311) Signed-off-by: Shivesh Ranjan