Implement light subscriptions stats and refactor #861

lmontrieux · Apr 15, 2018

Implement light subscriptions stats and refactor

Zalando ticket : ARUHA-1551

Description

light subscription stats in /subscriptions endpoint
refactor Zk subscription client to not acquire locks to read
subscription data
remove ServiceUnavailableException from the project, preferring the
unchecked ServiceTemporarilyUnavailableException instead
update the swagger definition

Review

Tests
Documentation
CHANGELOG

codecov-io · Apr 15, 2018

Codecov Report

Merging #861 into master will decrease coverage by 0.03%.
The diff coverage is 62.9%.

@@             Coverage Diff              @@
##             master     #861      +/-   ##
============================================
- Coverage     53.59%   53.56%   -0.04%     
- Complexity     1680     1691      +11     
============================================
  Files           310      309       -1     
  Lines          9327     9359      +32     
  Branches        837      841       +4     
============================================
+ Hits           4999     5013      +14     
- Misses         4029     4046      +17     
- Partials        299      300       +1

Impacted Files	Coverage Δ	Complexity Δ
...alando/nakadi/service/CursorOperationsService.java	`59.8% <ø> (ø)`	`21 <0> (ø)`	⬇️
...org/zalando/nakadi/repository/TopicRepository.java	`0% <ø> (ø)`	`0 <0> (ø)`	⬇️
...nakadi/service/converter/VersionZeroConverter.java	`29.31% <ø> (ø)`	`5 <0> (ø)`	⬇️
.../nakadi/service/converter/CursorConverterImpl.java	`51.42% <ø> (ø)`	`7 <0> (ø)`	⬇️
...ava/org/zalando/nakadi/service/CursorsService.java	`10.3% <ø> (ø)`	`2 <0> (ø)`	⬇️
.../service/subscription/zk/ZkSubscriptionClient.java	`31.7% <ø> (ø)`	`0 <0> (ø)`	⬇️
...va/org/zalando/nakadi/service/CursorConverter.java	`87.5% <ø> (ø)`	`0 <0> (ø)`	⬇️
...alando/nakadi/controller/PartitionsController.java	`79.22% <ø> (-7.8%)`	`12 <0> (ø)`
...g/zalando/nakadi/controller/ExceptionHandling.java	`52.5% <ø> (ø)`	`12 <0> (ø)`	⬇️
...a/org/zalando/nakadi/service/BlacklistService.java	`13.04% <0%> (ø)`	`3 <0> (ø)`	⬇️
... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25e9497...e06f28d. Read the comment docs.

adyach · Apr 16, 2018

adyach reviewed Apr 16, 2018

View changes

adyach · Apr 16, 2018

docs/_data/nakadi-event-bus-api.yaml

@@ -964,6 +964,11 @@ paths:
          required: false
          default: 0
          minimum: 0
+        - name: show_status


do you think it is better to have separate endpoint for that ?

@adyach I followed the requirements in our internal ticket:
GET /subscriptions endpoint should have additional parameter show_status = true/false (false by default).

To be honest I don't really have a preference one way or another. Do you think it would be better as a separate endpoint?

v-stepanov · Apr 16, 2018

v-stepanov reviewed Apr 16, 2018

View changes

v-stepanov · Apr 16, 2018

docs/_data/nakadi-event-bus-api.yaml

+            type: object
+            description: statistics of partition within a subscription context
+            properties:
+              partition:


there's no description for partition field

v-stepanov · Apr 16, 2018

v-stepanov reviewed Apr 16, 2018

View changes

v-stepanov · Apr 16, 2018

src/main/java/org/zalando/nakadi/service/subscription/SubscriptionService.java

            throws InconsistentStateException, ServiceTemporarilyUnavailableException {
+        final List<EventType> eventTypes = getEventTypesForSubscription(subscription);
+        final List<PartitionEndStatistics> topicPartitions = loadPartitionEndStatistics(eventTypes);


As I see we do this call not looking at includeDistance value. But this method will actually go to Kafka and do this expensive seek operation. I thought that in case of light-stats we will not check the offsets in Kafka.

That is correct, I thought of doing it then forgot to do so. Looks like there will be some more refactoring.

@v-stepanov 9436dff

v-stepanov · Apr 16, 2018

v-stepanov reviewed Apr 16, 2018

View changes

v-stepanov · Apr 16, 2018

src/main/java/org/zalando/nakadi/service/subscription/zk/AbstractZkSubscriptionClient.java

            throws SubscriptionNotInitializedException, NakadiRuntimeException {
        if (!isSubscriptionCreatedAndInitialized()) {
            return Optional.empty();
        }

-        return Optional.of(runLocked(() -> new ZkSubscriptionNode(


Why have you removed running under lock?

Because it isn't necessary anymore, since we migrated to new zk subscription format (I double checked with @antban )

v-stepanov · Apr 17, 2018

v-stepanov reviewed Apr 17, 2018

View changes

v-stepanov · Apr 17, 2018

src/main/java/org/zalando/nakadi/service/subscription/SubscriptionService.java

+        if (includeDistance) {
+            return loadStats(eventTypes, zkSubscriptionNode, subscriptionClient);
+        }
+        else {


I used to put else on a new line like that but the team told me not to do that :(

@v-stepanov faaee92. What the team wants, the team gets :)

v-stepanov · Apr 18, 2018

@lmontrieux code looks good for me now.
Are there any tests for the new functionality?

v-stepanov · Apr 18, 2018

deploy validation please

v-stepanov · Apr 19, 2018

v-stepanov reviewed Apr 19, 2018

View changes

v-stepanov · Apr 19, 2018

src/main/java/org/zalando/nakadi/domain/Subscription.java

@@ -18,6 +20,8 @@ public Subscription(final String id, final DateTime createdAt, final Subscriptio

    private DateTime createdAt;

+    private List<SubscriptionEventTypeStats> stats;


maybe you can add @JsonInclude(Include.NON_NULL) here? Because in other case it will show up for all endpoints where we return subscription.

v-stepanov · Apr 19, 2018

v-stepanov reviewed Apr 19, 2018

View changes

v-stepanov · Apr 19, 2018

src/main/java/org/zalando/nakadi/domain/Subscription.java

@@ -34,6 +38,14 @@ public void setCreatedAt(final DateTime createdAt) {
        this.createdAt = createdAt;
    }

+    public List<SubscriptionEventTypeStats> getStats() {


probably it should be marked as nullable as it is null in most of the cases

v-stepanov · Apr 19, 2018

v-stepanov reviewed Apr 19, 2018

View changes

v-stepanov · Apr 19, 2018

src/main/java/org/zalando/nakadi/service/subscription/SubscriptionService.java

@@ -157,10 +157,16 @@ public Result listSubscriptions(@Nullable final String owningApplication, @Nulla
                    subscriptionRepository.listSubscriptions(eventTypesFilter, owningAppOption, offset, limit);
            final PaginationLinks paginationLinks = SubscriptionsUriHelper.createSubscriptionPaginationLinks(
                    owningAppOption, eventTypesFilter, offset, limit, subscriptions.size());


new field show_status should also be included to pagination link.

lmontrieux · Apr 19, 2018

@v-stepanov there are 2 acceptance tests in 2ffeb20

lmontrieux · Apr 19, 2018

👍

v-stepanov · Apr 19, 2018

v-stepanov reviewed Apr 19, 2018

View changes

v-stepanov · Apr 19, 2018

src/acceptance-test/java/org/zalando/nakadi/webservice/hila/SubscriptionAT.java

+    public void whenLightStatsOnNotInitializedSubscriptionThenCorrectResponse() throws IOException {
+        final String et = createEventType().getName();
+        final Subscription s = createSubscriptionForEventType(et);
+        final Response response = when().get("/subscriptions?show_status=true").thenReturn();


it's an acceptance test, what if there will be other subscriptions created by other tests? I think it's better to differ the subscription you created (e.g. by owning_application)

v-stepanov · Apr 19, 2018

v-stepanov reviewed Apr 19, 2018

View changes

v-stepanov · Apr 19, 2018

src/acceptance-test/java/org/zalando/nakadi/webservice/hila/SubscriptionAT.java

+            if (subscription.getId().equals(s.getId())) {
+                Assert.assertNotNull(subscription.getStats());
+                Assert.assertEquals("assigned", subscription.getStats().get(0).getPartitions().get(0).getState());
+                Assert.assertNotEquals("", subscription.getStats().get(0).getPartitions().get(0).getStreamId());


you can actually take streamId from TestStreamingClient and use it here for assertion.

v-stepanov · Apr 19, 2018

v-stepanov reviewed Apr 19, 2018

View changes

v-stepanov · Apr 19, 2018

src/acceptance-test/java/org/zalando/nakadi/webservice/hila/SubscriptionAT.java

+                .start();
+        waitFor(() -> assertThat(client.getBatches(), hasSize(15)));
+
+        final Response response = when().get("/subscriptions?show_status=true").thenReturn();


here it's also possible to have the same problem as I described several lines ago in another test (subscriptions left by other tests).

v-stepanov · Apr 19, 2018

👍

lmontrieux · Apr 19, 2018

👍

lmontrieux · Apr 19, 2018

Deploy validation please

lmontrieux · Apr 20, 2018

👍

lmontrieux · Apr 20, 2018

👍

v-stepanov · Apr 20, 2018

👍

v-stepanov · Apr 23, 2018

deploy validation please

v-stepanov · Apr 25, 2018

👍

antban · Apr 25, 2018

👍

Only get light stats for subscriptions in the returned page

Verified

This commit was signed with a verified signature.

lmontrieux Lionel Montrieux

GPG key ID: E130099683D7765F Learn about signing commits

Loading status checks…

da1c25c

formatting

Verified

This commit was signed with a verified signature.

lmontrieux Lionel Montrieux

GPG key ID: E130099683D7765F Learn about signing commits

Loading status checks…

faaee92

Improve acceptance tests

Verified

This commit was signed with a verified signature.

lmontrieux Lionel Montrieux

GPG key ID: E130099683D7765F Learn about signing commits

Loading status checks…

42210eb

lmontrieux changed the title from [WIP] Implement light subscriptions stats and refactor to Implement light subscriptions stats and refactor Apr 19, 2018

change stats to status

Verified

This commit was signed with a verified signature.

lmontrieux Lionel Montrieux

GPG key ID: E130099683D7765F Learn about signing commits

Loading status checks…

74fb913

Merge remote-tracking branch 'origin/master' into light-subscription-…
…stats # Conflicts: # CHANGELOG.md

Loading status checks…

e06f28d

zalando/nakadi

Join GitHub today

Implement light subscriptions stats and refactor #861

Conversation

lmontrieux commented Apr 15, 2018

Implement light subscriptions stats and refactor

Description

Review

lmontrieux added some commits Apr 5, 2018

This comment has been minimized.

codecov-io commented Apr 15, 2018 • edited

Codecov Report

adyach reviewed Apr 16, 2018 View changes

This comment has been minimized.

This comment has been minimized.

lmontrieux Apr 16, 2018 • edited

v-stepanov reviewed Apr 16, 2018 View changes

This comment has been minimized.

This comment has been minimized.

v-stepanov reviewed Apr 16, 2018 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

v-stepanov reviewed Apr 16, 2018 View changes

This comment has been minimized.

This comment has been minimized.

lmontrieux added some commits Apr 16, 2018

v-stepanov reviewed Apr 17, 2018 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

v-stepanov commented Apr 18, 2018

This comment has been minimized.

v-stepanov commented Apr 18, 2018

v-stepanov reviewed Apr 19, 2018 View changes

This comment has been minimized.

This comment has been minimized.

v-stepanov reviewed Apr 19, 2018 View changes

This comment has been minimized.

This comment has been minimized.

v-stepanov reviewed Apr 19, 2018 View changes

This comment has been minimized.

This comment has been minimized.

lmontrieux added some commits Apr 19, 2018

This comment has been minimized.

lmontrieux commented Apr 19, 2018

This comment has been minimized.

lmontrieux commented Apr 19, 2018

lmontrieux added some commits Apr 19, 2018

v-stepanov reviewed Apr 19, 2018 View changes

This comment has been minimized.

This comment has been minimized.

v-stepanov reviewed Apr 19, 2018 View changes

This comment has been minimized.

This comment has been minimized.

v-stepanov reviewed Apr 19, 2018 View changes

This comment has been minimized.

v-stepanov Apr 19, 2018 • edited

This comment has been minimized.

lmontrieux changed the title from [WIP] Implement light subscriptions stats and refactor to Implement light subscriptions stats and refactor Apr 19, 2018

This comment has been minimized.

v-stepanov commented Apr 19, 2018

This comment has been minimized.

lmontrieux commented Apr 19, 2018

This comment has been minimized.

lmontrieux commented Apr 19, 2018

lmontrieux added some commits Apr 20, 2018

This comment has been minimized.

lmontrieux commented Apr 20, 2018

This comment has been minimized.

lmontrieux commented Apr 20, 2018

This comment has been minimized.

v-stepanov commented Apr 20, 2018

This comment has been minimized.

v-stepanov commented Apr 23, 2018

This comment has been minimized.

v-stepanov commented Apr 25, 2018

This comment has been minimized.

antban commented Apr 25, 2018

Hide details View details v-stepanov merged commit f56de8a into master Apr 25, 2018 6 of 7 checks passed

codecov-io commented Apr 15, 2018 •

edited

adyach reviewed Apr 16, 2018

View changes

lmontrieux Apr 16, 2018 •

edited

v-stepanov reviewed Apr 16, 2018

View changes

v-stepanov reviewed Apr 16, 2018

View changes

v-stepanov reviewed Apr 16, 2018

View changes

v-stepanov reviewed Apr 17, 2018

View changes

v-stepanov reviewed Apr 19, 2018

View changes

v-stepanov reviewed Apr 19, 2018

View changes

v-stepanov reviewed Apr 19, 2018

View changes

v-stepanov reviewed Apr 19, 2018

View changes

v-stepanov reviewed Apr 19, 2018

View changes

v-stepanov reviewed Apr 19, 2018

View changes

v-stepanov Apr 19, 2018 •

edited

v-stepanov merged commit `f56de8a` into master Apr 25, 2018
6 of 7 checks passed