Skip to content

Commit

Permalink
Reorganizes the asynchronous classes.
Browse files Browse the repository at this point in the history
-- Before everything was shoved into the single jobs package. Now, the
package structure has been made much more organized (and hopefully)
tractable.
  • Loading branch information
archolewa authored and cdeszaq committed Aug 31, 2016
1 parent 84e4be9 commit 0c154a6
Show file tree
Hide file tree
Showing 57 changed files with 206 additions and 158 deletions.
15 changes: 14 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,31 @@ Current
### Removed:


#### Changed:
### Changed:

- [Reorganizes asynchronous package structure](https://github.com/yahoo/fili/pull/19)
* The `jobs` package is renamed to `async` and split into the following subpackages:
- `broadcastchannels` - Everything dealing with broadcast channels
- `jobs` - Everything related to `jobs`, broken into subpackages
* `jobrows` - Everything related to the content of the job metadata
* `payloads` - Everything related to building the version of the job metadata to send to the user
* `stores` - Everything related to the databases for job data
- `preresponses` - Everything related to `PreResponses`, broken into subpackages
* `stores` - Everything related to the the databases for PreResponse data
- `workflows` - Everything related to the asynchronous workflow

- [Removed timing component in JobsApiRequestSpec](https://github.com/yahoo/fili/pull/27)
* Rather than setting an async timeout, and then sleeping,
`JobsApiReqeustSpec::handleBroadcastChannelNotification returns an empty Observable if a timeout occurs before the notification is received`
now verifies that the Observable returned terminates without sending any
messages.

- [Fix Dimension Serialization Problem with Nested Queries](https://github.com/yahoo/fili/pull/15)
* Modified `DimensionToDefaultDimensionSpec` serializer to serialize dimension to apiName if it is not the inner most query
* Added helper `hasInnerQuery` to `Util` in serializer package to determine if current query is the inner most query or not
* Added tests for `DimensionToDefaultDimensionSpec`


### Fixed:

- [`DruidDimensionsLoader` doesn't set the dimension's lastUpdated date](https://github.com/yahoo/fili/pull/24)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@
import com.yahoo.bard.webservice.application.healthchecks.DruidDimensionsLoaderHealthCheck;
import com.yahoo.bard.webservice.application.healthchecks.SegmentMetadataLoaderHealthCheck;
import com.yahoo.bard.webservice.application.healthchecks.VersionHealthCheck;
import com.yahoo.bard.webservice.jobs.AsynchronousWorkflowsBuilder;
import com.yahoo.bard.webservice.jobs.DefaultAsynchronousWorkflowsBuilder;
import com.yahoo.bard.webservice.async.workflows.AsynchronousWorkflowsBuilder;
import com.yahoo.bard.webservice.async.workflows.DefaultAsynchronousWorkflowsBuilder;
import com.yahoo.bard.webservice.config.BardFeatureFlag;
import com.yahoo.bard.webservice.config.FeatureFlag;
import com.yahoo.bard.webservice.config.FeatureFlagRegistry;
Expand Down Expand Up @@ -55,17 +55,17 @@
import com.yahoo.bard.webservice.druid.util.FieldConverterSupplier;
import com.yahoo.bard.webservice.druid.util.FieldConverters;
import com.yahoo.bard.webservice.druid.util.SketchFieldConverter;
import com.yahoo.bard.webservice.jobs.ApiJobStore;
import com.yahoo.bard.webservice.jobs.BroadcastChannel;
import com.yahoo.bard.webservice.jobs.DefaultJobField;
import com.yahoo.bard.webservice.jobs.DefaultJobPayloadBuilder;
import com.yahoo.bard.webservice.jobs.DefaultJobRowBuilder;
import com.yahoo.bard.webservice.jobs.JobPayloadBuilder;
import com.yahoo.bard.webservice.jobs.JobRowBuilder;
import com.yahoo.bard.webservice.jobs.NoOpApiJobStore;
import com.yahoo.bard.webservice.jobs.NoOpPreResponseStore;
import com.yahoo.bard.webservice.jobs.PreResponseStore;
import com.yahoo.bard.webservice.jobs.SimpleBroadcastChannel;
import com.yahoo.bard.webservice.async.jobs.stores.ApiJobStore;
import com.yahoo.bard.webservice.async.broadcastchannels.BroadcastChannel;
import com.yahoo.bard.webservice.async.jobs.jobrows.DefaultJobField;
import com.yahoo.bard.webservice.async.jobs.payloads.DefaultJobPayloadBuilder;
import com.yahoo.bard.webservice.async.jobs.jobrows.DefaultJobRowBuilder;
import com.yahoo.bard.webservice.async.jobs.payloads.JobPayloadBuilder;
import com.yahoo.bard.webservice.async.jobs.jobrows.JobRowBuilder;
import com.yahoo.bard.webservice.async.jobs.stores.NoOpApiJobStore;
import com.yahoo.bard.webservice.async.preresponses.stores.NoOpPreResponseStore;
import com.yahoo.bard.webservice.async.preresponses.stores.PreResponseStore;
import com.yahoo.bard.webservice.async.broadcastchannels.SimpleBroadcastChannel;
import com.yahoo.bard.webservice.metadata.DataSourceMetadataLoader;
import com.yahoo.bard.webservice.metadata.DataSourceMetadataService;
import com.yahoo.bard.webservice.metadata.QuerySigningService;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async;

import com.yahoo.bard.webservice.data.ResultSet;
import com.yahoo.bard.webservice.druid.model.query.AllGranularity;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async;

import javax.validation.constraints.NotNull;
import javax.ws.rs.ProcessingException;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async;

import static com.yahoo.bard.webservice.web.handlers.workflow.DruidWorkflow.RESPONSE_WORKFLOW_TIMER;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async;

import com.yahoo.bard.webservice.druid.model.query.DruidAggregationQuery;
import com.yahoo.bard.webservice.druid.model.query.DruidQuery;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.broadcastchannels;

import rx.Observable;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.broadcastchannels;

import com.yahoo.bard.webservice.config.SystemConfig;
import com.yahoo.bard.webservice.config.SystemConfigProvider;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.broadcastchannels;

import rx.Observable;
import rx.subjects.Subject;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.jobrows;

import com.yahoo.bard.webservice.util.EnumUtils;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.jobrows;

import org.joda.time.DateTime;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.jobrows;

import java.util.Locale;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.jobrows;

/**
* Represents a column in the table of job metadata.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.jobrows;

import com.yahoo.bard.webservice.util.StreamUtils;
import com.yahoo.bard.webservice.web.ErrorMessageFormat;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.jobrows;

import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.core.UriInfo;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.jobrows;

/**
* A job status describes the current state of a job (i.e. 'pending', 'success', 'failure').
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.payloads;

import com.yahoo.bard.webservice.async.jobs.jobrows.DefaultJobField;
import com.yahoo.bard.webservice.async.jobs.jobrows.JobRow;
import com.yahoo.bard.webservice.web.ErrorMessageFormat;
import com.yahoo.bard.webservice.web.JobRequestFailedException;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.payloads;

import com.yahoo.bard.webservice.async.jobs.jobrows.JobRow;
import com.yahoo.bard.webservice.web.endpoints.JobsServlet;

import java.util.Map;
Expand Down Expand Up @@ -44,7 +45,7 @@ static String getResultsUrl(String ticket, UriInfo uriInfo) {
/**
* Get the url for obtaining the job results synchronously.
* <pre>
* "http://localhost:9998/jobs/ticket1/results?async=never"
* "http://localhost:9998/jobs/ticket1/results?asyncAfter=never"
* </pre>
*
* @param ticket The ticket that can uniquely identify a Job
Expand All @@ -60,11 +61,11 @@ static String getSyncResultsUrl(String ticket, UriInfo uriInfo) {
}

/**
* Get the UriBuilder for the /job/ticket/results endpoint.
* Get the UriBuilder for the /jobs/ticket/results endpoint.
*
* @param uriInfo UriInfo of the request
*
* @return the UriBuilder for the /job/ticket/results endpoint
* @return the UriBuilder for the /jobs/ticket/results endpoint
*/
static UriBuilder getResultsBaseUrl(UriInfo uriInfo) {
return uriInfo.getBaseUriBuilder()
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.stores;

import com.yahoo.bard.webservice.async.jobs.jobrows.JobRow;

import rx.Observable;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.stores;

import com.yahoo.bard.webservice.async.jobs.jobrows.JobRow;

/**
* Class for exception thrown while saving a record in ApiJobStore.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.stores;

import com.yahoo.bard.webservice.async.jobs.jobrows.JobRow;

import rx.Observable;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.jobs.stores;

import com.yahoo.bard.webservice.async.jobs.jobrows.JobRow;

import rx.Observable;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.preresponses.stores;

import com.yahoo.bard.webservice.web.PreResponse;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.preresponses.stores;

import com.yahoo.bard.webservice.web.PreResponse;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.preresponses.stores;

import com.yahoo.bard.webservice.web.PreResponse;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.workflows;

import com.yahoo.bard.webservice.async.jobs.jobrows.JobRow;
import com.yahoo.bard.webservice.web.PreResponse;

import rx.Observable;
Expand Down Expand Up @@ -34,7 +35,7 @@ public class AsynchronousWorkflows {
* @param queryResultsPayload Emits query results to send to the user
* @param jobMetadataPayload Emits job metadata to send to the user
* @param preResponseReadyNotifications Emits notifications that the query results have been stored in the
* {@link PreResponseStore}
* {@link com.yahoo.bard.webservice.async.preresponses.stores.PreResponseStore}
* @param jobMarkedCompleteNotifications Emits notifications that the JobRow has been updated with success or error
* upon completion of the query
*/
Expand Down Expand Up @@ -70,10 +71,10 @@ public Observable<String> getAsynchronousPayload() {

/**
* Returns an Observable that emits notifications that the query results have been stored in the
* {@link PreResponseStore}.
* {@link com.yahoo.bard.webservice.async.preresponses.stores.PreResponseStore}.
*
* @return an Observable that emits notifications that the query results have been stored in the
* {@link PreResponseStore}
* {@link com.yahoo.bard.webservice.async.preresponses.stores.PreResponseStore}
*/
public Observable<String> getPreResponseReadyNotifications() {
return preResponseReadyNotifications;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;
package com.yahoo.bard.webservice.async.workflows;

import com.yahoo.bard.webservice.async.jobs.jobrows.JobRow;
import com.yahoo.bard.webservice.util.Either;
import com.yahoo.bard.webservice.web.PreResponse;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,17 @@
// Copyright 2016 Yahoo Inc.
// Licensed under the terms of the Apache license. Please see LICENSE file distributed with this work for terms.
package com.yahoo.bard.webservice.jobs;

import static com.yahoo.bard.webservice.jobs.DefaultJobField.DATE_UPDATED;
import static com.yahoo.bard.webservice.jobs.DefaultJobField.STATUS;
import static com.yahoo.bard.webservice.jobs.DefaultJobStatus.FAILURE;
import static com.yahoo.bard.webservice.jobs.DefaultJobStatus.SUCCESS;

package com.yahoo.bard.webservice.async.workflows;

import static com.yahoo.bard.webservice.async.jobs.jobrows.DefaultJobField.DATE_UPDATED;
import static com.yahoo.bard.webservice.async.jobs.jobrows.DefaultJobField.STATUS;
import static com.yahoo.bard.webservice.async.jobs.jobrows.DefaultJobStatus.FAILURE;
import static com.yahoo.bard.webservice.async.jobs.jobrows.DefaultJobStatus.SUCCESS;

import com.yahoo.bard.webservice.async.AsyncUtils;
import com.yahoo.bard.webservice.async.jobs.stores.ApiJobStore;
import com.yahoo.bard.webservice.async.jobs.jobrows.JobField;
import com.yahoo.bard.webservice.async.jobs.jobrows.JobRow;
import com.yahoo.bard.webservice.async.preresponses.stores.PreResponseStore;
import com.yahoo.bard.webservice.util.Either;
import com.yahoo.bard.webservice.web.ErrorMessageFormat;
import com.yahoo.bard.webservice.web.PreResponse;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

import static com.yahoo.bard.webservice.web.handlers.workflow.DruidWorkflow.RESPONSE_WORKFLOW_TIMER;

import com.yahoo.bard.webservice.jobs.ResponseException;
import com.yahoo.bard.webservice.async.ResponseException;
import com.yahoo.bard.webservice.logging.RequestLog;
import com.yahoo.bard.webservice.web.DataApiRequest;
import com.yahoo.bard.webservice.web.PreResponse;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ public static Boolean hasInnerQuery(JsonGenerator gen) {
*
* @param gen the Json Generator to retrieve the tree to walk on.
* @param mapper a function that takes an DruidQuery as an argument and return the final desired returned result.
* @param <T> Type of result from the mapper
*
* @return an Optional of the desired result T if DruidQuery is found, Optional.empty otherwise
*/
Expand Down
Loading

0 comments on commit 0c154a6

Please sign in to comment.