Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TE] clean up decprecated/unused code #5435

Merged
merged 6 commits into from Jun 4, 2020
Merged

[TE] clean up decprecated/unused code #5435

merged 6 commits into from Jun 4, 2020

Conversation

vincentchenjl
Copy link
Contributor

  • Clean up deprecated code

  • Refactor a bit to remove class dependency

harleyjj
harleyjj approved these changes May 23, 2020
View reviewed changes
Copy link
Contributor

@harleyjj harleyjj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-8215, that's some serious cleaning.

xiaohui-sun
xiaohui-sun reviewed May 24, 2020
View reviewed changes
Copy link

@xiaohui-sun xiaohui-sun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great progress. @kishoreg FYI

...irdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/ThirdEyeAnomalyApplication.java Outdated Show resolved Hide resolved
...irdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/ThirdEyeAnomalyApplication.java Outdated Show resolved Hide resolved
...irdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/ThirdEyeAnomalyApplication.java Outdated Show resolved Hide resolved
...eye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/task/TaskInfoFactory.java Show resolved Hide resolved
...e/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/task/TaskRunnerFactory.java Show resolved Hide resolved
...inot/src/main/java/org/apache/pinot/thirdeye/completeness/checker/DataCompletenessUtils.java
@@ -261,13 +261,4 @@ public static DateTimeFormatter getDateTimeFormatterForDataset(TimeSpec timeSpec
return bucketNameToCountStar;
}

public static double getPercentCompleteness(PercentCompletenessFunctionInput input) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole class could be deleted.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is used in Wo4WAvgDataCompletenessAlgorithm. We need to drop both of them if we want to drop this. Wo4WAvgDataCompletenessAlgorithm cannot be dropped due to the issue below.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove it and also remove getter/setter.
That should be safe.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remove Wo4WAvgDataCompletenessAlgorithm completely and also the field DatasetConfigBean.dataCompletenessAlgorithm from DatasetConfigBean.

...in/java/org/apache/pinot/thirdeye/completeness/checker/Wo4WAvgDataCompletenessAlgorithm.java Outdated Show resolved Hide resolved
jihaozh
jihaozh approved these changes May 26, 2020
View reviewed changes
Copy link
Member

@jihaozh jihaozh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work. Thank you for keeping the code base healthy!

akshayrai
akshayrai approved these changes May 27, 2020
View reviewed changes
Copy link
Contributor

@akshayrai akshayrai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice diligent effort!
Make sure to remove any traces in db and config if any.

xiaohui-sun
xiaohui-sun reviewed May 27, 2020
View reviewed changes
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/task/TaskConstants.java Outdated Show resolved Hide resolved
...eye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/task/TaskInfoFactory.java Outdated Show resolved Hide resolved
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/task/TaskConstants.java Outdated Show resolved Hide resolved
...e/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/common/BaseThirdEyeApplication.java Outdated Show resolved Hide resolved
...ye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/ThirdEyeDashboardApplication.java
@@ -160,14 +155,12 @@ public void run(ThirdEyeDashboardConfiguration config, Environment env)

AnomalyFunctionFactory anomalyFunctionFactory = new AnomalyFunctionFactory(config.getFunctionConfigPath());
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AnomalyFunctionFactory is not used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

anomalyFunctionFactory is used to initialize AnomalyResource, DataResource, and AnomaliesResource. Could we confirm all these three resources are not used?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jihaozh can you check which resource is used? I believe most endpoints in AnomalyResource and AnomaliesResource are not used. From Harley's doc only "/anomalies/search/time/" is used, but let's double check with them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think AnomalyResource is not used. In AnomaliesResource, only the search endpoints listed by Harley is used. DataResource is being used, but it shouldn't need anomalyFunctionFactory anymore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we search for anomalies by using /anomalies/search/time/, we construct AnomalyDetails, and it requires BaseAnomalyFunction to do it.

...rdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/AnomalyResource.java
private DatasetConfigManager datasetConfigDAO;
private AnomalyFunctionFactory anomalyFunctionFactory;
private AlertFilterFactory alertFilterFactory;
private LoadingCache<String, Long> collectionMaxDataTimeCache;

private static final DAORegistry DAO_REGISTRY = DAORegistry.getInstance();

public AnomalyResource(AnomalyFunctionFactory anomalyFunctionFactory, AlertFilterFactory alertFilterFactory,
AlertFilterAutotuneFactory alertFilterAutotuneFactory) {
public AnomalyResource(AnomalyFunctionFactory anomalyFunctionFactory, AlertFilterFactory alertFilterFactory) {
this.anomalyFunctionDAO = DAO_REGISTRY.getAnomalyFunctionDAO();
this.anomalyMergedResultDAO = DAO_REGISTRY.getMergedAnomalyResultDAO();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the difference between anomalyMergedResultDAO and mergedAnomalyResultDAO. Seems duplicate.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both these two are used in the class. If we r sure AnomalyResource is not used, we can remove the whole class.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The question is duplication. We should remove one of them.

...rdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/AnomalyResource.java Outdated Show resolved Hide resolved
...rdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/AnomalyResource.java
@@ -101,21 +99,18 @@
private MergedAnomalyResultManager anomalyMergedResultDAO;
private AlertConfigManager emailConfigurationDAO;
private MergedAnomalyResultManager mergedAnomalyResultDAO;
private AutotuneConfigManager autotuneConfigDAO;
private DatasetConfigManager datasetConfigDAO;
private AnomalyFunctionFactory anomalyFunctionFactory;
private AlertFilterFactory alertFilterFactory;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is used in line 141.
anomalyResults = AlertFilterHelper.applyFiltrationRule(anomalyResults, alertFilterFactory);

...thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/pojo/DatasetConfigBean.java Outdated Show resolved Hide resolved
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/datalayer/DaoTestUtils.java
autotuneConfigDTO.setPerformance(performance);
return autotuneConfigDTO;
}

public static ClassificationConfigDTO getTestClassificationConfig(String name, List<Long> mainFunctionIdList,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be removed in next PR with all the unused DAO objects.

@xiaohui-sun
Copy link

@cecilynie @cyenjung FYI.

xiaohui-sun
xiaohui-sun approved these changes Jun 4, 2020
View reviewed changes
Copy link

@xiaohui-sun xiaohui-sun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for cleaning it up.

@xiaohui-sun xiaohui-sun merged commit 7ac4cd2 into apache:master Jun 4, 2020
@vincentchenjl vincentchenjl deleted the code_cleanup branch June 26, 2020 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jihaozh jihaozh jihaozh approved these changes

@akshayrai akshayrai akshayrai approved these changes

@harleyjj harleyjj harleyjj approved these changes

@xiaohui-sun xiaohui-sun xiaohui-sun approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@vincentchenjl @xiaohui-sun @akshayrai @harleyjj @jihaozh