FALCON-2200 Update API support for extension job (user extension) #331

sandeepSamudrala · 2016-12-28T10:38:36Z

No description provided.

…me with Delay.

…ters being added into entity definition

pallavi-rao · 2016-12-29T06:17:12Z

client/src/main/java/org/apache/falcon/client/FalconClient.java

+        if (StringUtils.isNotBlank(extensionName)) {
+            extensionDetailJson = getExtensionDetailJson(extensionName);
+        } else {
+            extensionDetailJson = getExtensionJobDetailJson(jobName);


Would prefer to keep this cleaner by making a call to getExtensionJobDetails and then another to getExtensionDetail.

Will make the change to make 2 api calls.

pallavi-rao · 2016-12-29T08:35:16Z

prism/src/main/java/org/apache/falcon/resource/AbstractExtensionManager.java

@@ -121,6 +122,8 @@ private JSONObject buildExtensionJobDetailResult(final String jobName) throws Fa
            detailsObject.put(CONFIG, jobsBean.getConfig());
            detailsObject.put(CREATION_TIME, jobsBean.getCreationTime());
            detailsObject.put(LAST_UPDATE_TIME, jobsBean.getLastUpdatedTime());
+            detailsObject.put(EXTENSION_LOCATION, extensionBean.getLocation());


This is not part of the details of the Job. Lets not add this here. Instead lets have update make 2 calls.

Will make the change to make 2 api calls.

pallavi-rao · 2016-12-29T09:19:39Z

prism/src/main/java/org/apache/falcon/resource/proxy/ExtensionManagerProxy.java

+
+        for (Map.Entry<EntityType, List<Entity>> entry : entityMap.entrySet()) {
+            for (final Entity entity : entry.getValue()) {
+                final String entityType = entity.getEntityType().toString();


This repeats code from SchedulableEntityManagerProxy. Any changes/fixes will need to be made to 2 places. Can you consider introducing a Util class and using it in both SchedulableEntityManagerProxy and ExtensionManagerProxy? The problem existed when submit and submitAndSchedule were introduced. As we introduce more APIs, have repetition will get worse.

made the chcanges accordingly

…ies to proxyutil and making 2 api calls to get location in case of update extension

sandeepSamudrala · 2016-12-29T11:24:50Z

prism/src/main/java/org/apache/falcon/resource/proxy/ExtensionManagerProxy.java

+
+        for (Map.Entry<EntityType, List<Entity>> entry : entityMap.entrySet()) {
+            for (final Entity entity : entry.getValue()) {
+                final String entityType = entity.getEntityType().toString();


Makes sense. I will move the submit and update Apis to a new util.

sandeepSamudrala · 2016-12-29T11:25:17Z

client/src/main/java/org/apache/falcon/client/FalconClient.java

+        if (StringUtils.isNotBlank(extensionName)) {
+            extensionDetailJson = getExtensionDetailJson(extensionName);
+        } else {
+            extensionDetailJson = getExtensionJobDetailJson(jobName);


Will make the change to make 2 api calls.

sandeepSamudrala · 2016-12-29T11:25:25Z

prism/src/main/java/org/apache/falcon/resource/AbstractExtensionManager.java

@@ -121,6 +122,8 @@ private JSONObject buildExtensionJobDetailResult(final String jobName) throws Fa
            detailsObject.put(CONFIG, jobsBean.getConfig());
            detailsObject.put(CREATION_TIME, jobsBean.getCreationTime());
            detailsObject.put(LAST_UPDATE_TIME, jobsBean.getLastUpdatedTime());
+            detailsObject.put(EXTENSION_LOCATION, extensionBean.getLocation());


Will make the change to make 2 api calls.

pallavi-rao

Have these changes been dev-tested in both embedded and distributed mode?

pallavi-rao · 2016-12-30T09:25:15Z

prism/src/main/java/org/apache/falcon/resource/proxy/ExtensionManagerProxy.java

@@ -96,8 +96,9 @@

    private boolean embeddedMode = DeploymentUtil.isEmbeddedMode();
    private String currentColo = DeploymentUtil.getCurrentColo();
-    private final Map<String, Channel> configSyncChannels = new HashMap<String, Channel>();
-    private final Map<String, Channel> entityManagerChannels = new HashMap<String, Channel>();
+    private final Map<String, Channel> configSyncChannels = new HashMap<>();


Why do we still need these 2 variables if we are using the the EntityProxyUtil?

These are not required. Missed to removed them.

sandeepSamudrala · 2016-12-30T10:20:54Z

I have tested them in distributed mode. For embedded mode the unit tests run from FalconCLIIT so that should be sufficient

pallavi-rao

👍

…l from ExtensionManager Proxy as they are now used from proxyUtil

@pallavi-rao

This pull request is dependent on #331 Author: sandeep <sandysmdl@gmail.com> Reviewers: @pallavi-rao Closes #333 from sandeepSamudrala/FALCON-2199

@pallavi-rao

Author: sandeep <sandysmdl@gmail.com> Reviewers: @pallavi-rao Closes apache#331 from sandeepSamudrala/FALCON-2200 and squashes the following commits: 737fad3 [sandeep] FALCON-2200 fixed checkstyle issues. removed unused imports 1780416 [sandeep] Incorporated review comments. Removed entitychannel and config channel from ExtensionManager Proxy as they are now used from proxyUtil 8a4d035 [sandeep] FALCON-2200 Incorporated review comments. Moved common code from proxies to proxyutil and making 2 api calls to get location in case of update extension c8d0ab7 [sandeep] FALCON-2200 Adding changes related to clusters being removed and clusters being added into entity definition cc7c9e9 [sandeep] FALCON-2200 Update API support for extension job (user extension) 456d4ee [sandeep] Merge branch 'master' of https://github.com/apache/falcon 0cf9af6 [sandeep] Merge branch 'master' of https://github.com/apache/falcon 4a2e23e [sandeep] Merge branch 'master' of https://github.com/apache/falcon b1546ed [sandeep] Merge branch 'master' of https://github.com/apache/falcon 0a433fb [sandeep] Merge branch 'master' of https://github.com/apache/falcon 194f36a [sandeep] Merge branch 'master' of https://github.com/apache/falcon e0ad358 [sandeep] Merge branch 'master' of https://github.com/apache/falcon f96a084 [sandeep] Merge branch 'master' of https://github.com/apache/falcon 9cf36e9 [sandeep] Merge branch 'master' of https://github.com/apache/falcon bbca081 [sandeep] Merge branch 'master' of https://github.com/apache/falcon 48f6afa [sandeep] Merge branch 'master' of https://github.com/apache/falcon 250cc46 [sandeep] Merge branch 'master' of https://github.com/apache/falcon d0393e9 [sandeep] Merge branch 'master' of https://github.com/apache/falcon a178805 [sandeep] Merge branch 'master' of https://github.com/apache/falcon d6dc8bf [sandeep] Merge branch 'master' of https://github.com/apache/falcon 1bb8d3c [sandeep] Merge branch 'master' of https://github.com/apache/falcon c065566 [sandeep] reverting last line changes made 1a4dcd2 [sandeep] rebased and resolved the conflicts from master 271318b [sandeep] FALCON-2097. Adding UT to the new method for getting next instance time with Delay. a94d4fe [sandeep] rebasing from master 9e68a57 [sandeep] FALCON-298. Feed update with replication delay creates holes

@pallavi-rao

This pull request is dependent on apache#331 Author: sandeep <sandysmdl@gmail.com> Reviewers: @pallavi-rao Closes apache#333 from sandeepSamudrala/FALCON-2199

sandeepSamudrala added 22 commits July 25, 2016 12:06

FALCON-298. Feed update with replication delay creates holes

9e68a57

rebasing from master

a94d4fe

FALCON-2097. Adding UT to the new method for getting next instance ti…

271318b

…me with Delay.

rebased and resolved the conflicts from master

1a4dcd2

reverting last line changes made

c065566

Merge branch 'master' of https://github.com/apache/falcon

1bb8d3c

Merge branch 'master' of https://github.com/apache/falcon

d6dc8bf

Merge branch 'master' of https://github.com/apache/falcon

a178805

Merge branch 'master' of https://github.com/apache/falcon

d0393e9

Merge branch 'master' of https://github.com/apache/falcon

250cc46

Merge branch 'master' of https://github.com/apache/falcon

48f6afa

Merge branch 'master' of https://github.com/apache/falcon

bbca081

Merge branch 'master' of https://github.com/apache/falcon

9cf36e9

Merge branch 'master' of https://github.com/apache/falcon

f96a084

Merge branch 'master' of https://github.com/apache/falcon

e0ad358

Merge branch 'master' of https://github.com/apache/falcon

194f36a

Merge branch 'master' of https://github.com/apache/falcon

0a433fb

Merge branch 'master' of https://github.com/apache/falcon

b1546ed

Merge branch 'master' of https://github.com/apache/falcon

4a2e23e

Merge branch 'master' of https://github.com/apache/falcon

0cf9af6

Merge branch 'master' of https://github.com/apache/falcon

456d4ee

FALCON-2200 Update API support for extension job (user extension)

cc7c9e9

sandeepSamudrala mentioned this pull request Dec 28, 2016

FALCON-2199 Delete API support for extension job (user extension) #333

Closed

FALCON-2200 Adding changes related to clusters being removed and clus…

c8d0ab7

…ters being added into entity definition

pallavi-rao suggested changes Dec 29, 2016

View reviewed changes

FALCON-2200 Incorporated review comments. Moved common code from prox…

8a4d035

…ies to proxyutil and making 2 api calls to get location in case of update extension

sandeepSamudrala commented Dec 30, 2016

View reviewed changes

pallavi-rao suggested changes Dec 30, 2016

View reviewed changes

pallavi-rao approved these changes Dec 30, 2016

View reviewed changes

sandeepSamudrala added 2 commits December 30, 2016 15:54

Incorporated review comments. Removed entitychannel and config channe…

1780416

…l from ExtensionManager Proxy as they are now used from proxyUtil

FALCON-2200 fixed checkstyle issues. removed unused imports

737fad3

asfgit closed this in 4f42dc1 Dec 30, 2016

asfgit pushed a commit that referenced this pull request Jan 2, 2017

FALCON-2199 Delete API support for extension job (user extension)

42d3794

This pull request is dependent on #331 Author: sandeep <sandysmdl@gmail.com> Reviewers: @pallavi-rao Closes #333 from sandeepSamudrala/FALCON-2199

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FALCON-2200 Update API support for extension job (user extension) #331

FALCON-2200 Update API support for extension job (user extension) #331

sandeepSamudrala commented Dec 28, 2016

pallavi-rao Dec 29, 2016

sandeepSamudrala Dec 29, 2016

pallavi-rao Dec 29, 2016

sandeepSamudrala Dec 29, 2016

pallavi-rao Dec 29, 2016

sandeepSamudrala Dec 30, 2016

sandeepSamudrala Dec 29, 2016

sandeepSamudrala Dec 29, 2016

sandeepSamudrala Dec 29, 2016

pallavi-rao left a comment

pallavi-rao Dec 30, 2016

sandeepSamudrala Dec 30, 2016

sandeepSamudrala commented Dec 30, 2016

pallavi-rao left a comment

FALCON-2200 Update API support for extension job (user extension) #331

FALCON-2200 Update API support for extension job (user extension) #331

Conversation

sandeepSamudrala commented Dec 28, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pallavi-rao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandeepSamudrala commented Dec 30, 2016

pallavi-rao left a comment

Choose a reason for hiding this comment