FALCON-2210 Server side changes in submit and submitAndSchedule apis to accept list of feeds and processes #313

sandeepSamudrala · 2016-12-06T10:54:22Z

No description provided.

…me with Delay.

…-2201

… tests for extensions and fixes.

…-2201

…client. I will move that to Server side in the server side changes

…-2210

…to accept list of feeds and processes

pallavi-rao · 2016-12-07T08:20:01Z

common/src/main/java/org/apache/falcon/entity/parser/ProcessEntityParser.java

+        validate(process, true);
+    }
+
+    public void validate(Process process, boolean checkDependantFeeds) throws FalconException {


Nit: Typo checkDependentFeeds (not checkDependantFeeds)

Ack. Fixed it

pallavi-rao · 2016-12-07T08:23:59Z

prism/src/main/java/org/apache/falcon/resource/extensions/ExtensionManager.java

            @FormDataParam("config") InputStream config) {
        checkIfExtensionServiceIsEnabled();
+        List<Process> processes = getProcesses(processForms);
+        List<Feed> feeds = getFeeds(feedForms);
+        Pair<List<Feed>, List<Process>> entityPairs;


Map<EntityType, List> is more extendable. Can be passed along directly as arg to submitEntities.

Made the respective changes.

pallavi-rao · 2016-12-07T08:30:13Z

prism/src/main/java/org/apache/falcon/resource/extensions/ExtensionManager.java

-                feeds.add(entity.getName());
-            } else if (EntityType.PROCESS.equals(entity.getEntityType())) {
-                processes.add(entity.getName());
+    private List<Feed> getFeeds(List<FormDataBodyPart> feedForms) {


Can have single method that takes in Class as an argument.

Thats not possible as the entity class cannot be deserialized.

pallavi-rao · 2016-12-07T08:35:42Z

prism/src/main/java/org/apache/falcon/resource/extensions/ExtensionManager.java

            @FormDataParam("config") InputStream config) {
        checkIfExtensionServiceIsEnabled();
+        List<Process> processes = getProcesses(processForms);


Can move getProcesses and getFeeds inside getEntityList, cleaner and less repetition between submit and submitAndSchedule.

Ack. Moved it

pallavi-rao · 2016-12-07T08:37:08Z

prism/src/main/java/org/apache/falcon/resource/extensions/ExtensionManager.java

+    }
+
+
+    private void validateFeeds(List<Feed> feeds) throws FalconException {


validateFeeds and validateEntities can be clubbed into single method, with type as an argument.

Merging causes a lot of issues and also the functionality being different there is not much gain on merging them. Won't be merging them.

pallavi-rao · 2016-12-07T08:39:49Z

unit/src/main/java/org/apache/falcon/unit/FalconUnitClient.java

@@ -285,7 +285,16 @@ public APIResult submitExtensionJob(String extensionName, String jobName, String
        InputStream configStream = getServletInputStream(configPath);
        try {
            List<Entity> entities = getEntities(extensionName, jobName, configStream);


If getEntities can return a Map<EntityType, List>. It can be directly passed to submitExtensionJob. Less redundant code.

Made the changes.

pallavi-rao · 2016-12-07T11:13:18Z

prism/src/main/java/org/apache/falcon/resource/extensions/ExtensionManager.java

-        List<Process> processes = getProcesses(processForms);
-        List<Feed> feeds = getFeeds(feedForms);
-        Pair<List<Feed>, List<Process>> entityPairs;
+        Map<EntityType, List> entityMap;


Have type for List List

pallavi-rao

+1

@pallavi-rao

…to accept list of feeds and processes Author: sandeep <sandysmdl@gmail.com> Reviewers: @pallavi-rao Closes apache#313 from sandeepSamudrala/FALCON-2210 and squashes the following commits: 3b7a3ae [sandeep] FALCON-2210 applied tags to the entities for user extensions f08cf15 [sandeep] FALCON-2210 Fixed falcon unit client build issues 98d382e [sandeep] FALCON-2210 Incorporated review comments 4004ae7 [sandeep] FALCON-2210 Incorporated review comments d5e4c53 [sandeep] FALCON-2210 Server side changes in submit and submitAndSchedule apis to accept list of feeds and processes 05b87fb [sandeep] Merge branch 'master' of https://github.com/apache/falcon into FALCON-2210 2bd685f [sandeep] FALCON-2201 Fixed checkstyle issues c7422e6 [sandeep] FALCON-2201 Incorporated review comments. Removed applying tags from client. I will move that to Server side in the server side changes 86446ad [sandeep] Merge branch 'master' of https://github.com/apache/falcon into FALCON-2201 432cdfd [sandeep] FALCON-2201 Incorporated review comments a0ce5e0 [sandeep] Merge branch 'master' of https://github.com/apache/falcon into FALCON-2201 519a877 [sandeep] FALCON-2201 Fixed checkstyle issues c101c7b [sandeep] FALCON-2201 Incorporated review comments bf0e6ed [sandeep] FALCON-2201 Incorporated review comments and few client side changes adfd318 [sandeep] FALCON-2201 Falcon Unit changes for extension support and falcon unit tests for extensions and fixes. 03f0c3c [sandeep] Merge branch 'master' of https://github.com/apache/falcon into FALCON-2201 9cf36e9 [sandeep] Merge branch 'master' of https://github.com/apache/falcon bbca081 [sandeep] Merge branch 'master' of https://github.com/apache/falcon 48f6afa [sandeep] Merge branch 'master' of https://github.com/apache/falcon 250cc46 [sandeep] Merge branch 'master' of https://github.com/apache/falcon d0393e9 [sandeep] Merge branch 'master' of https://github.com/apache/falcon a178805 [sandeep] Merge branch 'master' of https://github.com/apache/falcon d6dc8bf [sandeep] Merge branch 'master' of https://github.com/apache/falcon 1bb8d3c [sandeep] Merge branch 'master' of https://github.com/apache/falcon c065566 [sandeep] reverting last line changes made 1a4dcd2 [sandeep] rebased and resolved the conflicts from master 271318b [sandeep] FALCON-2097. Adding UT to the new method for getting next instance time with Delay. a94d4fe [sandeep] rebasing from master 9e68a57 [sandeep] FALCON-298. Feed update with replication delay creates holes

sandeepSamudrala added 25 commits July 25, 2016 12:06

FALCON-298. Feed update with replication delay creates holes

9e68a57

rebasing from master

a94d4fe

FALCON-2097. Adding UT to the new method for getting next instance ti…

271318b

…me with Delay.

rebased and resolved the conflicts from master

1a4dcd2

reverting last line changes made

c065566

Merge branch 'master' of https://github.com/apache/falcon

1bb8d3c

Merge branch 'master' of https://github.com/apache/falcon

d6dc8bf

Merge branch 'master' of https://github.com/apache/falcon

a178805

Merge branch 'master' of https://github.com/apache/falcon

d0393e9

Merge branch 'master' of https://github.com/apache/falcon

250cc46

Merge branch 'master' of https://github.com/apache/falcon

48f6afa

Merge branch 'master' of https://github.com/apache/falcon

bbca081

Merge branch 'master' of https://github.com/apache/falcon

9cf36e9

Merge branch 'master' of https://github.com/apache/falcon into FALCON…

03f0c3c

…-2201

FALCON-2201 Falcon Unit changes for extension support and falcon unit…

adfd318

… tests for extensions and fixes.

FALCON-2201 Incorporated review comments and few client side changes

bf0e6ed

FALCON-2201 Incorporated review comments

c101c7b

FALCON-2201 Fixed checkstyle issues

519a877

Merge branch 'master' of https://github.com/apache/falcon into FALCON…

a0ce5e0

…-2201

FALCON-2201 Incorporated review comments

432cdfd

Merge branch 'master' of https://github.com/apache/falcon into FALCON…

86446ad

…-2201

FALCON-2201 Incorporated review comments. Removed applying tags from …

c7422e6

…client. I will move that to Server side in the server side changes

FALCON-2201 Fixed checkstyle issues

2bd685f

Merge branch 'master' of https://github.com/apache/falcon into FALCON…

05b87fb

…-2210

FALCON-2210 Server side changes in submit and submitAndSchedule apis …

d5e4c53

…to accept list of feeds and processes

pallavi-rao suggested changes Dec 7, 2016

View reviewed changes

FALCON-2210 Incorporated review comments

4004ae7

pallavi-rao suggested changes Dec 7, 2016

View reviewed changes

sandeepSamudrala added 2 commits December 7, 2016 17:04

FALCON-2210 Incorporated review comments

98d382e

FALCON-2210 Fixed falcon unit client build issues

f08cf15

FALCON-2210 applied tags to the entities for user extensions

3b7a3ae

pallavi-rao approved these changes Dec 12, 2016

View reviewed changes

asfgit closed this in 3e7e08f Dec 12, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FALCON-2210 Server side changes in submit and submitAndSchedule apis to accept list of feeds and processes #313

FALCON-2210 Server side changes in submit and submitAndSchedule apis to accept list of feeds and processes #313

sandeepSamudrala commented Dec 6, 2016

pallavi-rao Dec 7, 2016

sandeepSamudrala Dec 7, 2016

pallavi-rao Dec 7, 2016

sandeepSamudrala Dec 7, 2016

pallavi-rao Dec 7, 2016

sandeepSamudrala Dec 7, 2016

pallavi-rao Dec 7, 2016

sandeepSamudrala Dec 7, 2016

pallavi-rao Dec 7, 2016

sandeepSamudrala Dec 7, 2016

pallavi-rao Dec 7, 2016

sandeepSamudrala Dec 7, 2016

pallavi-rao Dec 7, 2016

pallavi-rao left a comment

		}


		private void validateFeeds(List<Feed> feeds) throws FalconException {

FALCON-2210 Server side changes in submit and submitAndSchedule apis to accept list of feeds and processes #313

FALCON-2210 Server side changes in submit and submitAndSchedule apis to accept list of feeds and processes #313

Conversation

sandeepSamudrala commented Dec 6, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pallavi-rao left a comment

Choose a reason for hiding this comment