FALCON-2226 Submit ,Schedule and submitAndSchedule API for extension #327

PraveenAdlakha · 2016-12-22T12:01:12Z

No description provided.

…in distributed mode

pallavi-rao · 2016-12-23T09:45:58Z

common/src/main/resources/startup.properties

@@ -52,6 +52,7 @@

 ##Add if you want to use Trusted or User Extensions
 ## In case of distributed Mode enable ExtensionService only on Prism via prism.application.services
+


Nit : Unnecessary change.

pallavi-rao · 2016-12-23T09:49:59Z

prism/src/main/java/org/apache/falcon/resource/AbstractExtensionManager.java

@@ -58,6 +46,7 @@
    private static final String CONFIG  = "config";
    private static final String CREATION_TIME  = "creationTime";
    private static final String LAST_UPDATE_TIME  = "lastUpdatedTime";
+    public static final String FALCON_TAG = "falcon";


Don't think this is used in this class. May be move it proxy?

pallavi-rao · 2016-12-23T10:01:34Z

prism/src/main/java/org/apache/falcon/resource/proxy/ExtensionManagerProxy.java

@@ -317,8 +330,8 @@ public APIResult submit(

        try {
            entityMap = getEntityList(extensionName, jobName, feedForms, processForms, config);
-            submitEntities(extensionName, doAsUser, jobName, entityMap, config);
-        } catch (FalconException | IOException e) {
+            submitEntities(extensionName, jobName, entityMap, config, request);


Why has doAsUser been knocked off? I know we don't use it now. But, since there is a plan to enable it later, we shouldn't be removing it.

submit API in SchedulableEntityManagerProxy doesnot take doAs as a param we should not do it too.

fair enough.

pallavi-rao · 2016-12-23T10:09:45Z

prism/src/main/java/org/apache/falcon/resource/proxy/ExtensionManagerProxy.java

+
+        for(Map.Entry<EntityType, List<Entity>> entry : entityMap.entrySet()){
+            for(final Entity entity : entry.getValue()){
+                final HttpServletRequest httpServletRequest = getEntityStream(entity, entity.getEntityType(), request);


Since the eventual goal is to get the bufferedRequest I think getEntityStream and getBufferedRequest can be merged into a single method.

pallavi-rao · 2016-12-23T10:12:37Z

prism/src/main/java/org/apache/falcon/resource/proxy/ExtensionManagerProxy.java

-            submitInternal(process, doAsUser);
-            processNames.add(process.getName());
+
+        for(Map.Entry<EntityType, List<Entity>> entry : entityMap.entrySet()){


The feeds have to be submitted before processes are. Else, the process submission will fail due to non-existence of a feed.

pallavi-rao · 2016-12-23T10:14:23Z

prism/src/main/java/org/apache/falcon/resource/proxy/ExtensionManagerProxy.java

@@ -691,4 +778,61 @@ private static void checkIfExtensionServiceIsEnabled() {
                    ExtensionService.SERVICE_NAME + " is not enabled.", Response.Status.NOT_FOUND);
        }
    }
+
+    private abstract class EntitiesProxy<T extends APIResult> {


Why is this class here? Where is it used?

not needed will remove.

pallavi-rao · 2016-12-23T10:17:39Z

unit/src/main/java/org/apache/falcon/unit/LocalExtensionManager.java

+
+        for(Map.Entry<EntityType, List<Entity>> entry : entityMap.entrySet()){
+            for(Entity entity : entry.getValue()){
+                submitInternal(entity, "falconUser");


Feeds before processes, here too. May be create a method and call from both submit and submitAndSchedule.

pallavi-rao · 2016-12-23T10:18:36Z

unit/src/main/java/org/apache/falcon/unit/LocalExtensionManager.java

+        List<String> processNames = new ArrayList<>();
+        for(Map.Entry<EntityType, List<Entity>> entry : entityMap.entrySet()){
+            for(Entity entity : entry.getValue()){
+                submitInternal(entity, "falconUser");


feeds before processes.

pallavi-rao · 2016-12-23T10:20:25Z

unit/src/main/java/org/apache/falcon/unit/LocalExtensionManager.java

 import java.util.List;
 import java.util.Map;

 /**
 * A proxy implementation of the extension operations in local mode.
 */
-public class LocalExtensionManager extends AbstractExtensionManager {
+public class LocalExtensionManager extends ExtensionManagerProxy {


Why extend Proxy? Falcon Unit shouldn't worry about Proxy.

pallavi-rao · 2016-12-26T05:34:20Z

prism/src/main/java/org/apache/falcon/resource/proxy/ExtensionManagerProxy.java

@@ -326,7 +325,7 @@ public APIResult submit(
            @FormDataParam("feeds") List<FormDataBodyPart> feedForms,
            @FormDataParam("config") InputStream config) {
        checkIfExtensionServiceIsEnabled();
-        Map<EntityType, List<Entity>> entityMap;
+        TreeMap<EntityType, List<Entity>> entityMap;


Declaration should only have Map... Only instantiation should be TreeMap.

pallavi-rao · 2016-12-26T05:34:35Z

prism/src/main/java/org/apache/falcon/resource/proxy/ExtensionManagerProxy.java

@@ -338,15 +337,15 @@ public APIResult submit(
        return new APIResult(APIResult.Status.SUCCEEDED, "Extension job submitted successfully:" + jobName);
    }

-    private Map<EntityType, List<Entity>> getEntityList(String extensionName, String jobName,
+    private TreeMap<EntityType, List<Entity>> getEntityList(String extensionName, String jobName,


Declaration should just be Map

I want the contract to be tightly coupled so have changed it to SortedMap.

Fair enough, as long as it is an Interface as you have used (SortedMap) and not the impl., it is fine.

pallavi-rao · 2016-12-26T05:35:44Z

prism/src/main/java/org/apache/falcon/resource/proxy/ExtensionManagerProxy.java

                                                        List<FormDataBodyPart> feedForms,
                                                        List<FormDataBodyPart> processForms, InputStream config)
        throws FalconException, IOException{
        List<Entity> processes = getProcesses(processForms);
        List<Entity> feeds = getFeeds(feedForms);
        ExtensionType extensionType = getExtensionType(extensionName);
        List<Entity> entities;
-        Map<EntityType, List<Entity>> entityMap = new HashMap<>();
+        TreeMap<EntityType, List<Entity>> entityMap = new TreeMap<>();


Believe you made this a tree map, so that during submission, feeds go first? But, while putting elements into this map, processes are going in first.

Internally tree will store in the natural ordering and Enum maintains it in the ordinal value in which feed it the first one so we should be good here.

Oh.. right.. My bad. In my head, I mixed it up with LinkedHashMap.

pallavi-rao · 2016-12-26T05:36:45Z

prism/src/main/java/org/apache/falcon/resource/proxy/ExtensionManagerProxy.java

@@ -442,7 +441,7 @@ private BufferedRequest getBufferedRequest(HttpServletRequest request) {
    }

    protected void submitEntities(String extensionName, String jobName,
-                                  Map<EntityType, List<Entity>> entityMap, InputStream configStream,
+                                  TreeMap<EntityType, List<Entity>> entityMap, InputStream configStream,


Method signature should just have the interface name, Map.

changed it to SortedMap.

pallavi-rao · 2016-12-26T05:37:53Z

unit/src/main/java/org/apache/falcon/unit/FalconUnitClient.java

@@ -285,14 +285,14 @@ public APIResult submitExtensionJob(String extensionName, String jobName, String

        InputStream configStream = getServletInputStream(configPath);
        try {
-            Map<EntityType, List<Entity>> entityMap = getEntityTypeListMap(extensionName, jobName, configStream);
+            TreeMap<EntityType, List<Entity>> entityMap = getEntityTypeListMap(extensionName, jobName, configStream);


Same comment as above for the declarations in this class too. Method signatures and declaration should just have Map.

pallavi-rao

Please change the TreeMap to SortedMap both in FalconUnitClient and LocalExtensionManager

pallavi-rao

👍

pallavi-rao · 2016-12-26T11:26:30Z

Unrelated build failure.. merging.

@pallavi-rao

Author: Praveen Adlakha <adlakha.praveen@gmail.com> Reviewers: @pallavi-rao Closes apache#327 from PraveenAdlakha/2226 and squashes the following commits: 59a43ef [Praveen Adlakha] minor changes in falcon client and localextensionmanager 473f04a [Praveen Adlakha] comments addressed 6a346aa [Praveen Adlakha] comments addressed 8733f53 [Praveen Adlakha] checkstyle issue resolved 9ba005e [Praveen Adlakha] FALCON-2226 Submit ,Schedule and submitAndSchedule API for extension in distributed mode 699b06f [Praveen Adlakha] WIP 29da911 [Praveen Adlakha] WIP 0b7d02a [Praveen Adlakha] merge conflicts resolved 6b77cc1 [Praveen Adlakha] FALCON-2223 Distributed mode support for User Extension

PraveenAdlakha added 5 commits December 22, 2016 17:20

FALCON-2223 Distributed mode support for User Extension

6b77cc1

merge conflicts resolved

0b7d02a

WIP

29da911

WIP

699b06f

FALCON-2226 Submit ,Schedule and submitAndSchedule API for extension …

9ba005e

…in distributed mode

PraveenAdlakha changed the title ~~2226~~ FALCON-2226 Submit ,Schedule and submitAndSchedule API for extension Dec 22, 2016

checkstyle issue resolved

8733f53

PraveenAdlakha force-pushed the 2226 branch from dbb4cc2 to 8733f53 Compare December 22, 2016 15:48

pallavi-rao suggested changes Dec 23, 2016

View reviewed changes

comments addressed

6a346aa

pallavi-rao suggested changes Dec 26, 2016

View reviewed changes

comments addressed

473f04a

pallavi-rao suggested changes Dec 26, 2016

View reviewed changes

minor changes in falcon client and localextensionmanager

59a43ef

pallavi-rao approved these changes Dec 26, 2016

View reviewed changes

asfgit closed this in 9946758 Dec 26, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FALCON-2226 Submit ,Schedule and submitAndSchedule API for extension #327

FALCON-2226 Submit ,Schedule and submitAndSchedule API for extension #327

PraveenAdlakha commented Dec 22, 2016

pallavi-rao Dec 23, 2016

pallavi-rao Dec 23, 2016

pallavi-rao Dec 23, 2016 •

edited

Loading

PraveenAdlakha Dec 23, 2016

pallavi-rao Dec 26, 2016

pallavi-rao Dec 23, 2016 •

edited

Loading

pallavi-rao Dec 23, 2016

pallavi-rao Dec 23, 2016

PraveenAdlakha Dec 23, 2016

pallavi-rao Dec 23, 2016

pallavi-rao Dec 23, 2016

pallavi-rao Dec 23, 2016

pallavi-rao Dec 26, 2016

pallavi-rao Dec 26, 2016

PraveenAdlakha Dec 26, 2016

pallavi-rao Dec 26, 2016

pallavi-rao Dec 26, 2016

PraveenAdlakha Dec 26, 2016

pallavi-rao Dec 26, 2016

pallavi-rao Dec 26, 2016

PraveenAdlakha Dec 26, 2016

pallavi-rao Dec 26, 2016

pallavi-rao left a comment

pallavi-rao left a comment

pallavi-rao commented Dec 26, 2016

		@@ -52,6 +52,7 @@

		##Add if you want to use Trusted or User Extensions
		## In case of distributed Mode enable ExtensionService only on Prism via prism.application.services

FALCON-2226 Submit ,Schedule and submitAndSchedule API for extension #327

FALCON-2226 Submit ,Schedule and submitAndSchedule API for extension #327

Conversation

PraveenAdlakha commented Dec 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pallavi-rao Dec 23, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pallavi-rao Dec 23, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pallavi-rao left a comment

Choose a reason for hiding this comment

pallavi-rao left a comment

Choose a reason for hiding this comment

pallavi-rao commented Dec 26, 2016

pallavi-rao Dec 23, 2016 •

edited

Loading

pallavi-rao Dec 23, 2016 •

edited

Loading