FALCON-1811 Status API does not honour start option #40

PraveenAdlakha · 2016-02-11T11:47:12Z

$ bin/falcon instance -type process -name process1 -start 2015-10-09T14:00Z -status -numResults 2
Consolidated Status: SUCCEEDED

Instances:

Instance Cluster SourceCluster Status Start End Details Log

2015-12-31T01:00Z local - SUSPENDED 2016-02-11T09:34Z - - http://blr-test-129.corp.inmobi.com:11000/oozie?job=0000028-151229192519233-oozie-oozi-W
actions:
mr-node null null
failed-post-processing null null
2016-01-01T01:00Z local - SUSPENDED 2016-02-11T10:18Z - - http://blr-test-129.corp.inmobi.com:11000/oozie?job=0000030-151229192519233-oozie-oozi-W
actions:
mr-node null null
failed-post-processing null null

Additional Information:
Response: default/STATUS
Request Id: default/124451500@qtp-822600088-12 - 2ca2d36d-6e03-4e14-9e97-1448aed90084

Note we will not get the same result when status is waiting as at that time start time is null.

deepakbarr · 2016-02-11T12:05:10Z

prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java

            });
-        }//Default : no sort
-
+        }


I think "else" is missing here. Look like last block of sorting should be under "else".

Also since the code is redundant. You can simple change the condition on line 468 to if (orderBy.equals("starttime") || orderBy.isEmpty())

looks valid making the change now..

deepakbarr · 2016-02-11T17:43:17Z

+1. LGTM.

pallavi-rao · 2016-02-12T09:30:23Z

prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java

                }
            });
-        } else if (orderBy.equals("starttime")){
+        } else if (orderBy.equals("starttime") || orderBy.isEmpty()){


This means we are ordering the results by startTime, by default. But, the current behavior is to order by instanceTime in descending order and this change breaks that.

PraveenAdlakha · 2016-02-15T16:44:28Z

Testing output :

$ bin/falcon instance -type process -name process1 -start 2016-01-10T14:00Z -status
Consolidated Status: SUCCEEDED

Instances:

Instance Cluster SourceCluster Status Start End Details Log

2016-01-11T01:00Z local - SUSPENDED 2016-02-11T10:24Z - - http://falcon-server:11000/oozie?job=0000034-151229192519233-oozie-oozi-W
actions:
mr-node null null
failed-post-processing null null
2016-01-12T01:00Z local - KILLED 2016-02-11T10:23Z 2016-02-11T10:23Z - http://falcon-server:11000/oozie?job=0000032-151229192519233-oozie-oozi-W
actions:
mr-node null null
failed-post-processing FAILED/KILLED http://falcon-server:8088/proxy/application_1451227881601_0015/
2016-01-13T01:00Z local - WAITING - - hdfs://falcon-server:8020/user/dataqa/testEffective/2016/01/12/23/00 -
2016-01-14T01:00Z local - WAITING - - hdfs://falcon-server:8020/user/dataqa/testEffective/2016/01/13/23/00 -
2016-01-15T01:00Z local - WAITING - - hdfs://falcon-server:8020/user/dataqa/testEffective/2016/01/14/23/00 -
2016-01-16T01:00Z local - WAITING - - hdfs://falcon-server:8020/user/dataqa/testEffective/2016/01/15/23/00 -
2016-01-17T01:00Z local - WAITING - - hdfs://falcon-server:8020/user/dataqa/testEffective/2016/01/16/23/00 -
2016-01-18T01:00Z local - WAITING - - hdfs://falcon-server:8020/user/dataqa/testEffective/2016/01/17/23/00 -
2016-01-19T01:00Z local - WAITING - - hdfs://falcon-server:8020/user/dataqa/testEffective/2016/01/18/23/00 -
2016-01-20T01:00Z local - WAITING - - hdfs://falcon-server:8020/user/dataqa/testEffective/2016/01/19/23/00 -

pallavi-rao · 2016-02-18T04:41:06Z

prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java

            AbstractWorkflowEngine wfEngine = getWorkflowEngine(entityObject);
-            return getInstanceResultSubset(wfEngine.getStatus(entityObject,
-                            startAndEndDate.first, startAndEndDate.second, lifeCycles, allAttempts),
+            InstancesResult instancesResult = getInstanceResultSubset(wfEngine.getStatus(entityObject,


Doesn't getInstanceResultSubset already apply numResults before returning? If there are i1-i20 instances and if startTime is i11 and numResults is 5, does this code return: i16 - i20 OR i11 - i15 ?

From the code changes, I think it will be i16-i20. But, what Pragya is asking for is i11-i15. Can you please confirm?

Yes it used to return after applying numResults but I have move that code to this method from line number 310 to 184.Now it will give us i11-i15.

I have also pasted the same in the testing.

pallavi-rao · 2016-02-19T09:53:24Z

I believe this will require more changes:

Change method signature of getInstanceResultSubset to indicate numResults is not used. Else, it is misleading. Even better would be if you could move this logic to getInstanceResultSubset as other methods can use it.
Make this behavior consistent across all APIs that filter by start time.
getRunningInstances uses getInstanceResultSubset. Will need changes there too.

pallavi-rao · 2016-02-24T10:56:50Z

+1. Sorry @PraveenAdlakha , I totally missed your update. Will merge right away.

$ bin/falcon instance -type process -name process1 -start 2015-10-09T14:00Z -status -numResults 2 Consolidated Status: SUCCEEDED Instances: Instance Cluster SourceCluster Status Start End Details Log ----------------------------------------------------------------------------------------------- 2015-12-31T01:00Z local - SUSPENDED 2016-02-11T09:34Z - - http://blr-test-129.corp.inmobi.com:11000/oozie?job=0000028-151229192519233-oozie-oozi-W actions: mr-node null null failed-post-processing null null 2016-01-01T01:00Z local - SUSPENDED 2016-02-11T10:18Z - - http://blr-test-129.corp.inmobi.com:11000/oozie?job=0000030-151229192519233-oozie-oozi-W actions: mr-node null null failed-post-processing null null Additional Information: Response: default/STATUS Request Id: default/124451500qtp-822600088-12 - 2ca2d36d-6e03-4e14-9e97-1448aed90084 Note we will not get the same result when status is waiting as at that time start time is null. Author: Praveen Adlakha <adlakha.praveen@gmail.com> Reviewers: Pallavi Rao <pallavi.rao@inmobi.com> Closes apache#40 from PraveenAdlakha/1811 and squashes the following commits: d9925d0 [Praveen Adlakha] pallavi's comments addressed 7784468 [Praveen Adlakha] checkstyle voilation issue solved 4530233 [Praveen Adlakha] changes done as per pallavi's comment 3c3da61 [Praveen Adlakha] Deepak's comments addressed de08573 [Praveen Adlakha] FALCON-1811 Status API does not honour start option

FALCON-1811 Status API does not honour start option

de08573

deepakbarr reviewed Feb 11, 2016
View reviewed changes

Deepak's comments addressed

3c3da61

pallavi-rao reviewed Feb 12, 2016
View reviewed changes

PraveenAdlakha force-pushed the 1811 branch from d539860 to 2de7135 Compare February 15, 2016 15:28

PraveenAdlakha force-pushed the 1811 branch from 2de7135 to 7cf6302 Compare February 15, 2016 16:49

changes done as per pallavi's comment

4530233

PraveenAdlakha force-pushed the 1811 branch from 7cf6302 to 4530233 Compare February 15, 2016 18:17

checkstyle voilation issue solved

7784468

pallavi-rao reviewed Feb 18, 2016
View reviewed changes

pallavi's comments addressed

d9925d0

asfgit closed this in e270b5c Feb 24, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

FALCON-1811 Status API does not honour start option #40

FALCON-1811 Status API does not honour start option #40

Uh oh!

PraveenAdlakha commented Feb 11, 2016

Uh oh!

deepakbarr Feb 11, 2016

Uh oh!

PraveenAdlakha Feb 11, 2016

Uh oh!

deepakbarr commented Feb 11, 2016

Uh oh!

pallavi-rao Feb 12, 2016

Uh oh!

PraveenAdlakha commented Feb 15, 2016

Uh oh!

pallavi-rao Feb 18, 2016

Uh oh!

PraveenAdlakha Feb 18, 2016

Uh oh!

pallavi-rao commented Feb 19, 2016

Uh oh!

pallavi-rao commented Feb 24, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

FALCON-1811 Status API does not honour start option #40

FALCON-1811 Status API does not honour start option #40

Uh oh!

Conversation

PraveenAdlakha commented Feb 11, 2016

Instance Cluster SourceCluster Status Start End Details Log

Uh oh!

deepakbarr Feb 11, 2016

Choose a reason for hiding this comment

Uh oh!

PraveenAdlakha Feb 11, 2016

Choose a reason for hiding this comment

Uh oh!

deepakbarr commented Feb 11, 2016

Uh oh!

pallavi-rao Feb 12, 2016

Choose a reason for hiding this comment

Uh oh!

PraveenAdlakha commented Feb 15, 2016

Instance Cluster SourceCluster Status Start End Details Log

Uh oh!

pallavi-rao Feb 18, 2016

Choose a reason for hiding this comment

Uh oh!

PraveenAdlakha Feb 18, 2016

Choose a reason for hiding this comment

Uh oh!

pallavi-rao commented Feb 19, 2016

Uh oh!

pallavi-rao commented Feb 24, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants