Skip to content
This repository has been archived by the owner on Apr 4, 2021. It is now read-only.

FALCON-2276 Falcon Trusted extensions submission and validation was failing #360

Closed
wants to merge 35 commits into from
Closed

FALCON-2276 Falcon Trusted extensions submission and validation was failing #360

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
9e68a57
FALCON-298. Feed update with replication delay creates holes
sandeepSamudrala Jul 25, 2016
a94d4fe
rebasing from master
sandeepSamudrala Aug 5, 2016
271318b
FALCON-2097. Adding UT to the new method for getting next instance ti…
sandeepSamudrala Aug 5, 2016
1a4dcd2
rebased and resolved the conflicts from master
sandeepSamudrala Aug 5, 2016
c065566
reverting last line changes made
sandeepSamudrala Aug 5, 2016
1bb8d3c
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Aug 9, 2016
d6dc8bf
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Aug 14, 2016
a178805
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Sep 27, 2016
d0393e9
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Oct 19, 2016
250cc46
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Nov 14, 2016
48f6afa
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Nov 18, 2016
bbca081
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Nov 21, 2016
9cf36e9
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Nov 25, 2016
f96a084
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Dec 1, 2016
e0ad358
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Dec 7, 2016
194f36a
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Dec 15, 2016
0a433fb
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Dec 15, 2016
b1546ed
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Dec 22, 2016
4a2e23e
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Dec 26, 2016
0cf9af6
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Dec 26, 2016
456d4ee
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Dec 27, 2016
089b10d
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Dec 29, 2016
cc28658
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Dec 30, 2016
73fbf75
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Jan 3, 2017
26e3350
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Jan 5, 2017
a234d94
FALCON-2231 Incoporated review comments and small fixes for duplicate…
sandeepSamudrala Jan 10, 2017
7e16263
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Jan 17, 2017
c5da0a2
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Jan 23, 2017
7de7798
go -b FALCON-2263Merge branch 'master' of https://github.com/apache/f…
sandeepSamudrala Jan 24, 2017
79e8d64
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Jan 25, 2017
cb2b00d
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Jan 31, 2017
3f67fed
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Feb 1, 2017
db425c5
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Feb 7, 2017
a3bd0e9
Merge branch 'master' of https://github.com/apache/falcon
sandeepSamudrala Feb 8, 2017
84169d6
FALCON-2276 Falcon Trusted extensions submission and validation was f…
sandeepSamudrala Feb 8, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions client/src/main/java/org/apache/falcon/client/FalconClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1169,7 +1169,7 @@ private JSONObject getExtensionJobDetailJson(String jobName) {
private List<Entity> getEntities(String extensionName, String jobName, InputStream configStream,
String extensionType, String extensionBuildLocation) {
List<Entity> entities = null;
if (!extensionType.equals(ExtensionType.CUSTOM.name())) {
if (!extensionType.equals(ExtensionType.TRUSTED.toString())) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the comparison be with ExtensionType.TRUSTED.name()? Doesn't ExtensionType.TRUSTED.toString() result in "Trusted extension", rather than "TRUSTED"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comes as Trusted extension from the server as its .toString while coming from the server.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah.. got it. Thanks for clarifying.

try {
entities = ExtensionHandler.loadAndPrepare(extensionName, jobName, configStream,
extensionBuildLocation);
Expand Down Expand Up @@ -1209,7 +1209,7 @@ public APIResult validateExtensionJob(final String extensionName, final String j
final String configPath, final String doAsUser) {
String extensionType = ExtensionHandler.getExtensionType(extensionName, getExtensionDetailJson(extensionName));
InputStream configStream = getServletInputStream(configPath);
if (ExtensionType.TRUSTED.name().equalsIgnoreCase(extensionType)) {
if (extensionType.equals(ExtensionType.TRUSTED.toString())) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above name() vs. toString().

ClientResponse clientResponse = new ResourceBuilder()
.path(ExtensionOperations.VALIDATE.path, extensionName)
.addQueryParam(DO_AS_OPT, doAsUser)
Expand Down
4 changes: 4 additions & 0 deletions extensions/src/main/java/org/apache/falcon/extensions/Extension.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import org.apache.commons.lang3.StringUtils;
import org.apache.falcon.FalconException;
import org.apache.falcon.Pair;
import org.apache.falcon.entity.EntityUtil;
import org.apache.falcon.entity.v0.Entity;
import org.apache.falcon.entity.v0.feed.Schema;
import org.apache.falcon.extensions.store.ExtensionStore;
Expand Down Expand Up @@ -109,6 +110,9 @@ public List<Entity> getEntities(final String extensionName, final InputStream co
throw new FalconException("Entity created from the extension template cannot be null");
}
LOG.info("Extension processing complete");
// add tags on extension name and job
String jobName = configProperties.getProperty(ExtensionProperties.JOB_NAME.getName());
EntityUtil.applyTags(extensionName, jobName, Collections.singletonList(entity));
return Collections.singletonList(entity);
}

Expand Down
19 changes: 2 additions & 17 deletions prism/src/main/java/org/apache/falcon/resource/proxy/ExtensionManagerProxy.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
import java.util.SortedMap;
import java.util.TreeMap;
Expand Down Expand Up @@ -56,7 +55,6 @@
import org.apache.falcon.entity.v0.feed.Feed;
import org.apache.falcon.entity.v0.process.Process;
import org.apache.falcon.extensions.Extension;
import org.apache.falcon.extensions.ExtensionProperties;
import org.apache.falcon.extensions.ExtensionService;
import org.apache.falcon.extensions.ExtensionType;
import org.apache.falcon.extensions.jdbc.ExtensionMetaStore;
Expand Down Expand Up @@ -296,7 +294,7 @@ private SortedMap<EntityType, List<Entity>> getEntityList(String extensionName,
List<Entity> entities;
TreeMap<EntityType, List<Entity>> entityMap = new TreeMap<>();
if (ExtensionType.TRUSTED.equals(extensionType)) {
entities = generateEntities(extensionName, config);
entities = extension.getEntities(extensionName, config);
List<Entity> trustedFeeds = new ArrayList<>();
List<Entity> trustedProcesses = new ArrayList<>();
for (Entity entity : entities) {
Expand Down Expand Up @@ -564,7 +562,7 @@ public APIResult validate(
throw FalconWebException.newAPIException("Extension validation is supported only for trusted extensions");
}
try {
List<Entity> entities = generateEntities(extensionName, request.getInputStream());
List<Entity> entities = extension.getEntities(extensionName, request.getInputStream());
for (Entity entity : entities) {
super.validate(entity);
}
Expand Down Expand Up @@ -715,19 +713,6 @@ public APIResult enableExtension(
}
}

private List<Entity> generateEntities(String extensionName, InputStream configStream)
throws FalconException, IOException {
// get entities for extension job
Properties properties = new Properties();
properties.load(configStream);
List<Entity> entities = extension.getEntities(extensionName, configStream);

// add tags on extension name and job
String jobName = properties.getProperty(ExtensionProperties.JOB_NAME.getName());
EntityUtil.applyTags(extensionName, jobName, entities);
return entities;
}

private static void checkIfExtensionServiceIsEnabled() {
if (!Services.get().isRegistered(ExtensionService.SERVICE_NAME)) {
LOG.error(ExtensionService.SERVICE_NAME + " is not enabled.");
Expand Down