This repository has been archived by the owner on Apr 4, 2021. It is now read-only.
FALCON-2196 User Extensions Documentation #381
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… submission and colo addition to schedule command
pallavi-rao
suggested changes
Apr 3, 2017
docs/src/site/twiki/Extensions.twiki
Outdated
| @@ -2,64 +2,11 @@ | |||
|
|
|||
| ---++ Overview | |||
|
|
|||
| A Falcon extension is a static process template with parameterized workflow to realize a specific use case and enable non-programmers to capture and re-use very complex business logic. Extensions are defined in server space. Objective of the extension is to solve a standard data management function that can be invoked as a tool using the standard Falcon features (REST API, CLI and UI access) supporting standard falcon features. | |||
| A Falcon extension is an implementation comprising entities with a specific use case and enable non-programmers to capture and re-use very complex business logic. Objective of an extension is to solve a standard data processing/management function that can be invoked as a tool using the standard Falcon features (REST API, CLI and UI access) supporting standard falcon features. | |||
There was a problem hiding this comment.
Can we reword - "A Falcon extension is a solution template comprising of entities that solve a specific use case."
There was a problem hiding this comment.
Ack. Made the changes.
docs/src/site/twiki/Extensions.twiki
Outdated
|
|
||
| For example: | ||
| There are two types of extensions: |
There was a problem hiding this comment.
Falcon currently support two types of extensions:
There was a problem hiding this comment.
Ack. Made the changes.
|
|
||
| ---++ Overview | ||
|
|
||
| A Falcon Trusted extension is a static process template with parameterized workflow to realize a specific use case and enable non-programmers to capture and re-use very complex business logic. Trusted Extensions are defined in server space. Objective of the trusted extension is to solve a standard data management function that can be invoked as a tool using the standard Falcon features (REST API, CLI and UI access) supporting standard falcon features. |
There was a problem hiding this comment.
"Objective of .... " is a repetition of uber level overview.
There was a problem hiding this comment.
Modified accordingly
docs/src/site/twiki/Extensions.twiki
Outdated
| @@ -2,64 +2,11 @@ | |||
|
|
|||
| ---++ Overview | |||
|
|
|||
| A Falcon extension is a static process template with parameterized workflow to realize a specific use case and enable non-programmers to capture and re-use very complex business logic. Extensions are defined in server space. Objective of the extension is to solve a standard data management function that can be invoked as a tool using the standard Falcon features (REST API, CLI and UI access) supporting standard falcon features. | |||
| A Falcon extension is an implementation comprising entities with a specific use case and enable non-programmers to capture and re-use very complex business logic. Objective of an extension is to solve a standard data processing/management function that can be invoked as a tool using the standard Falcon features (REST API, CLI and UI access) supporting standard falcon features. | |||
| * Replicating between HDFS and Hive - either way | ||
| * Data masking etc. | ||
|
|
||
| ---++ Proposal |
There was a problem hiding this comment.
Not a proposal.. Consider moving this to overview section.
| @@ -0,0 +1,232 @@ | |||
| ---+ Falcon User Extensions | |||
|
|
||
| ---++ Overview | ||
|
|
||
| A Falcon User extension is an Extension Developer's implementation of Falcon Extension builder. A user extension can be used by all falcon users. |
There was a problem hiding this comment.
This doesn't bring out the intent. May be something like this:
A falcon user extension is a solution template that comprises of Falcon entities that a Falcon user can build. The extension so built can be made available to and discovered by other users (sharing the same Falcon installation). Extension users can instantiate the solution template and deploy it by providing relevant parameters.
|
|
||
| ---+++ Extension Registration API | ||
| ---++++ Extension Registration: | ||
| An extension developer has to register his extension with Falcon in order to make it discoverable to other users. Registration of the extension, enables it by default. |
|
|
||
|
|
||
| ---++ Extension Developer | ||
| This user is the developer of an user extension.Following are the various actions a falcon extension developer performs: |
| Sample Output: | ||
| falcon/default/resume successful (feed) out-feed-abc | ||
| submit/falcon/default/resume successful (process) pig-process-abc | ||
| </verbatim> |
There was a problem hiding this comment.
Do you also want to add a note saying instance operations will have to be performed using the falcon instance API?
|
|
||
| For example: | ||
|
|
||
| * Replicating directories from one HDFS cluster to another (not timed partitions) |
There was a problem hiding this comment.
Feed replication from one cluster to another.
There was a problem hiding this comment.
Not necessarily feed replication, for example hdfs snapshot mirroring just replicates the directory without any feed.
| |-- <extension name> | ||
| |-- README // This file will be ‘cat’ when extension user issues the “describe” command. | ||
| |-- META | ||
| |-- properties file // This file will be ‘cat’ when extension user issues the “definition” command. |
There was a problem hiding this comment.
give an example for definition command here.
|
|
||
| ---+++ Details of an extension job | ||
| Gets the details of a particular job: | ||
| ---++++Command: bin/falcon extension -get -jobName <jobName> |
There was a problem hiding this comment.
bin/falcon extension -detail -jobName
There was a problem hiding this comment.
Corrected it.
|
Please add a note that extensions commands can be executed on only prism. |
| ---+++Suspend an Extension Job(Idempotent) | ||
| Suspending an extension job will suspend all the entities that are part of the extension. | ||
| As of now there is no support for suspending in specific colos in distributed mode. | ||
| ---++++Commands: falcon extension -suspend -extensionName <extension name> -jobName <name to be used in all the extension entities> |
There was a problem hiding this comment.
extensionName is not required in suspend command.
| ---+++Resume an Extension Job(Idempotent) | ||
| Resuming an extension job will resume all the entities that are part of the extension. | ||
| As of now there is no support for resuming in specific colos in distributed mode. | ||
| ---++++Commands: falcon extension -resume -extensionName <extension name> -jobName <name to be used in all the extension entities> |
There was a problem hiding this comment.
extensionName argument is not required for resume.
| ---+++ Extension Instatiation validation. | ||
| Validates the configs supplied for an instantiation of the given extension. | ||
| <verbatim> | ||
| Sample Command: falcon extension -validate -extensionName data-quality-check -file extensions/hdfs-mirroring//META/hdfs-mirroring-properties.json |
There was a problem hiding this comment.
jobName is required for validate feature.
https://github.com/apache/falcon/blob/master/cli/src/main/java/org/apache/falcon/cli/FalconExtensionCLI.java#L134
| <verbatim> | ||
| Sample Output: | ||
| falcon/default/Submit successful (feed) out-feed-abc | ||
| submit/falcon/default/Submit successful (process) pig-process-abc |
There was a problem hiding this comment.
you may move the schedule documentation here.
There was a problem hiding this comment.
Moved it below submit, but don't want to merge with submitAndSchedule.
| ---+++Delete an Extension Job(Idempotent) | ||
| Deletion of the entities that are part of an extension can only be deleted by deleting an extension. | ||
| All the write operations(submit, submitAndSchedule, update and delete) of entites that are part of an extension job can be done only via extension job apis. | ||
| ---++++Commands: falcon extension -delete -extensionName <extension name> -jobName <name to be used in all the extension entities> |
There was a problem hiding this comment.
extensionName is not required.
| Users will have an option to update a particular instance of an extension. He can supply new parameters during update. However, an update will always update the entities to use the latest libs of the extension. | ||
| ---++++Commands: | ||
| <verbatim> | ||
| falcon extension -update -extensionName <extension name> -jobName <name to be used in all the extension entities> -file <path to extension parameters> |
There was a problem hiding this comment.
extensionName is not required.
| ---+++Schedule an Extension Job(Idempotent) | ||
| A submitted extension job can be shceduled. | ||
| As of now there is no support for scheduling in specific colos in distributed mode. | ||
| ---++++Commands: falcon extension -schedule -extensionName <extension name> -jobName <name to be used in all the extension entities> |
There was a problem hiding this comment.
extensionName not required.
|
|
||
| ---+++ List of the instantiations of an extension | ||
| Lists all the instances of a given extension | ||
| ---++++Command: bin/falcon extension -list -extensionName <extension name> |
There was a problem hiding this comment.
Also mention, if extensionName not given - it will return all extension jobs
There was a problem hiding this comment.
Added info accordingly
pallavi-rao
approved these changes
Apr 7, 2017
docs/src/site/twiki/Extensions.twiki
Outdated
|
|
||
| There are two types of extensions: | ||
| Falcon currently support two types of extensions: |
| @@ -2,16 +2,16 @@ | |||
|
|
|||
| ---++ Overview | |||
|
|
|||
| A Falcon User extension is an Extension Developer's implementation of Falcon Extension builder. A user extension can be used by all falcon users. | |||
| There are mainly two roles into Falcon User Extensions: | |||
| A falcon user extension is a solution template that comprises of Falcon entities that a Falcon user can build. The extension so built can be made available to and discovered by other users (sharing the same Falcon installation). | |||
PracheerAgarwal-zz
approved these changes
Apr 10, 2017
pallavi-rao
pushed a commit
to pallavi-rao/falcon
that referenced
this pull request
Feb 16, 2018
Author: sandeep <sandysmdl@gmail.com> Author: sandeep <sandeep@kickseed.corp.inmobi.com> Reviewers: "Pallavi Rao <pallavi.rao@inmobi.com>, Pracheer Agarwal <pracheeragarwal@gmail.com>" Closes apache#381 from sandeepSamudrala/FALCON-2196 and squashes the following commits: 362dcec [sandeep] FALCON-2196 Incorporated review comments 485f38c [sandeep] FALCON-2196 Incorporated review comments b525b58 [sandeep] FALCON-2196 User Extensions Documentation 68b1a51 [sandeep] Merge branch 'master' of https://github.com/apache/falcon into FALCON-2196 cf7ea21 [sandeep] Merge branch 'master' of https://github.com/apache/falcon into FALCON-2196 79e8d64 [sandeep] Merge branch 'master' of https://github.com/apache/falcon 7de7798 [sandeep] go -b FALCON-2263Merge branch 'master' of https://github.com/apache/falcon c5da0a2 [sandeep] Merge branch 'master' of https://github.com/apache/falcon 7e16263 [sandeep] Merge branch 'master' of https://github.com/apache/falcon a234d94 [sandeep] FALCON-2231 Incoporated review comments and small fixes for duplicate submission and colo addition to schedule command 26e3350 [sandeep] Merge branch 'master' of https://github.com/apache/falcon 73fbf75 [sandeep] Merge branch 'master' of https://github.com/apache/falcon cc28658 [sandeep] Merge branch 'master' of https://github.com/apache/falcon 089b10d [sandeep] Merge branch 'master' of https://github.com/apache/falcon 456d4ee [sandeep] Merge branch 'master' of https://github.com/apache/falcon 0cf9af6 [sandeep] Merge branch 'master' of https://github.com/apache/falcon 4a2e23e [sandeep] Merge branch 'master' of https://github.com/apache/falcon b1546ed [sandeep] Merge branch 'master' of https://github.com/apache/falcon 0a433fb [sandeep] Merge branch 'master' of https://github.com/apache/falcon 194f36a [sandeep] Merge branch 'master' of https://github.com/apache/falcon e0ad358 [sandeep] Merge branch 'master' of https://github.com/apache/falcon f96a084 [sandeep] Merge branch 'master' of https://github.com/apache/falcon 9cf36e9 [sandeep] Merge branch 'master' of https://github.com/apache/falcon bbca081 [sandeep] Merge branch 'master' of https://github.com/apache/falcon 48f6afa [sandeep] Merge branch 'master' of https://github.com/apache/falcon 250cc46 [sandeep] Merge branch 'master' of https://github.com/apache/falcon d0393e9 [sandeep] Merge branch 'master' of https://github.com/apache/falcon a178805 [sandeep] Merge branch 'master' of https://github.com/apache/falcon d6dc8bf [sandeep] Merge branch 'master' of https://github.com/apache/falcon 1bb8d3c [sandeep] Merge branch 'master' of https://github.com/apache/falcon c065566 [sandeep] reverting last line changes made 1a4dcd2 [sandeep] rebased and resolved the conflicts from master 271318b [sandeep] FALCON-2097. Adding UT to the new method for getting next instance time with Delay. a94d4fe [sandeep] rebasing from master 9e68a57 [sandeep] FALCON-298. Feed update with replication delay creates holes
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.