Skip to content

Commit

Permalink
update developer documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@danhermann
danhermann committed Apr 15, 2019
1 parent 9bbc9b8 commit fd425b4
Show file tree
Hide file tree
Showing 6 changed files with 76 additions and 193 deletions.
149 changes: 49 additions & 100 deletions docs/static/include/javapluginpkg.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,122 +2,72 @@
=== Package and deploy

Java plugins are packaged as Ruby gems for dependency management and
interoperability with Ruby plugins.

NOTE: One of the goals for Java plugin support is to eliminate the need for any
knowledge of Ruby or its toolchain for Java plugin development. Future phases of
the Java plugin project will automate the packaging of Java plugins as Ruby gems
so no direct knowledge of or interaction with Ruby will be required. In the
current phase, Java plugins must still be manually packaged as Ruby gems
and installed with the `logstash-plugin` utility.

[float]
==== Compile to JAR file

The Java plugin should be compiled and assembled into a fat jar with the
`vendor` task in the Gradle build file. This will package all Java dependencies
into a single jar and write it to the correct folder for later packaging into a
Ruby gem.
interoperability with Ruby plugins. Once they are packaged as gems, they may
be installed with the `logstash-plugin` utility just as Ruby plugins are.
Because no knowledge of Ruby or its toolchain should be required for Java
plugin development, the procedure for packaging Java plugins as Ruby gems
has been automated through a custom task in the Gradle build file provided
with the example Java plugins. The following sections describe how to
configure and execute that packaging task as well as how to install the
packaged Java plugin in Logstash.

[float]
==== Manually package as Ruby gem

Several Ruby source files are required to package the jar file as a
Ruby gem. These Ruby files are used only at Logstash startup time to identify
the Java plugin and are not used during runtime event processing.

NOTE: These Ruby source files will be automatically generated in a future release.
==== Configuring the Gradle packaging task

**+logstash-{plugintype}-<{plugintype}-name>.gemspec+**
The following section appears near the top of the `build.gradle` file supplied
with the example Java plugins:

[source,txt]
[source,java]
[subs="attributes"]
-----
Gem::Specification.new do |s|
s.name = 'logstash-{plugintype}-java_{plugintype}_example'
s.version = PLUGIN_VERSION
s.licenses = ['Apache-2.0']
s.summary = "Example {plugintype} using Java plugin API"
s.description = ""
s.authors = ['Elasticsearch']
s.email = 'info@elastic.co'
s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
s.require_paths = ['lib', 'vendor/jar-dependencies']
# Files
s.files = Dir["lib/**/*","spec/**/*","*.gemspec","*.md","CONTRIBUTORS","Gemfile","LICENSE","NOTICE.TXT", "vendor/jar-dependencies/**/*.jar", "vendor/jar-dependencies/**/*.rb", "VERSION", "docs/**/*"]
# Special flag to let us know this is actually a logstash plugin
s.metadata = { 'logstash_plugin' => 'true', 'logstash_group' => '{plugintype}'}
# Gem dependencies
s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
s.add_runtime_dependency 'jar-dependencies'
s.add_development_dependency 'logstash-devutils'
end
// ===========================================================================
// plugin info
// ===========================================================================
group 'org.logstash.javaapi' // must match the package of the main plugin class
version "${file("VERSION").text.trim()}" // read from required VERSION file
description = "Example Java filter implementation"
pluginInfo.licenses = ['Apache-2.0'] // list of SPDX license IDs
pluginInfo.longDescription = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using \$LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
pluginInfo.authors = ['Elasticsearch']
pluginInfo.email = ['info@elastic.co']
pluginInfo.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
pluginInfo.pluginType = "filter"
pluginInfo.pluginClass = "JavaFilterExample"
pluginInfo.pluginName = "java_filter_example"
// ===========================================================================
-----

You can use this file with the following modifications:
You should configure the values above for your plugin.

* `s.name` must follow the +logstash-pass:attributes[{plugintype}]-<{plugintype}-name>+ pattern
* `s.version` must match the `project.version` specified in the `build.gradle` file.
Both versions should be set to be read from the `VERSION` file in this example.

**+lib/logstash/{plugintype}s/<{plugintype}-name>.rb+**

[source,ruby]
[subs="attributes"]
-----
# encoding: utf-8
require "logstash/{plugintype}s/base"
require "logstash/namespace"
require "logstash-{plugintype}-java_{plugintype}_example_jars"
require "java"
class LogStash::{plugintype}s::Java{plugintypecap}Example < LogStash::{pluginclass}::Base
config_name "java_{plugintype}_example"
def self.javaClass() org.logstash.javaapi.Java{plugintypecap}Example.java_class; end
end
-----
* The `version` value will be automatically read from the `VERSION` file in the
root of your plugin's codebase.
* `pluginInfo.pluginType` should be set to one of `input`, `filter`, `codec`,
or `output`.
* `pluginInfo.pluginName` must match the name specified on the `@LogstashPlugin`
annotation on the main plugin class. The Gradle packaging task will validate
that and return an error if they do not match.

Modify these items in the file above:
[float]
==== Running the Gradle packaging task

* Change the name to correspond with the {plugintype} name.
* Change +require "logstash-{plugintype}-java_{plugintype}_example_jars"+ to reference the appropriate "jars" file
as described below.
* Change +class LogStash::{pluginclass}::Java{plugintypecap}Example < LogStash::{pluginclass}::Base+ to provide a unique and
descriptive Ruby class name.
* Change +config_name "java_{plugintype}_example"+ to match the name of the plugin as specified in the `name` property of
the `@LogstashPlugin` annotation.
* Change +def self.javaClass() org.logstash.javaapi.Java{plugintypecap}Example.java_class; end+ to return the
class of the Java {plugintype}.
Several Ruby source files along with a `gemspec` file and a `Gemfile` are
required to package the plugin as a Ruby gem. These Ruby files are used only
for defining the Ruby gem structure or at Logstash startup time to register
the Java plugin. They are not used during runtime event processing. The
Gradle packaging task automatically generates all of these files based on
the values configured in the section above.

**+lib/logstash-{plugintype}-<{plugintype}-name>_jars.rb+**
You run the Gradle packaging task with the following command:

[source,txt]
[subs="attributes"]
[source,shell]
-----
require 'jar_dependencies'
require_jar('org.logstash.javaapi', 'logstash-{plugintype}-java_{plugintype}_example', {sversion})
./gradlew gem
-----

In the file above:

* Rename the file to correspond to the {plugintype} name.
* Change the `require_jar` directive to correspond to the `group` specified in the
Gradle build file, the name of the {plugintype} JAR file, and the version as
specified in both the gemspec and Gradle build file.

After you have created the previous files and the plugin JAR file, build the gem using the
following command:
For Windows platforms: Substitute `gradlew.bat` for `./gradlew` as appropriate in the command.

[source,shell]
[subs="attributes"]
-----
gem build logstash-{plugintype}-<{plugintype}-name>.gemspec
-----
That task will produce a gem file in the root directory of your
plugin's codebase with the name `logstash-{plugintype}-<pluginName>-<version>.gem`

[float]
==== Installing the Java plugin in Logstash
Expand All @@ -131,4 +81,3 @@ bin/logstash-plugin install --no-verify --local /path/to/javaPlugin.gem
-----

For Windows platforms: Substitute backslashes for forward slashes as appropriate in the command.

4 changes: 2 additions & 2 deletions docs/static/include/javapluginsetup.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ git clone --branch <branch_name> --single-branch https://github.com/elastic/logs
The `branch_name` should correspond to the version of Logstash containing the
preferred revision of the Java plugin API.

NOTE: The beta version of the Java plugin API is available in the `6.7`
branch of the Logstash codebase.
NOTE: The GA version of the Java plugin API is available in the `7.1`
and later branches of the Logstash codebase.

Specify the `target_folder` for your local copy of the Logstash codebase. If you
do not specify `target_folder`, it defaults to a new folder called `logstash`
Expand Down
92 changes: 16 additions & 76 deletions docs/static/java-codec.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,6 @@
[[java-codec-plugin]]
=== How to write a Java codec plugin

beta[]

// Pulls in shared section: Setting Up Environment
include::include/javapluginsetup.asciidoc[]

Expand Down Expand Up @@ -48,8 +46,6 @@ public class JavaCodecExample implements Codec {
private final String id;
private final String delimiter;
private final CharsetEncoder encoder;
private Event currentEncodedEvent;
private CharBuffer currentEncoding;
public JavaCodecExample(final Configuration config, final Context context) {
this(config.get(DELIMITER_CONFIG));
Expand Down Expand Up @@ -87,30 +83,8 @@ public class JavaCodecExample implements Codec {
}
@Override
public boolean encode(Event event, ByteBuffer buffer) throws EncodeException {
try {
if (currentEncodedEvent != null && event != currentEncodedEvent) {
throw new EncodeException("New event supplied before encoding of previous event was completed");
} else if (currentEncodedEvent == null) {
currentEncoding = CharBuffer.wrap(event.toString() + delimiter);
}
CoderResult result = encoder.encode(currentEncoding, buffer, true);
buffer.flip();
if (result.isError()) {
result.throwException();
}
if (result.isOverflow()) {
currentEncodedEvent = event;
return false;
} else {
currentEncodedEvent = null;
return true;
}
} catch (IOException e) {
throw new IllegalStateException(e);
}
public void encode(Event event, OutputStream outputStream) throws IOException {
outputStream.write((event.toString() + delimiter).getBytes(Charset.defaultCharset()));
}
@Override
Expand All @@ -128,6 +102,7 @@ public class JavaCodecExample implements Codec {
public String getId() {
return this.id;
}
}
-----

Expand Down Expand Up @@ -235,36 +210,17 @@ public void decode(ByteBuffer byteBuffer, Consumer<Map<String, Object>> consumer
@Override
public void flush(ByteBuffer byteBuffer, Consumer<Map<String, Object>> consumer) {
// if the codec maintains any internal state such as partially-decoded input, this
// method should flush that state along with any additional input supplied in
// the ByteBuffer
decode(byteBuffer, consumer); // this is a simplistic implementation
}
@Override
public boolean encode(Event event, ByteBuffer buffer) throws EncodeException {
try {
if (currentEncodedEvent != null && event != currentEncodedEvent) {
throw new EncodeException("New event supplied before encoding of previous event was completed");
} else if (currentEncodedEvent == null) {
currentEncoding = CharBuffer.wrap(event.toString() + delimiter);
}
CoderResult result = encoder.encode(currentEncoding, buffer, true);
buffer.flip();
if (result.isError()) {
result.throwException();
}
if (result.isOverflow()) {
currentEncodedEvent = event;
return false;
} else {
currentEncodedEvent = null;
return true;
}
} catch (IOException e) {
throw new IllegalStateException(e);
}
public void encode(Event event, OutputStream outputStream) throws IOException {
outputStream.write((event.toString() + delimiter).getBytes(Charset.defaultCharset()));
}
-----

The `decode`, `flush`, and `encode` methods provide the core functionality of
Expand All @@ -281,7 +237,7 @@ responsible for returning the buffer to write mode via either
`byteBuffer.clear()` or `byteBuffer.compact()` before resuming writes. In the
example above, the `decode` method simply splits the incoming byte stream on the
specified delimiter. A production-grade codec such as
https://github.com/elastic/logstash/blob/6.7/logstash-core/src/main/java/org/logstash/plugins/codecs/Line.java[`java-line`]
https://github.com/elastic/logstash/blob/master/logstash-core/src/main/java/org/logstash/plugins/codecs/Line.java[`java-line`]
would not make the simplifying assumption that the end of the supplied byte
stream corresponded with the end of an event.

Expand All @@ -303,25 +259,9 @@ above is a simplistic implementation that does not maintain any state about
partially-supplied byte streams across calls to `decode`.

The `encode` method encodes an event into a sequence of bytes and writes it into
the specified `ByteBuffer`. Under ideal circumstances, the entirety of the
event's encoding will fit into the supplied buffer. In cases where the buffer
has insufficient space to hold the event's encoding, the codec must fill the
buffer with as much of the event's encoding as possible, the `encode` must
return `false`, and the output must call the `encode` method with the same event
and a buffer that has more `buffer.remaining()` bytes. The output typically does
that by draining the partial encoding from the supplied buffer. This process
must be repeated until the event's entire encoding is written to the buffer at
which point the `encode` method will return `true`. Attempting to call this
method with a new event before the entirety of the previous event's encoding has
been written to a buffer must result in an `EncodeException`. As the coneptual
inverse of the `decode` method, the `encode` method must return the buffer in a
state from which it can be read, typically by calling `buffer.flip()` before
returning. In the example above, the `encode` method attempts to write the
event's encoding to the supplied buffer. If the buffer contains sufficient free
space, the entirety of the event is written and `true` is returned. Otherwise,
the method writes as much of the event's encoding to the buffer as possible,
returns `false`, and stores the remainder to be written to the buffer in the
next call to the `encode` method.
the specified `OutputStream`. Because a single codec instance is shared across
all pipeline workers in the output stage of the Logstash pipeline, codecs should
_not_ retain state across calls to their `encode` methods.

[float]
==== cloneCodec method
Expand Down Expand Up @@ -371,11 +311,11 @@ To test the plugin, start Logstash with:

[source,java]
-----
echo "foo,bar" | bin/logstash --java-execution -e 'input { java_stdin { codec => java_codec_example } } }'
echo "foo,bar" | bin/logstash -e 'input { java_stdin { codec => java_codec_example } } }'
-----

NOTE: The `--java-execution` flag to enable the Java execution engine is required as Java plugins are not supported
in the Ruby execution engine.
NOTE: The Java execution engine, the default execution engine since Logstash 7.0, is required
as Java plugins are not supported in the Ruby execution engine.

The expected Logstash output (excluding initialization) with the configuration above is:

Expand Down
8 changes: 3 additions & 5 deletions docs/static/java-filter.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@
[[java-filter-plugin]]
=== How to write a Java filter plugin

beta[]

// Pulls in shared section: Setting Up Environment
include::include/javapluginsetup.asciidoc[]

Expand Down Expand Up @@ -248,11 +246,11 @@ Start Logstash with:

[source,shell]
-----
bin/logstash --java-execution -f /path/to/java_filter.conf
bin/logstash -f /path/to/java_filter.conf
-----

NOTE: The `--java-execution` flag to enable the Java execution engine is
required as Java plugins are not supported in the Ruby execution engine.
NOTE: The Java execution engine, the default execution engine since Logstash 7.0, is required
as Java plugins are not supported in the Ruby execution engine.

The expected Logstash output (excluding initialization) with the configuration
above is:
Expand Down
8 changes: 3 additions & 5 deletions docs/static/java-input.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@
[[java-input-plugin]]
=== How to write a Java input plugin

beta[]

// Pulls in shared section: Setting Up Environment
include::include/javapluginsetup.asciidoc[]

Expand Down Expand Up @@ -274,11 +272,11 @@ Start {ls} with:

[source,shell]
-----
bin/logstash --java-execution -f /path/to/java_input.conf
bin/logstash -f /path/to/java_input.conf
-----

NOTE: The `--java-execution` flag to enable the Java execution engine is
required as Java plugins are not supported in the Ruby execution engine.
NOTE: The Java execution engine, the default execution engine since Logstash 7.0, is required
as Java plugins are not supported in the Ruby execution engine.

The expected Logstash output (excluding initialization) with the configuration above is:

Expand Down
Loading

0 comments on commit fd425b4

Please sign in to comment.