…irection if set in the configuration

This closes #1281

Before, this was not set, leading to incorrect results if KV-State was
used in the WindowFunction.

This also adds a test.

…failed jobs in new web dashboard

This closes #1287

…er appended

This closes #1279

This closes #1252

Before it would only trigger if expectedTime < time. Now it is
expectedTime <= time.

Before, the StreamRecords was not copied, now it is.

This fixes the aggregators to make copies of the objects so that it
works with window operators that are not mutable-object safe.

This also disables object copy in WindowOperator and
NonKeyedWindowOperator.

…k-runtime tests

Rename Match*Test to Join*Test and MapTaskTest to FlatMapTaskTest

This closes #1294

Rename config key prefix from 'ha.zookeeper' to 'recovery.zookeeper'
Rename config key from 'state.backend.fs.dir.recovery' => 'state.backend.fs.recoverydir'
Move ZooKeeper file system state backend configuration keys

This closes #1286

Operators defer object creation when object reuse is disabled.

This closes #1288

Before these trigger methods had no information about the window that
they are responsible for. This information might be required for
implementing more advanced trigger behaviour.

This enhances the TriggerResult enum with methods isFire() and isPurge()
that simplify the logic in WindowOperator.processTriggerResult().

Also, the operator now keeps track of the current watermark and fires
immediately if a trigger registers an event-time callback for a
timestamp that lies in the past. For this the TriggerResult now as
method merge() that allows to merge to TriggerResultS.

…handle

This closes #1282

…values

…ation of fast-path windows.

The HandlerRedirectUtils.getRedirectAddress decides whether the retrieved leader address is equal to the local job manager address. The local job manager address is, however, in the form akka.tcp://flink@url/user/jobmanager whereas the leader address can be akka://flink/user/jobmanager if the local job manager is the current leader. Such a case produced a warning which is not correct. This PR checks for the local job manager address and signals that no redirection has to be done if it receives akka://flink/user/jobmanager.

Add test for HandlerRedirectUtils

This closes #1280.

…nagement

… flink-conf of binary distribution

Otherwise the javadoc fails to generate..

….log and taskmanager.stdout

This closes #1307

…ning jobs by start time

This closes #1296

…nstead of http

This closes #1309

The Kryo serializer uses Kryo's Output class to buffer individual write operations before
it is written to the underlying output stream. This Output class is flushed by Flink's
KryoSerializer upon finishing its serialize call. However, in case of an exception when
flushing the Output, the buffered data is kept in the buffer. Since Flink uses EOFExceptions
to mark that an underlying buffer is full and has to be spilled, for example, it can happen
that the record triggering the spilling is written twice after it is rewritten. The reason
is that Kryo's Output buffer still contains the serialization data of the failed attempt which
is also flushed to the emptied output stream.

This duplication of records can lead to corrupted data which eventually let's the Flink program
crash. The problem is solved by clearing Kryo's Output when the flush operation was not successful.

This closes #1308

- Remove broken HTML tags like <br/>, <p/>, ...
- close unclosed HTML tags
- replaces special chars by HTML escaping, e.g., '<' by &lt;
- wrap code examples by {@code}
- fix incorrect @see and @link references
- fix incorrect @throws
- fix typos

This closes #1298

…versions

- Added tests for Scala DataSet sortPartition

- fix hard-coded defaults

The HashPartition did not count properly the number of occupied memory
segments, because it excluded the memory segments of the
BuildSideBuffer. That caused the MutableHashTable to fail when trying to
spill a partition which did not have any overflow segments. This PR
fixes the problem by also counting the memory segments of the
BuildSideBuffer.

…ived/sent.

[hotfix][web-dashboard] Fix taskmanager charts legend.

…unters.

… type information to WindowFunction

The fold operation needs the output type information to serialize the
initial value of it. Therefore, the OutputTypeConfigurable interface was
defined. It is called by the StreamGraph upon adding a StreamOperator to
the StreamGraph. Since some stream operators, such as the window stream
operator, don't work directly on the data, but instead call a
WindowFunction for the actual logic, the output type information has to
be forwarded to this function to set output type information at the
right place.

Thus, the AbstractUdfStreamOperator checks whether its udf function
supports the OutputTypeConfigurable interface. If this is the case, then
it forwards the output type information to the udf.

The problem was, that truncate is asynchronous and the RollingSink was
not taking this into account.

Now it has a loop after the truncate call that waits until the file is
actually truncated.

This also changes the Hadoop 2.6 travis build to 2.7, instead.

Previously, this fiel pointed to the different versions of the LICENSE
and NOTICE files (source version, bin version). Since we dropped the
convenience binary distribution LICENSE and NOTICE files and only ship the
source release related ones, this is no longer necessary.

…n to make Flink on Hadoop 2.6.0+ work

This closes #1340

This closes #1342

Previously the internal currentWatermark would be updated even if the
value returned from getCurrentWatermark was lower than the current
watermark.

This can lead to problems with chaining because the watermark is
directly forwarded without going through the watermark logic that
ensures correct behaviour (monotonically increasing).

This adds a test that verifies that the timestamp extractor does not
emit decreasing watermarks.

This closes #1348

…gy, give the local host address a higher priority

This closes #1391

…lization

This closes #1346

…0.7 for bugfixes

This closes #1365

…tor).of()

In addition to the REST-compliant "DELETE /jobs/<jobid>", allows
cancellation also via a special GET request of the form
"GET /jobs/<jobid>/yarn-cancel".

That enables us to cancel jobs from the web frontend on YARN while
keeping a REST-compliant DELETE alternative.

…ackend factory.

…mpletenessTest.scala

This closes #1356.

The Graph.fromCsvReader in Gelly's Scala API returns a wrongly typed Graph instance because
the implementation contains code paths with return types which are not compatible. This
lead to a bad user experience.

This commit fixes this. However, it also introduces a slightly different behaviour and is
thus API breaking. Before there were parameters hasEdgeValues and readVertices which
controlled whether edge values and vertices shall be read. This is now implicitly controlled
by the types of the vertex and edge value. If the type is NullValue for the edge values then
the edge value won't be read from the given csv file. If the vertex value type is NullValue,
then all vertices will have a value of a NullValue instance. If the pathVertices is not
specified and the vertex value type is unequal to NullValue, then the vertexValueInitializer
is used for the initialization.

Add documentation for Gelly's Scala API method Graph.fromCsvReader

This closes #1370.

This closes #1367.

This closes #1357

…on of Stream Programming Guide

…mming guide refers to an incorrect example 'TwitterLocal'

This closes #1379.

…serialization remote command execution vulnerability

This closes #1381

This also adds to the deprecated classes pointers forward to the designated classes.

This closes #1380

…FAILED to RESTARTING

Removes the possibility to go from FAILED state back to RESTARTING. This was only used in a test
case. It was a breaking the terminal state semantics of the FAILED state.

…Java

This fixes a problem that would occur if a Trigger registers a new
processing-time trigger in the onProcessingTime method. The problem is
that onProcessingTime() is called while traversing the set of active
triggers. If onProcessingTime() tries to register a new processing-time
trigger this leads to a concurrent modification exception.

…nces

This closes #1345

This commit is also changing how we build the "flink-shaded-hadoop" artifact.
In the past, we were including all Hadoop dependencies into a fat jar, without relocating all of them.
Maven was not able to see Hadoop's dependencies and classes ended up in the classpath multiple times.

With this change, only shaded Hadoop dependencies are included into the jar. The shade plugin will also
remove only the shaded dependencies from the pom file.

Streaming sources were directly assigned their InputFormat in the StreamingJobGraphGenerator. As a consequence, the input formats were directly serialized/deserialized by Akka when the JobGraph was sent to the JobManager. In cases where the user provided a custom input format or an input format with custom types, this could lead to a ClassDefNotFoundException, because the system class loader instead of the user code class loader is used by Akka for the deserialization.

The problem was fixed by wrapping the InputFormat into a UserCodeObjectWrapper which is shipped ot the JobManager via the JobVertex's configuration. By instantiating stream sources as InputFormatVertices, the corresponding InputFormat is retrieved from the Configuration in the initializeOnMaster method call.

This closes #1368.

…ng is disabled

When a candidate for a bulk iteration is instantiated, then the optimizer creates candidates
for the step function. It is then checked that there exists a candidate solution for the step
function whose properties met the properties of the input to the bulk iteration. Sometimes
it is necessary to add a no-op plan node to the end of the step function to generate the
correct properties. These new candidates have to be added to the final set of the accepted
candidates.

This commit adds that these new candidates are properly added to the set of accepted candidates.

Fix test and add new iteration tests

Add predecessor operator and dynamic path information to no op operator in bulk iterations

This closes #1388.

…nment.readSequenceFile()

This closes #1389.

…sm in local execution