Skip to content
Permalink
Browse files
IGNITE-16331 Add topology validator implementation extension. (#86)
  • Loading branch information
ololo3000 committed Jan 21, 2022
1 parent 06104c7 commit aa2a16864f7fa873ac86c0579b78f1225f77cb2d
Showing 8 changed files with 1,191 additions and 0 deletions.
@@ -0,0 +1,68 @@
#What problem this module is intended to solve?

Some network issues can cause the Ignite cluster to split into several isolated parts - segments. Nodes from different
segments cannot communicate with each other, while nodes from the same segment do not experience communication problems.
In this case, each segment marks the nodes with which the connection was lost as failed and considers itself as an
independent Ignite cluster. Let's call this scenario cluster segmentation.

Cluster segmentation can lead to cache data inconsistency across different segments because each segment can continue
to handle cache update requests independently.

Apache Ignite allows the user to provide custom validation logic for Ignite caches that will be applied to
each topology change, and if the validation fails, writes to the corresponding cache will be blocked. The mentioned
validation logic is passed to Ignite as an TopologyValidation interface implementation. It can be done through cache
configuration or through Ignite plugin extensions mechanism (see CacheTopologyValidatorProvider interface).

This module represents an implementation of the Ignite plugin that provides the guarantee that
after cluster segmentation, no more than one segment can process write requests to all caches. This is achieved by
providing implementation of the TopologyValidation interface as mentioned above.

The current implementation of TopologyValidation uses remaining Ignite baseline nodes in the topology to determine
segmentation.

#In what cases cache writes will be blocked for the segment?

The following rules are used to determine which segment can process cache write requests after segmentation and which
cannot:

1. The segment is allowed to process cache writes requests after segmentation if and only if more than configured
fraction of the baseline nodes remain in the segment, otherwise all writes to the cache will be blocked.
2. If the cluster is split into two equal segments, writing to both of them will be blocked.
3. Since Ignite treats segmentation as sequential node failures, even a single node failure in an active cluster in
which alive baseline nodes count is less or equals to segmentation threshold is considered as segmentation and results
in write block for all caches.

#Configuration

1. Configure CacheTopologyValidatorPluginProvider on each server node:

```
new IgniteConfiguration()
...
.setPluginProviders(new CacheTopologyValidatorPluginProvider());
```

2. Configure baseline nodes explicitly, or configure baseline nodes auto adjustment with a timeout that significantly
exceeds the node failure detection timeout. It can be done through Java Api or through control script.
See [1] and [2] for more info.

Note that it is illegal to use baseline nodes auto adjustment with a zero timeout along with current
TopologyValidator implementation.

3. Configure deactivation threshold.
The deactivation threshold is a fraction of nodes that determines how many nodes must remain in the baseline topology in
order to this segment was considered valid and continued to accept write requests.
This value must be in range from 0.5 (inclusively) to 1. Default value is 0.5. If the default value suits you, nothing
to do is required.

To set up custom deactivation threshold value set the `org.apache.ignite.topology.validator.deactivation.threshold`
distributed configuration property via control script (see https://ignite.apache.org/docs/latest/tools/control-script#working-with-cluster-properties)

#Manual segmentation resolving

The state of each segment for which cache writes were blocked will be eventually switched to the READ-ONLY mode.
Manually switching the cluster state back to ACTIVE mode will restore cache write availability. It can be done through
Java Api or through control script. See [1] and [2] for more info.

[1] - https://ignite.apache.org/docs/latest/clustering/baseline-topology \
[2] - https://ignite.apache.org/docs/latest/tools/control-script#activation-deactivation-and-topology-management
@@ -0,0 +1,40 @@
<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE log4j:configuration PUBLIC "-//APACHE//DTD LOG4J 1.2//EN"
"http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/xml/doc-files/log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">
<appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
<param name="Target" value="System.out"/>
<param name="Threshold" value="DEBUG"/>

<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="[%d{ISO8601}][%-5p][%t][%c{1}] %m%n"/>
</layout>

<filter class="org.apache.log4j.varia.LevelRangeFilter">
<param name="levelMin" value="DEBUG"/>
<param name="levelMax" value="INFO"/>
</filter>
</appender>

<appender name="CONSOLE_ERR" class="org.apache.log4j.ConsoleAppender">
<param name="Target" value="System.err"/>

<param name="Threshold" value="WARN"/>

<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="[%d{ISO8601}][%-5p][%t][%c{1}] %m%n"/>
</layout>
</appender>

<category name="org">
<level value="INFO"/>
</category>

<root>
<level value="INFO"/>

<appender-ref ref="CONSOLE"/>
<appender-ref ref="CONSOLE_ERR"/>
</root>
</log4j:configuration>
@@ -0,0 +1,67 @@
<?xml version="1.0" encoding="UTF-8"?>

<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

<!--
POM file.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-extensions-parent</artifactId>
<version>1</version>
<relativePath>../../parent</relativePath>
</parent>

<artifactId>ignite-topology-validator-ext</artifactId>
<version>1.0.0-SNAPSHOT</version>
<url>http://ignite.apache.org</url>

<dependencies>
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-core</artifactId>
<version>${ignite.version}</version>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-core</artifactId>
<version>${ignite.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-log4j</artifactId>
<version>${ignite.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-spring</artifactId>
<version>${ignite.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>

0 comments on commit aa2a168

Please sign in to comment.