This readme defines the Command and control configuration options that work with Apache NiFi. All options defined are located in minifi.properties.
Apache NiFi MiNiFI C++ can communicates with a C2 Server via a number of protocols. These protocols deliver a C2 response the server, expecting requests in a hearbeat response. The protocol transforms the C2 messages into a protocol specific representation. The internal representation is an AST therefore you must define the root classes, which configure the classes that branch from the root. You can define arbitrary nodes and sub-trees, but this isn't necessary and only advantageous for custom C2 servers. That will be explained in greater detail in the metrics section.
For more more insight into the API used within the C2 agent, please visit: https://cwiki.apache.org/confluence/display/MINIFI/C2+Design+Proposal
in minifi.properties
# Disable/Enable C2
nifi.c2.enable=true
# specify classes for the AST response
nifi.c2.root.classes=DeviceInfoNode,AgentInformation,FlowInformation
# specify C2 protocol -- default is RESTSender if this is not specified
c2.agent.protocol.class=RESTSender
# may also use MQTT
# c2.agent.protocol.class=MQTTC2Protocol
# control c2 heartbeat interval in millisecocnds
c2.agent.heartbeat.period=3000
# enable reporter classes
c2.agent.heartbeat.reporter.class=RESTReciver
# specify the rest URIs if using RESTSender
c2.rest.url=http://localhost:10080/minifi-c2-api/c2-protocol/heartbeat
c2.rest.url.ack=http://localhost:10080/minifi-c2-api/c2-protocol/acknowledge
# c2 agent identifier
nifi.c2.agent.identifier=<your identifier>
# c2 agent class
nifi.c2.agent.class=<your agent class>
# configure SSL Context service for REST Protocol
c2.rest.ssl.context.service
Command and Control metrics can be used to send metrics through the heartbeat or via the DESCRIBE operation. Since responses are formed in an AST, metrics can be formed as a sub tree. Metrics classes are defined apriori and may reference a metrics class specific to a processor. The following example describes a configuration of an agent
# in minifi.properties
nifi.c2.root.class.definitions=metrics
nifi.c2.root.class.definitions.metrics.name=metrics
nifi.c2.root.class.definitions.metrics.metrics=typedmetrics,processorMetrics
nifi.c2.root.class.definitions.metrics.metrics.typedmetrics.name=RuntimeMetrics
nifi.c2.root.class.definitions.metrics.metrics.typedmetrics.classes=ProcessMetrics,SystemInformation
nifi.c2.root.class.definitions.metrics.metrics.processorMetrics.name=ProcessorMetrics
nifi.c2.root.class.definitions.metrics.metrics.processorMetrics.classes=GetFileMetrics
This example shows a metrics sub tree defined by the option 'nifi.c2.root.class.definitions'.
This is a comma separated list of all sub trees. In the example, above, only one sub tree exists: metrics.
The options below metrics define the sub-trees within metrics: typedmetrics and processorMetrics. Each of these has a name. The classes sub option will define the metrics classes that are placed within this sub-tree. For the RESTProtocol, the above configuration produces the following JSON:
"metrics": {
"ProcessorMetrics": {
"GetFileMetrics": {
"AcceptedFiles": 22,
"InputBytes": 61755,
"OnTriggerInvocations": 1
}
},
"RuntimeMetrics": {
"ProcessMetrics": {
"CpuMetrics": {
"involcs": 1
},
"MemoryMetrics": {
"maxrss": 145804
}
},
"systeminfo": {
"systemInfo": {
"machinearch": "x86_64",
"physicalMem": 67361411072,
"vCores": 12
},
"identifier": "identifier"
}
}
}
The default protocol is a RESTFul service; however, there is an MQTT protocol with a translation to use the RESTFul C2 server. This is useful for cases where an MQTT C2 server isn't available, or enclave partioning requires a single ingress/egress through a gateway. In these classes of devices, MQTT can be used as the intermediate or RESTSender can be used for C2 operations.
As defined, above, MQTTC2Protocol can be used for the agent protocol class. If you wish to communicate with a RESTFul C2 server, you may use the ConvertBase, ConvertHeartBeat, ConvertJSONAack, and ConvertUpdate classes on an agent to perform the transation. State is not kept with an intermediate agent other than the broker. The broker is not embedded with the agent to simplify the agent.
An example configuration, below, defines an agent that receives and forward MQTT C2 requests to a C2 server. Additionally, this agent will forward responses and updates to the heartbeating agents.
MiNiFi Config Version: 3
Flow Controller:
name: GetFile
Core Properties:
flow controller graceful shutdown period: 10 sec
flow service write delay interval: 500 ms
administrative yield duration: 30 sec
bored yield duration: 10 millis
max concurrent threads: 1
variable registry properties: ''
FlowFile Repository:
partitions: 256
checkpoint interval: 2 mins
always sync: false
Swap:
threshold: 20000
in period: 5 sec
in threads: 1
out period: 5 sec
out threads: 4
Content Repository:
content claim max appendable size: 10 MB
content claim max flow files: 100
always sync: false
Provenance Repository:
provenance rollover time: 1 min
Component Status Repository:
buffer size: 1440
snapshot frequency: 1 min
Security Properties:
keystore: ''
keystore type: ''
keystore password: ''
key password: ''
truststore: ''
truststore type: ''
truststore password: ''
ssl protocol: ''
Sensitive Props:
key:
algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
provider: BC
Processors:
- id: 24493a28-015a-1000-0000-000000000000
name: convert
class: org.apache.nifi.processors.standard.ConvertHeartBeat
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 10 msec
penalization period: 30 sec
yield period: 2 sec
run duration nanos: 10 msec
auto-terminated relationships list:
Properties:
MQTT Controller Service: mqttservice
Listening Topic: heartbeats
- id: 24493a28-015a-1000-0000-000000000006
name: convertJSON
class: org.apache.nifi.processors.standard.ConvertJSONAck
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 10 msec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 10 msec
auto-terminated relationships list:
- success
Properties:
MQTT Controller Service: mqttservice
- id: 24493a28-015a-1000-0000-000000000007
name: convertupdate
class: org.apache.nifi.processors.standard.ConvertUpdate
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 10 msec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 10 msec
auto-terminated relationships list:
- success
Properties:
MQTT Controller Service: mqttservice
Listening Topic: updates
- id: 24493a28-015a-1000-0000-000000000021
name: httpheartbeat
class: org.apache.nifi.processors.standard.InvokeHTTP
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 10 msec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 10 msec
auto-terminated relationships list:
Properties:
HTTP Method: POST
Remote URL: http://localhost:10080/minifi-c2-api/c2-protocol/heartbeat
Content-type: application/json
- id: 24493a28-015a-1000-0000-000000000022
name: log
class: org.apache.nifi.processors.standard.LogAttribute
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 100 msec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 1 msec
auto-terminated relationships list:
- success
Properties:
Controller Services:
- id: 94491a38-015a-1000-0000-000000000001
name: mqttservice
class: MQTTContextService
Properties:
Broker URI: localhost:1883
Client ID: hiblajl
Quality of Service: 2
Process Groups: []
Input Ports: []
Output Ports: []
Funnels: []
Connections:
- id: 1d09c015-015d-1000-0000-000000000000
name: convert/success/httpheartbeat
source id: 24493a28-015a-1000-0000-000000000000
source relationship name: success
destination id: 24493a28-015a-1000-0000-000000000021
max work queue size: 10000
max work queue data size: 1 GB
flowfile expiration: 0 sec
queue prioritizer class: org.apache.nifi.prioritizer.FirstInFirstOutPrioritizer
- id: 1d09c015-015d-1000-0000-000000000002
name: httpheartbeat/success/convertJSON
source id: 24493a28-015a-1000-0000-000000000021
source relationship name: success
destination id: 24493a28-015a-1000-0000-000000000006
max work queue size: 10000
max work queue data size: 1 GB
flowfile expiration: 0 sec
queue prioritizer class: org.apache.nifi.prioritizer.FirstInFirstOutPrioritizer
Remote Process Groups: []
NiFi Properties Overrides: {}
Updates to MiNiFi C++ properties can be controlled through an UpdatePolicyControllerService named C2UpdatePolicy. The service supports several configuration options. They are defined in the following example:
Controller Services:
- id: 94491a38-015a-1000-0000-000000000001
name: C2UpdatePolicy
class: UpdatePolicyControllerService
Properties:
# true enables all properties to be updated.
Allow All Properties: true
# allowed properties are those which can be updated
Allowed Properties:
Property_1:true
Property_2:true
Disallowed Properties:
Property_3:true
Property_4:true