Skip to content

Commit

Permalink
Feature to pass additional cluster settings from command line
Browse files Browse the repository at this point in the history 
Signed-off-by: Rishabh Singh <sngri@amazon.com>
  • Loading branch information
@rishabh6788
rishabh6788 committed Mar 25, 2023
1 parent cf335dc commit cd54212
Show file tree
Hide file tree
Showing 4 changed files with 56 additions and 31 deletions.
51 changes: 26 additions & 25 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,31 +32,32 @@ There are two stacks that get deployed:

In order to deploy both the stacks the user needs to provide a set of required and optional parameters listed below:

| Name | Type | Description |
|-----------------------------------|:--------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| distVersion (required) | string | The OpenSearch distribution version (released/un-released) the user wants to deploy |
| securityDisabled (required) | boolean | Enable or disable security plugin |
| minDistribution (required) | boolean | Is it the minimal OpenSearch distribution with no security and plugins |
| distributionUrl (required) | string | OpenSearch tar distribution url |
| cpuArch (required) | string | CPU platform for EC2, could be either `x64` or `arm64` |
| singleNodeCluster (required) | boolean | Set `true` for single-node cluster else `false` for multi-node |
| serverAccessType (required) | string | Restrict server access based on ip address (ipv4/ipv6), prefix list and/or security group. See [Restricting Server Access](#restricting-server-access) for more details. |
| restrictServerAccessTo (required) | string | The value for `serverAccessType`, e.g., 10.10.10.10/32, pl-12345, sg-12345. See [Restricting Server Access](#restricting-server-access) for more details. |
| dashboardsUrl (Optional) | string | OpenSearch Dashboards tar distribution url |
| vpcId (Optional) | string | Re-use existing vpc, provide vpc id |
| securityGroupId (Optional) | boolean | Re-use existing security group, provide security group id |
| cidr (Optional) | string | User provided CIDR block for new Vpc, default is `10.0.0.0/16` |
| managerNodeCount (Optional) | integer | Number of cluster manager nodes, default is 3 |
| dataNodeCount (Optional) | integer | Number of data nodes, default is 2 |
| clientNodeCount (Optional) | integer | Number of dedicated client nodes, default is 0 |
| ingestNodeCount (Optional) | integer | Number of dedicated ingest nodes, default is 0 |
| mlNodeCount (Optional) | integer | Number of dedicated machine learning nodes, default is 0 |
| jvmSysProps (Optional) | string | A comma-separated list of key=value pairs that will be added to `jvm.options` as JVM system properties. |
| suffix (Optional) | string | An optional string identifier to be concatenated with infra stack name. |
| region (Optional) | string | User provided aws region |
| account (Optional) | string | User provided aws account |
| dataNodeStorage (Optional) | string | User provided ebs block storage size, defaults to 100Gb |
| mlNodeStorage (Optional) | string | User provided ebs block storage size, defaults to 100Gb |
| Name | Type | Description |
|-----------------------------------|:--------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| distVersion (required) | string | The OpenSearch distribution version (released/un-released) the user wants to deploy |
| securityDisabled (required) | boolean | Enable or disable security plugin |
| minDistribution (required) | boolean | Is it the minimal OpenSearch distribution with no security and plugins |
| distributionUrl (required) | string | OpenSearch tar distribution url |
| cpuArch (required) | string | CPU platform for EC2, could be either `x64` or `arm64` |
| singleNodeCluster (required) | boolean | Set `true` for single-node cluster else `false` for multi-node |
| serverAccessType (required) | string | Restrict server access based on ip address (ipv4/ipv6), prefix list and/or security group. See [Restricting Server Access](#restricting-server-access) for more details. |
| restrictServerAccessTo (required) | string | The value for `serverAccessType`, e.g., 10.10.10.10/32, pl-12345, sg-12345. See [Restricting Server Access](#restricting-server-access) for more details. |
| dashboardsUrl (Optional) | string | OpenSearch Dashboards tar distribution url |
| vpcId (Optional) | string | Re-use existing vpc, provide vpc id |
| securityGroupId (Optional) | boolean | Re-use existing security group, provide security group id |
| cidr (Optional) | string | User provided CIDR block for new Vpc, default is `10.0.0.0/16` |
| managerNodeCount (Optional) | integer | Number of cluster manager nodes, default is 3 |
| dataNodeCount (Optional) | integer | Number of data nodes, default is 2 |
| clientNodeCount (Optional) | integer | Number of dedicated client nodes, default is 0 |
| ingestNodeCount (Optional) | integer | Number of dedicated ingest nodes, default is 0 |
| mlNodeCount (Optional) | integer | Number of dedicated machine learning nodes, default is 0 |
| jvmSysProps (Optional) | string | A comma-separated list of key=value pairs that will be added to `jvm.options` as JVM system properties. |
| additionalConfig (Optional) | string | Additional opensearch.yml config parameters passed as JSON. e.g., `--context additionalConfig='{"plugins.security.nodes_dn": ["CN=*.example.com, OU=SSL, O=Test, L=Test, C=DE", "CN=node.other.com, OU=SSL, O=Test, L=Test, C=DE"], "plugins.security.nodes_dn_dynamic_config_enabled": false}'` |
| suffix (Optional) | string | An optional string identifier to be concatenated with infra stack name. |
| region (Optional) | string | User provided aws region |
| account (Optional) | string | User provided aws account |
| dataNodeStorage (Optional) | string | User provided ebs block storage size, defaults to 100Gb |
| mlNodeStorage (Optional) | string | User provided ebs block storage size, defaults to 100Gb |


* Before starting this step, ensure that your AWS CLI is correctly configured with access credentials.
Expand Down
13 changes: 12 additions & 1 deletion lib/infra/infra-stack.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,8 @@ export interface infraProps extends StackProps{
readonly mlNodeCount: number,
readonly dataNodeStorage: number,
readonly mlNodeStorage: number,
readonly jvmSysPropsString?: string
readonly jvmSysPropsString?: string,
readonly additionalConfig?: string,
}

export class InfraStack extends Stack {
Expand Down Expand Up @@ -467,6 +468,16 @@ export class InfraStack extends Stack {
}));
}

// @ts-ignore
if (props.additionalConfig.toString() !== 'undefined') {
// @ts-ignore
cfnInitConfig.push(InitCommand.shellCommand(`set -ex; cd opensearch; echo "${props.additionalConfig}">>config/opensearch.yml`,
{
cwd: '/home/ec2-user',
ignoreErrors: false,
}));
}

// final run command based on whether the distribution type is min or bundle
if (props.minDistribution) { // using (stackProps.minDistribution) condition is not working when false value is being sent
cfnInitConfig.push(InitCommand.shellCommand('set -ex;cd opensearch; sudo -u ec2-user nohup ./bin/opensearch >> install.log 2>&1 &',
Expand Down
10 changes: 5 additions & 5 deletions lib/opensearch-config/node-config.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,26 +8,26 @@ compatible open source license. */
export const nodeConfig = new Map<string, object>();

nodeConfig.set('manager', {
'node.roles': [ 'cluster_manager' ]
'node.roles': ['cluster_manager'],
});

nodeConfig.set('data', {
'node.roles': [ 'data', 'ingest' ]
'node.roles': ['data', 'ingest'],
});

nodeConfig.set('seed-manager', {
'node.name': 'seed',
'node.roles': [ 'cluster_manager' ]
'node.roles': ['cluster_manager'],
});

nodeConfig.set('seed-data', {
'node.name': 'seed',
'node.roles': [ 'cluster_manager', 'data' ]
'node.roles': ['cluster_manager', 'data'],
});

nodeConfig.set('client', {
'node.name': 'client-node',
'node.roles': []
'node.roles': [],
});

nodeConfig.set('ml', {
Expand Down
13 changes: 13 additions & 0 deletions lib/os-cluster-entrypoint.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { Stack, StackProps } from 'aws-cdk-lib';
import {
AmazonLinuxCpuType, IVpc, SecurityGroup, Vpc,
} from 'aws-cdk-lib/aws-ec2';
import { dump } from 'js-yaml';
import { NetworkStack } from './networking/vpc-stack';
import { InfraStack } from './infra/infra-stack';

Expand All @@ -35,6 +36,7 @@ export class OsClusterEntrypoint {
let infraStackName: string;
let dataNodeStorage: number;
let mlNodeStorage: number;
let ymlConfig: string = 'undefined';

const vpcId: string = scope.node.tryGetContext('vpcId');
const securityGroupId = scope.node.tryGetContext('securityGroupId');
Expand Down Expand Up @@ -134,6 +136,16 @@ export class OsClusterEntrypoint {

const jvmSysProps = `${scope.node.tryGetContext('jvmSysProps')}`;

const osConfig = `${scope.node.tryGetContext('additionalConfig')}`;
if (osConfig.toString() !== 'undefined') {
try {
const jsonObj = JSON.parse(osConfig);
ymlConfig = dump(jsonObj);
} catch (e) {
throw new Error(`Encountered following error parsing json: ${e}`);
}
}

const suffix = `${scope.node.tryGetContext('suffix')}`;

const network = new NetworkStack(scope, 'opensearch-network-stack', {
Expand Down Expand Up @@ -179,6 +191,7 @@ export class OsClusterEntrypoint {
dataNodeStorage,
mlNodeStorage,
jvmSysPropsString: jvmSysProps,
additionalConfig: ymlConfig,
...props,
});

Expand Down

0 comments on commit cd54212

Please sign in to comment.