Troubleshooting
This section contains information about how to monitor the system and how to troubleshoot specific issues that may occur.
This section contains information about where information can be found within RES.
There are various sources of information retained that can be referenced for troubleshooting and monitoring uses.
Log files exist on the EC2 instances in use by RES. The SSM Session Manager can be used to open a session to the instance for examining these files.
On infrastructure instances such as the cluster-manager and vdc-controller, application and other logs can be found at the following locations.
- /opt/idea/app/logs/application.log
- /root/bootstrap/logs/
- /var/log/
- /var/log/sssd/
- /var/log/messages
- /var/log/user-data.log
- /var/log/cloud-init.log
- /var/log/cloud-init-output.log
On a Linux virtual desktop, the following contain useful log files
- /var/log/dcv/
- /root/bootstrap/logs/userdata.log
- /var/log/messages
On Windows virtual desktop instances logs can be found at
- PS C:\ProgramData\nice\dcv\log
- PS C:\ProgramData\nice\DCVSessionManagerAgent\log
On Windows, some applications logging can be found at:
- PS C:\Program Files\NICE\DCV\Server\bin
On Windows, the NICE DCV certificate files can be found in:
- C:\Windows\System32\config\systemprofile\AppData\Local\NICE\dcv\
CloudWatch Log Groups
The EC2 and Lambda compute resources will log information to CloudWatch Log Groups. The log entries within them can provide useful information when troubleshooting potential issues or for general information.
Those groups will be named as follows:
- /aws/lambda/< envname >-/ - lambda related
- /< envname >/
- analytics/ - OpenSearch database related
- cluster-manager/ - main infrastructure host
- vdc/ - virtual desktop related
- dcv-broker/ - desktop related
- dcv-connection-gateway/ - desktop related
- controller/ - main desktop controller host
- dcv-session/ - desktop session related
When examining log groups, it can be helpful to filter using upper and lower case strings such as the following. This will output only those messages containing the noted strings.
?"ERROR" ?"error"
Another method of monitoring for issues is to create CloudWatch Dashboards that contain widgets displaying the data of interest.
An example is to create a widget that counts the occurrence of the strings error and ERROR and graph them as lines. This method makes it easier to detect the occurrence of potential issues or trends indicating a pattern change has occurred.
The following is an example of that for the infrastructure hosts. To use this, concatenate the query lines and replace the < envname > and < region > attributes with the appropriate values.
{
"widgets": [
{
"type": "log",
"x": 0,
"y": 0,
"width": 24,
"height": 6,
"properties": {
"query": "SOURCE '/< envname >/vdc/controller' |
SOURCE '/< envname >/cluster-manager' |
SOURCE '/< envname >/vdc/dcv-broker' |
SOURCE '/< envname >/vdc/dcv-connection-gateway' |
fields @timestamp, @message, @logStream, @log\n|
filter @message like /(?i)(error|ERROR)/\n|
sort @timestamp desc|
stats count() by bin(30s)",
"region": "< region >",
"title": "infrastructure hosts",
"view": "timeSeries",
"stacked": false
}
}
]
}
An example of the Dashboard might appear as follows.
The CloudFormation stacks created during environment creation contain resources, event, and output information associated with the configuration of the environment. For each of the stacks, the Events, Resources, and Outputs tab can be referred to for information about the stacks.
Batteries Included (BI) stacks
- < envname >
- < envname >-Networking
- < envname >-DirectoryService
- < envname >-Storage
- < envname >-WindowsManagementHost
RES stacks
- < envname >-bootstrap
- < envname >-cluster
- < envname >-metrics
- < envname >-directoryservice
- < envname >-identity-provider
- < envname >-analytics
- < envname >-shared-storage
- < envname >-cluster-manager
- < envname >-vdc
- < envname >-bastion-host
If the RES UIs are indicating server errors, the cause may be an application software or other issue.
Each of the infrastructure EC2 instance autoscaling groups (ASGs) contain an Activity tab that can be useful for detecting scaling activity for the instances. If UI pages note any errors or are not accessible, check the EC2 console for multiple terminated instances and check the Auto Scaling Group Activity tab for the related ASG to determine if EC2 instances are cycling.
If so, use the related CloudWatch log group for the instance to determine if errors are being logged that might indicate the cause of the issue. It may also be possible to use the SSM Session console to open a session to a running instance of that type and examine the log files on the instance to determine a cause before the instance is marked as unhealthy and terminated by the ASG.
The ASG console may show activity similar to the following if this issue is occurring.
This section contains screenshots of the system operating in various states.
The EC2 console, when no desktops are running, will typically look similar to the following. The instances that are show are the RES infrastructure EC2 hosts. The prefix in the name of the instances will be the RES environment name.
The EC2 console when virtual desktops are running, they will appear similar to the following. In this case, the virtual desktops are noted in red. The suffix to the instance name will be the user that created the desktop. The name in the center will be the Session Name set at launch time and will be either be the default MyDesktop or the name set by the user.
When the EC2 console shows terminated instances, they will generally be desktop hosts that have been terminated. If the console includes infrastructure hosts in a terminated state, particularly if there are multiple of the same type, that may indicate a system issue in progress.
The following show desktop instances that have been terminated.
The following are examples of ldap related commands that can be entered on infrastructure hosts to view AD configuration related information. The domain and other parameters used should reflect those entered at environment creation time.
ldapsearch "(cn=AWS Delegated Add Workstations To Domain Users)" -x -h corp.res.com
-b "DC=corp,DC=res,DC=com" -D "CN=Admin,OU=Users,OU=CORP,DC=corp,DC=res,DC=com"
-w <password>
ldapsearch "(&(objectClass=group))" -x -h corp.res.com
-b "DC=corp,DC=res,DC=com" -D "CN=Admin,OU=Users,OU=CORP,DC=corp,DC=res,DC=com"
-w <password>
On a Windows desktop, list the session associated with it using the following
PS C:\Windows\System32\config\systemprofile\AppData\Local\NICE\dcv> & 'C:\Program Files\NICE\DCV\Server\bin\dcv.exe'list-sessions
Session: 'a7953489-9dbf-492b-8135-7709dccc4cab' (owner:admin2 type:console name:windows1)
NICE DCV is utilized for virtual desktop sessions. https://aws.amazon.com/hpc/dcv/. The following are examples of determining the version of the DCV software installed.
Linux
[root@ip-10-3-157-194 ~]# /usr/bin/dcv version
NICE DCV 2023.0 (r14852)
Copyright (C) 2010-2023 NICE s.r.l.
All rights reserved.
This product is protected by copyright and
licenses restricting use, copying, distribution, and decompilation.
Windows
PS C:\Windows\System32\config\systemprofile\AppData\Local\NICE\dcv> & 'C:\Program Files\NICE\DCV\Server\bin\dcv.exe' version
NICE DCV 2023.0 (r15065)
Copyright (C) 2010-2023 NICE s.r.l.
All rights reserved.
This product is protected by copyright and
licenses restricting use, copying, distribution, and decompilation.
The following section contains issues that may occur, how to detect them, and suggestions on how to resolve the issue.
If you receive an error that you're not authorized to perform the iam:PassRole action, your policies must be updated to allow you to pass a role to RES.
Some AWS services allow you to pass an existing role to that service instead of creating a new service role or service-linked role. To do this, you must have permissions to pass the role to the service.
The following example error occurs when an IAM user named marymajor tries to use the console to perform an action in RES. However, the action requires the service to have permissions that are granted by a service role. Mary does not have permissions to pass the role to the service.
User: arn:aws:iam::123456789012:user/marymajor is not authorized to perform: iam:PassRole
In this case, Mary's policies must be updated to allow her to perform the iam:PassRole action. If you need help, contact your AWS administrator. Your administrator is the person who provided you with your sign-in credentials.
I want to allow people outside of my AWS account to access my Research and Engineering Studio on AWS resources
You can create a role that users in other accounts or people outside of your organization can use to access your resources. You can specify who is trusted to assume the role. For services that support resource-based policies or access control lists (ACLs), you can use those policies to grant people access to your resources.
To learn more, consult the following:
- To learn how to provide access to your resources across AWS accounts that you own, see Providing access to an IAM user in another AWS account that you own in the IAM User Guide.
- To learn how to provide access to your resources to third-party AWS accounts, see Providing access to AWS accounts owned by third parties in the IAM User Guide.
- To learn how to provide access through identity federation, see Providing access to externally authenticated users (identity federation) in the IAM User Guide.
- To learn the difference between using roles and resource-based policies for cross-account access, see How IAM roles differ from resource-based policies in the IAM User Guide.
The file systems need to be in “Available” state before they can be mounted by VDI hosts. Follow steps below to validate file system is in needed state
EFS
- Go to EFS console [us-east-1]
- Check File system state is Available
- If File system state is not Available, wait before launching VDI hosts
FSx ONTAP
- Go to FSx console [us-east-1]
- Check Status is Available
- If Status is not Available, wait before launching VDI hosts
The file systems onboarded on RES should have the needed security group rules configured to allow VDI hosts to mount the file systems. As these file systems are created externally to RES, RES doesn’t manage the associated security group rules.
The security group associated with the onboarded file systems should allow following inbound traffic
- NFS traffic (port: 2049) from the linux VDC hosts
- SMB traffic (port: 445) from the windows VDC hosts
ONTAP supports UNIX, NTFS and MIXED security style for the volumes. The security styles determine the type of permissions ONTAP uses to control data access and what client type can modify these permissions.
For example, if a volume uses UNIX security style, SMB clients can still access data (provided that they properly authenticate and authorize) due to the multi-protocol nature of ONTAP. However, ONTAP uses UNIX permissions that only UNIX clients can modify using native tools.
Using UNIX style volume with Linux workloads
Permissions can be configured by the sudoer for other users. For eg. following would give all members of <group-ID> full read/write permissions on /<project-name> directory
sudo chown root:<group-ID> /<project-name>
sudo chmod 770 /<project-name>
Using NTFS style volume with Linux and Windows workloads
Permissions can be configured on the file share using share properties. For eg. following would give pentest_user_01 full control permissions on the file share myshare.
If the volume is going to be used by both Linux and Windows client we need to set up a name mapping on SVM that will associate any Linux user name to the same user name with the NetBIOS domain name format of domain\username. This is needed to translate between Linux and Windows users. For reference see Enabling multiprotocol workloads with Amazon FSx for NetApp ONTAP
Currently, if you are creating FSx for NetApp ONTAP from RES console, the file system gets provisioned but it does not join the domain. To join the created ONTAP file system SVM to your domain, please see here Joining SVMs to a Microsoft Active Directory and follow the steps on Amazon FSx console [us-east-1]. Once the SVM joins the domain successfully, go to SVM Summary > Endpoints > SMB DNS name, copy the DNS name as it will be needed later.
After joining to domain, edit the SMB DNS config key in cluster settings DynamoDB table:
- Go to Amazon DynamoDB console [us-east-1] 2.** Tables >** select -cluster-settings
-
Explore table items > expand Filters, and put following filter
- Attribute name - key
- Condition - Equal to
- Value - shared-storage..fsx_netapp_ontap.svm.smb_dns
- Select the returned item > Actions > Edit item
- Update the value to string value (SMB DNS name copied earlier) 6.** Save and close**
Additionally, ensure security group associated with the file system allows traffic as recommended on File System Access Control with Amazon VPC. New VDI hosts using the file system will now be able to mount the domain joined SVM and file system.
Alternative, you may onboard existing file system which is already joined to your domain using RES Onboard File System capability from Environment Management > File Systems > Onboard File System
This situation indicates a misconfiguration of your SSO integration. To determine the issue, check the logs for the controller instance. Do this by:
- See logs
- Opening Log groups section of the AWS CloudWatch Console
- Search for “cluster-manager” and select the one that corresponds to your environment.
- Look for any errors that show up in the logs
- See SSO configuration settings, to know if there was a mistake
- AWS Console → DynamoDB
- Cluster settings table for the corresponding env
- Explore table data
- Filters → key = key, condition = contains, value = sso
- Run
- Verify that the sso configuration values are correct
- If wrong, manually set the sso_enabled key’s value to false in DDB
- Go back to the UI and you will be able to re-configure SSO