Troubleshooting
This section contains information about how to monitor the system and how to troubleshoot specific issues that may occur.
This section contains information about where information can be found within RES.
There are various sources of information retained that can be referenced for troubleshooting and monitoring uses.
Log files exist on the EC2 instances in use by RES. The SSM Session Manager can be used to open a session to the instance for examining these files.
On infrastructure instances such as the cluster-manager and vdc-controller, application and other logs can be found at the following locations.
- /opt/idea/app/logs/application.log
- /root/bootstrap/logs/
- /var/log/
- /var/log/sssd/
- /var/log/messages
- /var/log/user-data.log
- /var/log/cloud-init.log
- /var/log/cloud-init-output.log
On a Linux virtual desktop, the following contain useful log files
- /var/log/dcv/
- /root/bootstrap/logs/userdata.log
- /var/log/messages
On Windows virtual desktop instances logs can be found at
- PS C:\ProgramData\nice\dcv\log
- PS C:\ProgramData\nice\DCVSessionManagerAgent\log
On Windows, some applications logging can be found at:
- PS C:\Program Files\NICE\DCV\Server\bin
On Windows, the NICE DCV certificate files can be found in:
- C:\Windows\System32\config\systemprofile\AppData\Local\NICE\dcv\
CloudWatch Log Groups
The EC2 and Lambda compute resources will log information to CloudWatch Log Groups. The log entries within them can provide useful information when troubleshooting potential issues or for general information.
Those groups will be named as follows:
- /aws/lambda/-/ - lambda related
- //
- analytics/ - OpenSearch database related
- cluster-manager/ - main infrastructure host
- vdc/ - virtual desktop related
- dcv-broker/ - desktop related
- dcv-connection-gateway/ - desktop related
- controller/ - main desktop controller host
- dcv-session/ - desktop session related
When examining log groups, it can be helpful to filter using upper and lower case strings such as the following. This will output only those messages containing the noted strings.
?"ERROR" ?"error"
Another method of monitoring for issues is to create CloudWatch Dashboards that contain widgets displaying the data of interest.
An example is to create a widget that counts the occurrence of the strings error and ERROR and graph them as lines. This method makes it easier to detect the occurrence of potential issues or trends indicating a pattern change has occurred.
The following is an example of that for the infrastructure hosts. To use this, concatenate the query lines and replace the and attributes with the appropriate values.
{
"widgets": [
{
"type": "log",
"x": 0,
"y": 0,
"width": 24,
"height": 6,
"properties": {
"query": "SOURCE '/<envname>/vdc/controller' |
SOURCE '/<envname>/cluster-manager' |
SOURCE '/<envname>/vdc/dcv-broker' |
SOURCE '/<envname>/vdc/dcv-connection-gateway' |
fields @timestamp, @message, @logStream, @log\n|
filter @message like /(?i)(error|ERROR)/\n|
sort @timestamp desc|
stats count() by bin(30s)",
"region": "<region>",
"title": "infrastructure hosts",
"view": "timeSeries",
"stacked": false
}
}
]
}
An example of the Dashboard might appear as follows.
The CloudFormation stacks created during environment creation contain resources, event, and output information associated with the configuration of the environment. For each of the stacks, the Events, Resources, and Outputs tab can be referred to for information about the stacks.
Batteries Included (BI) stacks
- -Networking
- -DirectoryService
- -Storage
- -WindowsManagementHost
RES stacks
- -bootstrap
- -cluster
- -metrics
- -directoryservice
- -identity-provider
- -analytics
- -shared-storage
- -cluster-manager
- -vdc
- -bastion-host
If the RES UIs are indicating server errors, the cause may be an application software or other issue.
Each of the infrastructure EC2 instance autoscaling groups (ASGs) contain an Activity tab that can be useful for detecting scaling activity for the instances. If UI pages note any errors or are not accessible, check the EC2 console for multiple terminated instances and check the Auto Scaling Group Activity tab for the related ASG to determine if EC2 instances are cycling.
If so, use the related CloudWatch log group for the instance to determine if errors are being logged that might indicate the cause of the issue. It may also be possible to use the SSM Session console to open a session to a running instance of that type and examine the log files on the instance to determine a cause before the instance is marked as unhealthy and terminated by the ASG.
The ASG console may show activity similar to the following if this issue is occurring.
This section contains screenshots of the system operating in various states.
The EC2 console, when no desktops are running, will typically look similar to the following. The instances that are show are the RES infrastructure EC2 hosts. The prefix in the name of the instances will be the RES environment name.
The EC2 console when virtual desktops are running, they will appear similar to the following. In this case, the virtual desktops are noted in red. The suffix to the instance name will be the user that created the desktop. The name in the center will be the Session Name set at launch time and will be either be the default MyDesktop or the name set by the user.
When the EC2 console shows terminated instances, they will generally be desktop hosts that have been terminated. If the console includes infrastructure hosts in a terminated state, particularly if there are multiple of the same type, that may indicate a system issue in progress.
The following show desktop instances that have been terminated.
The following are examples of ldap related commands that can be entered on infrastructure hosts to view AD configuration related information. The domain and other parameters used should reflect those entered at environment creation time.
ldapsearch "(cn=AWS Delegated Add Workstations To Domain Users)" -x -h corp.res.com -b "DC=corp,DC=res,DC=com" -D "CN=Admin,OU=Users,OU=CORP,DC=corp,DC=res,DC=com" -w
ldapsearch "(&(objectClass=group))" -x -h corp.res.com -b "DC=corp,DC=res,DC=com" -D "CN=Admin,OU=Users,OU=CORP,DC=corp,DC=res,DC=com" -w
On a Windows desktop, list the session associated with it using the following
PS C:\Windows\System32\config\systemprofile\AppData\Local\NICE\dcv> & 'C:\Program Files\NICE\DCV\Server\bin\dcv.exe'list-sessions Session: 'a7953489-9dbf-492b-8135-7709dccc4cab' (owner:admin2 type:console name:windows1)
NICE DCV is utilized for virtual desktop sessions. https://aws.amazon.com/hpc/dcv/. The following are examples of determining the version of the DCV software installed.
Linux
[root@ip-10-3-157-194 ~]# /usr/bin/dcv version
NICE DCV 2023.0 (r14852) Copyright (C) 2010-2023 NICE s.r.l. All rights reserved.
This product is protected by copyright and licenses restricting use, copying, distribution, and decompilation.
Windows
PS C:\Windows\System32\config\systemprofile\AppData\Local\NICE\dcv> & 'C:\Program Files\NICE\DCV\Server\bin\dcv.exe' version
NICE DCV 2023.0 (r15065) Copyright (C) 2010-2023 NICE s.r.l. All rights reserved.
This product is protected by copyright and licenses restricting use, copying, distribution, and decompilation.