[Bug]: All integration tests for RPM, Debian and Windows are failing

[gaiksaya]

Describe the bug

Note: This is an observation for OpenSearch only. OSD might be a different story
Recently so many changes went in related to security plugin as well as in general. We enabled integration testing to be triggered via distribution pipeline for all distributions. (Earlier it was just running for tarball)

To reproduce

All windows failed: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/7136/pipeline
All debian failed: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/7135/pipeline
All RPM failed: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/7134/pipeline

Expected behavior

All tests should run. The infrastructure and test set up should be fixed.

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[peterzhuamazon]

Will check soon, most likely the deb/rpm is due to sudo permission not added to the ci-runner images.

PRs:

• Fix deb/rpm integTest not install distribution correctly due to 2.12 security changes #4376
• Improve bash checking in test_finder.sh for all OSes opensearch-dashboards-functional-test#1058
• Windows integTest fix for 2.12 with admin:admin security changes #4409
• Disable remote cluster test settings added in 998 PR of ftrepo #4415
• Disable video recording in Windows due to ffmpeg crash opensearch-dashboards-functional-test#1069
• [2.x] Adding placeholder to keep main/2.x/1.x consistent before #988 backports and disable windows video recording opensearch-dashboards-functional-test#1070
• [Backport 1.3] [2.x] Adding placeholder to keep main/2.x/1.x consistent before #988 backports and disable windows video recording opensearch-dashboards-functional-test#1071
• Install fonts on windows docker image for chromium-based browser and lock docker version #4417
• Switch ftrepo checkout based on ref instead of commit_id #4419
• Remove BROWSER_PATH=electron for Windows so integTest can run chromium114 #4457
• Switch windows/servercore to windows image as base to resolve multiple lib missing issues #4458

[peterzhuamazon]

k-NN: Test with @junqiu-lei on tar for concurrent search, resolved opensearch-project/k-NN#1419.

[peterzhuamazon]

Resolve rpm:

2024-01-26 23:15:19 INFO     rpm installation requires sudo, script will exit if current user does not have sudo access
LANG=C.UTF-8
HOSTNAME=1c85fa7fa882
TERM=xterm
MAIL=/var/mail/root
PATH=/sbin:/bin:/usr/sbin:/usr/bin
LOGNAME=root
USER=root
HOME=/root
SHELL=/bin/bash
SUDO_COMMAND=/bin/env OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123!
SUDO_USER=ci-runner
SUDO_UID=1000
SUDO_GID=1000
XDG_SESSION_ID=c1
XDG_RUNTIME_DIR=/run/user/0
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123!
No match for argument: opensearch
No packages marked for removal.
Dependencies resolved.
Nothing to do.
Complete!
Rocky Linux 8 - AppStream                                                                                                                                                                                                                                                                   7.1 MB/s |  10 MB     00:01
Rocky Linux 8 - BaseOS                                                                                                                                                                                                                                                                      4.6 MB/s | 5.1 MB     00:01
Rocky Linux 8 - Extras                                                                                                                                                                                                                                                                       29 kB/s |  14 kB     00:00
Extra Packages for Enterprise Linux 8 - x86_64                                                                                                                                                                                                                                              8.6 MB/s |  16 MB     00:01
packages for the GitHub CLI                                                                                                                                                                                                                                                                  15 kB/s | 2.6 kB     00:00
Dependencies resolved.
============================================================================================================================================================================================================================================================================================================================
 Package                                                                       Architecture                                                              Version                                                                      Repository                                                                       Size
============================================================================================================================================================================================================================================================================================================================
Installing:
 opensearch                                                                    x86_64                                                                    2.12.0-1                                                                     @commandline                                                                    660 M

Transaction Summary
============================================================================================================================================================================================================================================================================================================================
Install  1 Package

Total size: 660 M
Installed size: 931 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.

[peterzhuamazon]

Removed:
  opensearch-2.12.0-1.x86_64

Complete!
2024-01-26 23:33:17 INFO     Recording component test results for geospatial at /home/ci-runner/opensearch-build/test-results/57995af2c3444c6d86f67ce3fbad526e/integ-test/geospatial/with-security
2024-01-26 23:33:17 INFO     Walking tree from /home/ci-runner/opensearch-build/test-results/57995af2c3444c6d86f67ce3fbad526e/integ-test/geospatial/with-security
2024-01-26 23:33:17 INFO     Recording component test results for geospatial at /home/ci-runner/opensearch-build/test-results/57995af2c3444c6d86f67ce3fbad526e/integ-test/geospatial/without-security
2024-01-26 23:33:17 INFO     Walking tree from /home/ci-runner/opensearch-build/test-results/57995af2c3444c6d86f67ce3fbad526e/integ-test/geospatial/without-security
2024-01-26 23:33:17 INFO     Skipping ml-commons
2024-01-26 23:33:17 INFO     Skipping neural-search
2024-01-26 23:33:17 INFO     Skipping notifications-core
2024-01-26 23:33:17 INFO     Skipping notifications
2024-01-26 23:33:17 INFO     Skipping opensearch-observability
2024-01-26 23:33:17 INFO     Skipping anomaly-detection
2024-01-26 23:33:17 INFO     Skipping alerting
2024-01-26 23:33:17 INFO     Skipping security-analytics
2024-01-26 23:33:17 INFO     Skipping index-management
2024-01-26 23:33:17 INFO     Skipping flow-framework
2024-01-26 23:33:17 INFO     Removing /tmp/tmpun19ylg0
2024-01-26 23:33:17 INFO     | geospatial           | with-security        | PASS  |
2024-01-26 23:33:17 INFO     | geospatial           | without-security     | PASS  |

[peterzhuamazon]

Today I learn that bash shell inside the windows container using git bash is actually giving

$ echo $SHELL
/sbin/nologin

causing failures due to ftrepo specifically checking if it is a bash shell……

[peterzhuamazon]

More changes needed due to this:

• Add remote cypress orchestrator integration with integtest script opensearch-dashboards-functional-test#988

More issues:

• [BUG] Windows Docker Container failed with ffmpeg missing libs #4427
• [BUG] Windows Docker Container failed with Chromium based browser due to missing fonts #4416

[peterzhuamazon]

IntegTest should be fully fixed now on Windows.

[gaiksaya]

Windows looks good too now: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/7681/pipeline
Thanks @peterzhuamazon 🙌

[peterzhuamazon]

See issues:
https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/5530/pipeline/
x64, linux, osd-2.12.0-7254, os-2.12.0-9373, rpm

https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test-opensearch-dashboards/detail/integ-test-opensearch-dashboards/5527/pipeline/
arm64, linux, osd-2.12.0-7254, os-2.12.0-9373, tar

{"type":"log","@timestamp":"2024-02-09T18:57:36Z","tags":["fatal","root"],"pid":521,"message":"Error: Cannot find module '../../types'\nRequire stack:\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/utils/constants.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/clusters/createSecurityAnalyticsCluster.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/plugin.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/index.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/plugin.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/discovery/plugins_discovery.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/discovery/index.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/plugins_service.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/index.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/index.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/cli/serve/serve.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/cli/cli.js\n- /tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/cli/dist.js\n at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1075:15)\n at Module.Hook._require.Module.require (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/node_modules/require-in-the-middle/index.js:61:29)\n at require (node:internal/modules/cjs/helpers:110:18)\n at Object.<anonymous> (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/utils/constants.ts:6:1)\n at Module._compile (node:internal/modules/cjs/loader:1254:14)\n at Object.Module._extensions..js (node:internal/modules/cjs/loader:1308:10)\n at Module.load (node:internal/modules/cjs/loader:1117:32)\n at Function.Module._load (node:internal/modules/cjs/loader:958:12)\n at Module.require (node:internal/modules/cjs/loader:1141:19)\n at Module.Hook._require.Module.require (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/node_modules/require-in-the-middle/index.js:80:39)\n at Module.Hook._require.Module.require (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/node_modules/require-in-the-middle/index.js:80:39)\n at Module.Hook._require.Module.require (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/node_modules/require-in-the-middle/index.js:80:39)\n at Module.Hook._require.Module.require (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/node_modules/require-in-the-middle/index.js:80:39)\n at Module.Hook._require.Module.require (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/node_modules/require-in-the-middle/index.js:80:39)\n at require (node:internal/modules/cjs/helpers:110:18)\n at Object.<anonymous> (/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/clusters/createSecurityAnalyticsCluster.ts:6:1) {\n code: 'MODULE_NOT_FOUND',\n requireStack: [\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/utils/constants.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/clusters/createSecurityAnalyticsCluster.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/plugin.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/plugins/securityAnalyticsDashboards/server/index.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/plugin.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/discovery/plugins_discovery.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/discovery/index.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/plugins_service.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/plugins/index.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/core/server/index.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/cli/serve/serve.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/cli/cli.js',\n '/tmp/tmphnsq2vgw/local-test-cluster/opensearch-dashboards-2.12.0/src/cli/dist.js'\n ]\n}"}

[peterzhuamazon]

Should be solved by this PR:

• Add missing modules common and types security-analytics-dashboards-plugin#875

[peterzhuamazon]

New issue:

• [RELEASE] Release version 2.12.0 #4115 (comment)

[peterzhuamazon]

There seems to be a bug where Windows will run better if:

# OpenSearch:
http.host: 0.0.0.0
network.host: 127.0.0.1

# OpenSearch-Dashboards:
server.host: 0.0.0.0

If not this then it will nearly 100% fail the manual run of Windows ftrepo directly.

Very weird.

[peterzhuamazon]

Resolved now.