TestServiceSoftware/TestAndroidSoftwareIngestion is flaky

[jahzielv]

Fleet version: main as of 11-17-2025

Web browser and operating system: N/A

---

💥  Actual behavior

TestServiceSoftware/TestAndroidSoftwareIngestion fails during the daily CI runs.

Specific error:

level=error user= method=POST uri="/api/v1/fleet/android_enterprise/pubsub?token=YBdRqdsOdDpydOlC-Oh4Bnbbs4wlaojutapQr5BKB5UXGtlhpDz_ylfZZIyS1ly-d7R3S7Gf6MC3iy4_ty3slA%3D%3D" took=2.510352151s err="enrolling Android host: enrolling Android host: new Android host MDM info: upsert mdm solution: Error 1146 (42S02): Table 'server_mdm_android_tests_CreateNamedMySQLDS.mobile_device_management_solutions' doesn't exist"
    http.go:52: 
        	Error Trace:	/home/runner/work/fleet/fleet/server/test/httptest/http.go:52
        	            				/home/runner/work/fleet/fleet/server/mdm/android/tests/http.go:50
        	            				/home/runner/work/fleet/fleet/server/mdm/android/tests/http.go:37
        	            				/home/runner/work/fleet/fleet/server/mdm/android/tests/http.go:28
        	            				/home/runner/work/fleet/fleet/server/mdm/android/tests/integration_software_test.go:103
        	            				/opt/hostedtoolcache/go/1.25.3/x64/src/reflect/value.go:365
        	            				/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.11.1/suite/suite.go:196
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 422
        	Test:       	TestServiceSoftware/TestAndroidSoftwareIngestion
        	Messages:   	Fleet jsonError: {Message:Validation Failed Code:0 Errors:[map[name:base reason:Error 1146 (42S02): Table 'server_mdm_android_tests_CreateNamedMySQLDS.mobile_device_management_solutions' doesn't exist]] UUID:}

🛠️ To fix

Feels like a race condition during test setup? Hard to tell because local and randokiller tests couldn't reproduce.

🧑‍💻  Steps to reproduce

🕯️ More info (optional)

First step is being able to reproduce this.

[iansltx]

Sometimes TestServiceEnterprise fails on this as well: https://github.com/fleetdm/fleet/actions/runs/19453642559/job/55663323816

[iansltx]

Issue started overnight on Thursday: https://github.com/fleetdm/fleet/actions/runs/19353873362

EDIT: tl;dr: ff0ba41 introduced parallel tests that exposed the race condition on setup where the data store method not using the named DB started mattering.

[iansltx]

After pushing the branch, I ran this by Zed + Opus 4.1 Thinking to get some more context, and the diagnosis makes sense:

User (checkpoint)

Sometime on Thursday, a code change caused database errors relating to the database server_mdm_android_tests_CreateNamedMySQLDS not being found to show up in integration tests, particularly TestServiceEnterprise and TestServiceSoftware/TestAndroidSoftwareIngestion. Find the cause for this failure. #35847 has some more detail.

Assistant

<thinking>
Let me analyze this issue about database errors in integration tests. The error is about a database server named "server_mdm_android_tests_CreateNamedMySQLDS" not being found, which started happening on Thursday. Let me start by gathering more information from the GitHub issue.
</thinking>

I'll help you investigate this database error that appeared in your integration tests on Thursday. Let me start by fetching the GitHub issue to understand the problem better.

Tool Call: Fetch #35847
Status: Completed

<snip>

Assistant

Now let me search for recent changes related to Android tests and the database setup that might have been introduced on Thursday. Let me first find the Android test files and understand the structure:

Tool Call: Search files for regex server_mdm_android_tests_CreateNamedMySQLDS
Status: Completed

No matches found

Tool Call: Search files for regex CreateNamedMySQLDS
Status: Completed

Found 6 matches:

Matches in fleet/cmd/fleetctl/integrationtest/suite.go

func (ts *WithDS) SetupSuite › L24-34

func (ts *WithDS) SetupSuite(dbName string) {
	t := ts.Suite.T()
	ts.DS = mysql.CreateNamedMySQLDS(t, dbName)
	test.AddAllHostsLabel(t, ts.DS)

	// Set up the required fields on AppConfig
	appConf, err := ts.DS.AppConfig(context.Background())
	require.NoError(t, err)
	appConf.OrgInfo.OrgName = "FleetTest"
	appConf.ServerSettings.ServerURL = "https://example.org"
	err = ts.DS.SaveAppConfig(context.Background(), appConf)

2 lines remaining in ancestor node. Read the file to see all.

Matches in fleet/server/datastore/mysql/scep_test.go

func setup › L14-19

func setup(t *testing.T) scep_depot.Depot {
	ds := CreateNamedMySQLDS(t, t.Name())
	depot, err := ds.NewSCEPDepot()
	require.NoError(t, err)
	return depot
}

Matches in fleet/server/datastore/mysql/testing_utils.go

func CreateNamedMySQLDS › L410-418

func CreateNamedMySQLDS(t *testing.T, name string) *Datastore {
	if _, ok := os.LookupEnv("MYSQL_TEST"); !ok {
		t.Skip("MySQL tests are disabled")
	}

	ds := initializeDatabase(t, name, new(testing_utils.DatastoreTestOptions))
	t.Cleanup(func() { ds.Close() })
	return ds
}

Matches in fleet/server/mdm/android/tests/testing_utils.go

func (ts *WithServer) SetupSuite › L107-117

func (ts *WithServer) SetupSuite(t *testing.T, dbName string) {
	ts.DS.Datastore = CreateNamedMySQLDS(t, dbName)
	ts.CreateCommonDSMocks()

	ts.AndroidAPIClient = android_mock.Client{}
	ts.createCommonProxyMocks(t)

	logger := kitlog.NewLogfmtLogger(os.Stdout)
	activityModule := activities.NewActivityModule(&ts.DS.DataStore, logger)
	svc, err := service.NewServiceWithClient(logger, &ts.DS, &ts.AndroidAPIClient, "test-private-key", ts.DS.Datastore, activityModule)
	require.NoError(t, err)

4 lines remaining in ancestor node. Read the file to see all.

func CreateNamedMySQLDS › L244-250

func CreateNamedMySQLDS(t *testing.T, name string) *mysql.Datastore {
	if _, ok := os.LookupEnv("MYSQL_TEST"); !ok {
		t.Skip("MySQL tests are disabled")
	}
	// use the standard Fleet datastore for Android integration tests
	return mysql.CreateMySQLDS(t)
}

Matches in fleet/server/service/testing_client.go

func (ts *withDS) SetupSuite › L45-55

func (ts *withDS) SetupSuite(dbName string) {
	t := ts.s.T()
	ts.ds = mysql.CreateNamedMySQLDS(t, dbName)
	// remove any migration-created labels
	mysql.ExecAdhocSQL(t, ts.ds, func(q sqlx.ExtContext) error {
		_, err := q.ExecContext(context.Background(), `DELETE FROM labels`)
		return err
	})
	test.AddBuiltinLabels(t, ts.ds)

	// setup the required fields on AppConfig

7 lines remaining in ancestor node. Read the file to see all.

Assistant

I can see there are two CreateNamedMySQLDS functions - one in fleet/server/datastore/mysql/testing_utils.go and one in fleet/server/mdm/android/tests/testing_utils.go. The Android one actually doesn't use the name parameter and creates a standard MySQL DS. Let me look at the Android test integration file more closely:

Tool Call: Read file server/mdm/android/tests/testing_utils.go (lines 240-255)
Status: Completed

func testCtx() context.Context {
	return context.Background()
}

func CreateNamedMySQLDS(t *testing.T, name string) *mysql.Datastore {
	if _, ok := os.LookupEnv("MYSQL_TEST"); !ok {
		t.Skip("MySQL tests are disabled")
	}
	// use the standard Fleet datastore for Android integration tests
	return mysql.CreateMySQLDS(t)
}

Assistant

This is suspicious! The Android CreateNamedMySQLDS function ignores the name parameter and calls mysql.CreateMySQLDS(t) instead. Let me check what the regular MySQL one does:

Tool Call: Read file server/datastore/mysql/testing_utils.go (lines 410-425)
Status: Completed

func CreateNamedMySQLDS(t *testing.T, name string) *Datastore {
	if _, ok := os.LookupEnv("MYSQL_TEST"); !ok {
		t.Skip("MySQL tests are disabled")
	}

	ds := initializeDatabase(t, name, new(testing_utils.DatastoreTestOptions))
	t.Cleanup(func() { ds.Close() })
	return ds
}

func ExecAdhocSQL(tb testing.TB, ds *Datastore, fn func(q sqlx.ExtContext) error) {
	tb.Helper()
	err := fn(ds.primary)
	require.NoError(tb, err)
}

Assistant

Now let me check what CreateMySQLDS does:

Tool Call: Search files for regex func CreateMySQLDS\(
Status: Completed

Found 1 matches:

Matches in fleet/server/datastore/mysql/testing_utils.go

func CreateMySQLDS › L406-408

func CreateMySQLDS(t testing.TB) *Datastore {
	return createMySQLDSWithOptions(t, nil)
}

Tool Call: Read file server/datastore/mysql/testing_utils.go (lines 400-410)
Status: Completed

}

func CreateMySQLDSWithOptions(t *testing.T, opts *testing_utils.DatastoreTestOptions) *Datastore {
	return createMySQLDSWithOptions(t, opts)
}

func CreateMySQLDS(t testing.TB) *Datastore {
	return createMySQLDSWithOptions(t, nil)
}

func CreateNamedMySQLDS(t *testing.T, name string) *Datastore {

Tool Call: Read file server/datastore/mysql/testing_utils.go (lines 380-405)
Status: Completed

	}
	opts.RealReplica = true
	const numberOfAttempts = 10
	var ds *Datastore
	for attempt := 0; attempt < numberOfAttempts; {
		attempt++
		ds = createMySQLDSWithOptions(t, opts)
		status, err := ds.ReplicaStatus(context.Background())
		require.NoError(t, err)
		if status["Replica_SQL_Running"] != "Yes" {
			t.Logf("create replica attempt: %d replica status: %+v", attempt, status)
			if lastErr, ok := status["Last_Error"]; ok && lastErr != "" {
				t.Logf("replica not running after attempt %d; Last_Error: %s", attempt, lastErr)
			}
			continue
		}
		break
	}
	require.NotNil(t, ds)
	return ds
}

func CreateMySQLDSWithOptions(t *testing.T, opts *testing_utils.DatastoreTestOptions) *Datastore {
	return createMySQLDSWithOptions(t, opts)
}

Assistant

Let me see the createMySQLDSWithOptions function:

Tool Call: Search files for regex func createMySQLDSWithOptions
Status: Completed

Found 2 matches:

Matches in fleet/server/datastore/mysql/testing_utils.go

func createMySQLDSWithOptions › L370-375

func createMySQLDSWithOptions(t testing.TB, opts *testing_utils.DatastoreTestOptions) *Datastore {
	cleanTestName, opts := testing_utils.ProcessOptions(t, opts)
	ds := initializeDatabase(t, cleanTestName, opts)
	t.Cleanup(func() { ds.Close() })
	return ds
}

func CreateMySQLDSWithOptions › L402-404

func CreateMySQLDSWithOptions(t *testing.T, opts *testing_utils.DatastoreTestOptions) *Datastore {
	return createMySQLDSWithOptions(t, opts)
}

Assistant

So the key difference is that CreateNamedMySQLDS calls initializeDatabase(t, name, ...) with the provided name, whereas CreateMySQLDS calls initializeDatabase(t, cleanTestName, ...) after processing the test name. The issue is that the Android tests' version of CreateNamedMySQLDS ignores the name parameter completely!

Let me look at when the Android test's version was changed - let's look at the git history of this file:

Tool Call: git log -n 20 --oneline --since="2025-11-14" -- server/mdm/android/tests/testing_utils.go
Status: Completed

Terminal:

ce42562b4d (HEAD -> pass-through-name-on-android-testing-setup, origin/pass-through-name-on-android-testing-setup) 🤖 Pass through name when creating a named datastore for Android tests

Assistant

Interesting! There's already a fix for this. Let me look at that commit:

Tool Call: git show ce42562b4d --stat
Status: Completed

Terminal:

commit ce42562b4d67ad86bd3653a2375fddb24ac834cd (HEAD -> pass-through-name-on-android-testing-setup, origin/pass-through-name-on-android-testing-setup)
Author: Ian Littman <iansltx@gmail.com>
Date:   Tue Nov 18 10:51:26 2025 -0600

    🤖 Pass through name when creating a named datastore for Android tests

    Related to #35847. Flagged by both Junie Pro and Zed + Sonnet 4.5.

    Prompt for Junie:

    Sometime on Thursday, a code change caused database errors relating to the database server_mdm_android_tests_CreateNamedMySQLDS not being found to show up in integration tests, particularly TestServiceEnterprise and TestServiceSoftware/TestAndroidSoftwareIngestion. Find the cause for this failure.

    Prompt for Zed:

    Sometime on Thursday, a code change caused database errors relating to the database server_mdm_android_tests_CreateNamedMySQLDS not being found to show up in integration tests, particularly TestServiceEnterprise and TestServiceSoftware/TestAndroidSoftwareIngestion. Find the cause for this failure. https://github.com/fleetdm/fleet/issues/35847 has some more detail.

 server/mdm/android/tests/testing_utils.go | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Tool Call: git show ce42562b4d
Status: Completed

Terminal:

commit ce42562b4d67ad86bd3653a2375fddb24ac834cd (HEAD -> pass-through-name-on-android-testing-setup, origin/pass-through-name-on-android-testing-setup)
Author: Ian Littman <iansltx@gmail.com>
Date:   Tue Nov 18 10:51:26 2025 -0600

    🤖 Pass through name when creating a named datastore for Android tests

    Related to #35847. Flagged by both Junie Pro and Zed + Sonnet 4.5.

    Prompt for Junie:

    Sometime on Thursday, a code change caused database errors relating to the database server_mdm_android_tests_CreateNamedMySQLDS not being found to show up in integration tests, particularly TestServiceEnterprise and TestServiceSoftware/TestAndroidSoftwareIngestion. Find the cause for this failure.

    Prompt for Zed:

    Sometime on Thursday, a code change caused database errors relating to the database server_mdm_android_tests_CreateNamedMySQLDS not being found to show up in integration tests, particularly TestServiceEnterprise and TestServiceSoftware/TestAndroidSoftwareIngestion. Find the cause for this failure. https://github.com/fleetdm/fleet/issues/35847 has some more detail.

diff --git a/server/mdm/android/tests/testing_utils.go b/server/mdm/android/tests/testing_utils.go
index 16f30d2be6..a7541fec9a 100644
--- a/server/mdm/android/tests/testing_utils.go
+++ b/server/mdm/android/tests/testing_utils.go
@@ -245,6 +245,6 @@ func CreateNamedMySQLDS(t *testing.T, name string) *mysql.Datastore {
 	if _, ok := os.LookupEnv("MYSQL_TEST"); !ok {
 		t.Skip("MySQL tests are disabled")
 	}
-	// use the standard Fleet datastore for Android integration tests
-	return mysql.CreateMySQLDS(t)
+	// Use a named Fleet datastore for Android integration tests to ensure the expected database exists
+	return mysql.CreateNamedMySQLDS(t, name)
 }

Assistant

Perfect! The fix is already in a branch. Let me now look for what change might have been introduced on Thursday that exposed this issue. Let's look at the commit history:

Tool Call: git log --since="2025-11-14" --until="2025-11-15" --oneline | head -20
Status: Completed

Terminal:

876bdc9adf Update Fleet-maintained apps (#35797)
60295fb88d Creating Windows FMA for Firefox (#35453)
7c8d010153 Create new FMA for Zoom for Windows (#35501)
7678024260 Create new FMA for Slack for Windows (#35470)
ed73a7b601 FMA: PhpStorm for macOS (#35578)
65d82d676b Added Proton Mail as a macOS FMA (#35656)
999a7f1d23 Add ClickUp as a macOS FMA (#35657)
a31f8c5cbe Add GitKraken as a macOS FMA (#35658)
13c5326c05 Add Abstract as macOS FMA (#35662)
e4eccc8abe Add Webex as macOS FMA (#35396)
6c1fde7117 Update Fleet-maintained apps (#35790)
668346a5f6 Update README.md (#35772)
333e40a004 Update DRI for Fleet-maintained apps (#35784)
98452d4827 [API/YAML] Docs for new setup experience script PUT endpoint (#35736)
a074498668 API design: #33758 (OS vulnerabilities bug) (#33533)
28f58ed465 Fix iOS/iPadOS self-service UI routing and database errors (#35739)
caf9e83968 Configure osquery-perf memory and cpu (#35786)
4d0fba76ff Redistribute QA responsibilities (#35779)
c477c1f7a8 Add tools README (#35719)
7f67ac940f Okta IdP Apple profile endpoint + fixes (#35526)

Assistant

Let me look at commits that might have touched test files:

Tool Call: git log --since="2025-11-14" --until="2025-11-15" --stat | grep -E "test.*\.go" -B 5 -A 2 | head -50
Status: Completed

Terminal:

    * Centralized error handling for internal server errors with improved
    context logging.

    <!-- end of auto-generated comment: release notes by coderabbit.ai -->

 .../integrationtest/condaccess/condaccess_test.go  |  17 +-
 ee/server/service/condaccess/idp.go                | 127 ++++---
 server/authz/policy.rego                           |   4 +-
 server/fleet/app.go                                |  38 ++
 server/fleet/app_test.go                           |  83 +++++
 server/fleet/conditional_access_idp.go             |  11 +-
 server/fleet/service.go                            |   3 +
 server/mock/service/service_mock.go                |  12 +
 server/service/conditional_access_idp.go           | 337 ++++++++++++++++-
 server/service/conditional_access_idp_test.go      | 407 ++++++++++++++++++++-
 server/service/handler.go                          |   1 +
 server/service/middleware/log/log.go               |  33 ++

Assistant

Let me look for changes that might have touched database creation or test initialization:

Tool Call: git log --since="2025-11-13" --until="2025-11-16" --oneline -- "*testing_utils.go" "*test_setup.go"
Status: Completed

Terminal:

1a61b29092 Add PUT endpoint for setup experience script (#35651)
ff0ba413b7 Android app self service: backend support (#34711)

Assistant

Let me check that Android app self-service commit:

Tool Call: git show ff0ba413b7 --stat
Status: Completed

Terminal:

commit ff0ba413b7e4b81be0bc9e922842b7629e6e4623
Author: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
Date:   Thu Nov 13 18:10:24 2025 -0500

    Android app self service: backend support (#34711)

    <!-- Add the related story/sub-task/bug number, like Resolves #123, or
    remove if NA -->
    **Related issue:** Resolves #34389

    # Checklist for submitter

    If some of the following don't apply, delete the relevant line.

    - [x] Input data is properly validated, `SELECT *` is avoided, SQL
    injection is prevented (using placeholders for values in statements)

    ## Testing

    - [x] Added/updated automated tests
    - [x] Where appropriate, [automated tests simulate multiple hosts and
    test for host
    isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
    (updates to one hosts's records do not affect another)
    - [x] QA'd all new/changed functionality manually

    ## Database migrations

    - [x] Checked schema for all modified table for columns that will
    auto-update timestamps during migration.
    - [x] Ensured the correct collation is explicitly set for character
    columns (`COLLATE utf8mb4_unicode_ci`).

 cmd/fleet/cron.go                                               |   8 +-
 cmd/fleet/serve.go                                              |   7 +-
 ee/server/service/mdm_external_test.go                          |   1 +
 ee/server/service/service.go                                    |   4 +
 ee/server/service/software_installers.go                        |   8 +-
 ee/server/service/vpp.go                                        | 160 +++++++++++++-------
 main                                                            | Bin 2878258 -> 0 bytes
 server/datastore/mysql/android.go                               |  14 ++
 server/datastore/mysql/android_enterprises.go                   |  22 ++-
 server/datastore/mysql/android_mysql.go                         |   4 +
 server/datastore/mysql/mdm.go                                   |  12 +-
 .../tables/20240730374423_AddPlatformToVPPApps_test.go          |   2 +-
 .../migrations/tables/20251112191443_AddAndroidAppSupport.go    |  81 ++++++++++
 .../tables/20251112191443_AddAndroidAppSupport_test.go          |  35 +++++
 server/datastore/mysql/mysql.go                                 |   4 +
 server/datastore/mysql/schema.sql                               |  21 ++-
 server/datastore/mysql/software_installers.go                   |  35 ++++-
 server/datastore/mysql/vpp.go                                   | 131 ++++++++++++++--
 server/datastore/mysql/vpp_test.go                              |   2 +-
 server/fleet/activities.go                                      |  63 ++++----
 server/fleet/datastore.go                                       |   6 +-
 server/fleet/mdm.go                                             |  16 +-
 server/fleet/policies.go                                        |   6 +-
 server/fleet/setup_experience.go                                |   2 +-
 server/fleet/software.go                                        |  12 +-
 server/fleet/software_installer.go                              |   2 +-
 server/fleet/software_title_icons.go                            |  28 ++--
 server/fleet/vpp.go                                             |   4 +-
 server/mdm/android/arch_test.go                                 |   1 +
 server/mdm/android/mock/client.go                               |  24 +++
 server/mdm/android/service.go                                   |   7 +
 server/mdm/android/service/androidmgmt/client.go                |   4 +
 server/mdm/android/service/androidmgmt/google_client.go         | 104 ++++++++++---
 .../mdm/android/service/androidmgmt/policy_field_mask_test.go   |  15 ++
 server/mdm/android/service/androidmgmt/proxy_client.go          |  71 ++++++---
 server/mdm/android/service/enterprises_test.go                  |  55 ++++---
 server/mdm/android/service/pubsub.go                            |  25 +++-
 server/mdm/android/service/pubsub_test.go                       |   9 +-
 server/mdm/android/service/service.go                           | 131 ++++++++++++++--
 server/mdm/android/tests/enterprise/enterprise_test.go          |   4 -
 server/mdm/android/tests/testing_utils.go                       |   8 +-
 server/mock/datastore_mock.go                                   |  28 +++-
 server/service/activities.go                                    |  11 +-
 server/service/activities_test.go                               |   4 +-
 server/service/integration_android_software_test.go             | 224 ++++++++++++++++++++++++++++
 server/service/integration_mdm_test.go                          |  38 ++++-
 server/service/integrationtest/android/suite.go                 |   4 +-
 server/service/modules/activities/activities.go                 | 114 ++++++++++++++
 server/service/testing_utils.go                                 |  72 +++++++++
 server/service/vpp.go                                           |  16 +-
 server/worker/software_worker.go                                | 185 +++++++++++++++++++++++
 server/worker/software_worker_test.go                           |  16 ++
 tools/mdm/assets/main.go                                        |  19 +--
 .../controllers/android-proxy/get-enterprise-applications.js    |  99 ++++++++++++
 .../api/controllers/android-proxy/modify-android-policies.js    |   1 +
 .../controllers/android-proxy/modify-enterprise-app-policy.js   |  95 ++++++++++++
 website/config/routes.js                                        |  20 +--
 website/package.json                                            |   2 +-
 58 files changed, 1788 insertions(+), 308 deletions(-)

Assistant

This commit changes the Android tests! Let me look at what was changed in the testing_utils.go file:

Tool Call: git show ff0ba413b7 -- server/mdm/android/tests/testing_utils.go
Status: Completed

Terminal:

commit ff0ba413b7e4b81be0bc9e922842b7629e6e4623
Author: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
Date:   Thu Nov 13 18:10:24 2025 -0500

    Android app self service: backend support (#34711)

    <!-- Add the related story/sub-task/bug number, like Resolves #123, or
    remove if NA -->
    **Related issue:** Resolves #34389

    # Checklist for submitter

    If some of the following don't apply, delete the relevant line.

    - [x] Input data is properly validated, `SELECT *` is avoided, SQL
    injection is prevented (using placeholders for values in statements)

    ## Testing

    - [x] Added/updated automated tests
    - [x] Where appropriate, [automated tests simulate multiple hosts and
    test for host
    isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
    (updates to one hosts's records do not affect another)
    - [x] QA'd all new/changed functionality manually

    ## Database migrations

    - [x] Checked schema for all modified table for columns that will
    auto-update timestamps during migration.
    - [x] Ensured the correct collation is explicitly set for character
    columns (`COLLATE utf8mb4_unicode_ci`).

diff --git a/server/mdm/android/tests/testing_utils.go b/server/mdm/android/tests/testing_utils.go
index 34edd44729..16f30d2be6 100644
--- a/server/mdm/android/tests/testing_utils.go
+++ b/server/mdm/android/tests/testing_utils.go
@@ -7,6 +7,7 @@ import (
 	"os"
 	"sync"
 	"testing"
+	"time"

 	"github.com/fleetdm/fleet/v4/server/config"
 	"github.com/fleetdm/fleet/v4/server/datastore/mysql"
@@ -20,6 +21,7 @@ import (
 	"github.com/fleetdm/fleet/v4/server/service/middleware/auth"
 	"github.com/fleetdm/fleet/v4/server/service/middleware/endpoint_utils"
 	"github.com/fleetdm/fleet/v4/server/service/middleware/log"
+	"github.com/fleetdm/fleet/v4/server/service/modules/activities"
 	kithttp "github.com/go-kit/kit/transport/http"
 	kitlog "github.com/go-kit/log"
 	"github.com/gorilla/mux"
@@ -110,7 +112,8 @@ func (ts *WithServer) SetupSuite(t *testing.T, dbName string) {
 	ts.createCommonProxyMocks(t)

 	logger := kitlog.NewLogfmtLogger(os.Stdout)
-	svc, err := service.NewServiceWithClient(logger, &ts.DS, &ts.AndroidAPIClient, &ts.FleetSvc, "test-private-key", ts.DS.Datastore)
+	activityModule := activities.NewActivityModule(&ts.DS.DataStore, logger)
+	svc, err := service.NewServiceWithClient(logger, &ts.DS, &ts.AndroidAPIClient, "test-private-key", ts.DS.Datastore, activityModule)
 	require.NoError(t, err)
 	ts.Svc = svc

@@ -152,6 +155,9 @@ func (ts *WithServer) CreateCommonDSMocks() {
 	ts.DS.BulkSetAndroidHostsUnenrolledFunc = func(ctx context.Context) error {
 		return nil
 	}
+	ts.DS.NewActivityFunc = func(ctx context.Context, user *fleet.User, activity fleet.ActivityDetails, details []byte, createdAt time.Time) error {
+		return nil
+	}
 }

 func (ts *WithServer) createCommonProxyMocks(t *testing.T) {

Assistant

This commit didn't change the CreateNamedMySQLDS function itself - that was already problematic. But let me look for when that function was introduced in the Android tests:

Tool Call: git log --oneline -p -- server/mdm/android/tests/testing_utils.go | grep -B5 -A5 "CreateNamedMySQLDS" | head -30
Status: Completed

Terminal:

ce42562b4d 🤖 Pass through name when creating a named datastore for Android tests
diff --git a/server/mdm/android/tests/testing_utils.go b/server/mdm/android/tests/testing_utils.go
index 16f30d2be6..a7541fec9a 100644
--- a/server/mdm/android/tests/testing_utils.go
+++ b/server/mdm/android/tests/testing_utils.go
@@ -245,6 +245,6 @@ func CreateNamedMySQLDS(t *testing.T, name string) *mysql.Datastore {
 	if _, ok := os.LookupEnv("MYSQL_TEST"); !ok {
 		t.Skip("MySQL tests are disabled")
 	}
-	// use the standard Fleet datastore for Android integration tests
-	return mysql.CreateMySQLDS(t)
+	// Use a named Fleet datastore for Android integration tests to ensure the expected database exists
+	return mysql.CreateNamedMySQLDS(t, name)
 }
ff0ba413b7 Android app self service: backend support (#34711)
diff --git a/server/mdm/android/tests/testing_utils.go b/server/mdm/android/tests/testing_utils.go
index 34edd44729..16f30d2be6 100644
--- a/server/mdm/android/tests/testing_utils.go
--
-	mysql.Close(ts.DS.Datastore)
+	ts.DS.Datastore.Close()
 }

 type mockService struct {
@@ -194,7 +233,6 @@ func CreateNamedMySQLDS(t *testing.T, name string) *mysql.Datastore {
 	if _, ok := os.LookupEnv("MYSQL_TEST"); !ok {
 		t.Skip("MySQL tests are disabled")
 	}
-	ds := mysql.InitializeDatabase(t, name, new(testing_utils.DatastoreTestOptions))
-	t.Cleanup(func() { mysql.Close(ds) })

Assistant

Let me look for when this function was introduced:

Tool Call: git blame server/mdm/android/tests/testing_utils.go | grep -A2 "CreateNamedMySQLDS"
Status: Completed

Terminal:

a86253d2bfe (Victor Lyuboslavsky        2025-03-13 14:28:52 -0500 108) 	ts.DS.Datastore = CreateNamedMySQLDS(t, dbName)
a0158af6d86 (Victor Lyuboslavsky        2025-02-26 16:20:02 -0600 109) 	ts.CreateCommonDSMocks()
3d5666d4c6b (Victor Lyuboslavsky        2025-02-26 10:47:05 -0600 110)
--
3d5666d4c6b (Victor Lyuboslavsky        2025-02-26 10:47:05 -0600 244) func CreateNamedMySQLDS(t *testing.T, name string) *mysql.Datastore {
3d5666d4c6b (Victor Lyuboslavsky        2025-02-26 10:47:05 -0600 245) 	if _, ok := os.LookupEnv("MYSQL_TEST"); !ok {
3d5666d4c6b (Victor Lyuboslavsky        2025-02-26 10:47:05 -0600 246) 		 t.Skip("MySQL tests are disabled")
--
ce42562b4d6 (Ian Littman                2025-11-18 10:51:26 -0600 249) 	return mysql.CreateNamedMySQLDS(t, name)
3d5666d4c6b (Victor Lyuboslavsky        2025-02-26 10:47:05 -0600 250) }

Assistant

Let me look at commit a86253d where the function was changed to be called with a dbName:

Tool Call: git show a86253d2bfe --stat | head -15
Status: Completed

Terminal:

commit a86253d2bfedd6b8a176fe7d3dfb55df7e28f837
Author: Victor Lyuboslavsky <victor.lyuboslavsky@gmail.com>
Date:   Thu Mar 13 14:28:52 2025 -0500

    Refactoring Android datastore interface (#26982)

    For #26219

    Refactoring the interface between Android service and Android datastore
    to use 1 common datastore interface: `fleet.AndroidDatastore`

    These changes are based on feedback from the recent Backend Sync.

    ```mermaid
    ---

Tool Call: git show a86253d2bfe -- server/mdm/android/tests/testing_utils.go | grep -B5 -A5 "CreateNamedMySQLDS"
Status: Completed

Terminal:

 	Token    string
@@ -53,14 +57,14 @@ type WithServer struct {
 }

 func (ts *WithServer) SetupSuite(t *testing.T, dbName string) {
-	ts.DS = CreateNamedMySQLDS(t, dbName)
+	ts.DS.Datastore = CreateNamedMySQLDS(t, dbName)
 	ts.CreateCommonDSMocks()

 	ts.Proxy = proxy_mock.Proxy{}
 	ts.createCommonProxyMocks(t)

Assistant

So the issue is that prior to this refactoring on March 13, it was calling CreateNamedMySQLDS with a database name (dbName), but the Android version of the function always ignored this parameter. This worked fine for a while because the tests were still isolated.

However, the issue started appearing on Thursday (November 14). Let me check what might have changed in how the tests are run in parallel:

Tool Call: Search files for regex TestServiceSoftware|TestAndroidSoftwareIngestion
Status: Completed

Found 4 matches:

Matches in fleet/server/mdm/android/tests/integration_software_test.go

func TestServiceSoftware › L26-29

func TestServiceSoftware(t *testing.T) {
	testingSuite := new(softwareTestSuite)
	suite.Run(t, testingSuite)
}

func (s *softwareTestSuite) TestAndroidSoftwareIngestion › L40-50

func (s *softwareTestSuite) TestAndroidSoftwareIngestion() {
	ctx := context.Background()
	t := s.T()

	// Create enterprise
	var signupResp android.EnterpriseSignupResponse
	s.DoJSON("GET", "/api/v1/fleet/android_enterprise/signup_url", nil, http.StatusOK, &signupResp)
	assert.Equal(s.T(), EnterpriseSignupURL, signupResp.Url)
	s.T().Logf("callbackURL: %s", s.ProxyCallbackURL)

	s.FleetSvc.On("NewActivity", mock.Anything, mock.Anything, mock.AnythingOfType("fleet.ActivityTypeEnabledAndroidMDM")).Return(nil)

241 lines remaining in ancestor node. Read the file to see all.

Matches in fleet/server/service/software_test.go

func TestServiceSoftwareInventoryAuth › L55-65

func TestServiceSoftwareInventoryAuth(t *testing.T) {
	ds := new(mock.Store)

	ds.ListSoftwareFunc = func(ctx context.Context, opt fleet.SoftwareListOptions) ([]fleet.Software, *fleet.PaginationMetadata, error) {
		return []fleet.Software{}, &fleet.PaginationMetadata{}, nil
	}
	ds.CountSoftwareFunc = func(ctx context.Context, opt fleet.SoftwareListOptions) (int, error) {
		return 0, nil
	}
	ds.SoftwareByIDFunc = func(ctx context.Context, id uint, teamID *uint, includeCVEScores bool, tmFilter *fleet.TeamFilter) (*fleet.Software, error) {
		return &fleet.Software{}, nil

138 lines remaining in ancestor node. Read the file to see all.

Matches in fleet/server/service/software_titles_test.go

func TestServiceSoftwareTitlesAuth › L15-25

func TestServiceSoftwareTitlesAuth(t *testing.T) {
	ds := new(mock.Store)

	ds.ListSoftwareTitlesFunc = func(ctx context.Context, opt fleet.SoftwareTitleListOptions, tmf fleet.TeamFilter) ([]fleet.SoftwareTitleListResult, int, *fleet.PaginationMetadata, error) {
		return []fleet.SoftwareTitleListResult{}, 0, &fleet.PaginationMetadata{}, nil
	}
	ds.SoftwareTitleByIDFunc = func(ctx context.Context, id uint, teamID *uint, tmFilter fleet.TeamFilter) (*fleet.SoftwareTitle, error) {
		return &fleet.SoftwareTitle{}, nil
	}
	ds.TeamExistsFunc = func(ctx context.Context, teamID uint) (bool, error) { return true, nil }
	ds.SoftwareTitleByIDFunc = func(ctx context.Context, id uint, teamID *uint, tmFilter fleet.TeamFilter) (*fleet.SoftwareTitle, error) {

157 lines remaining in ancestor node. Read the file to see all.

Tool Call: Read file server/mdm/android/tests/integration_software_test.go (lines 30-40)
Status: Completed

type softwareTestSuite struct {
	WithServer
}

func (s *softwareTestSuite) SetupSuite() {
	s.WithServer.SetupSuite(s.T(), "androidSoftwareTestSuite")
	s.Token = "testtoken"
}

func (s *softwareTestSuite) TestAndroidSoftwareIngestion() {

Assistant

So the issue is that when SetupSuite is called with "androidSoftwareTestSuite" as the database name, it should create a database with that specific name. However, the Android tests' CreateNamedMySQLDS was not using that name and was just creating a default database.

Now let's check what might have changed to cause the actual failure. Let me look for more integration tests that were added around Thursday:

Tool Call: git log --since="2025-11-13" --until="2025-11-15" --stat | grep integration | head -20
Status: Completed

Terminal:

    integration.
 .../integrationtest/condaccess/condaccess_test.go  |  17 +-
 server/service/integration_mdm_profiles_test.go    |  29 +++--
 .../integration_mdm_setup_experience_test.go       | 204 +++++++++++++++++++++
 .../service/integration_android_software_test.go   | 224 +++++++++++++++++++++
 server/service/integration_mdm_test.go             |  38 +++-
 server/service/integrationtest/android/suite.go    |   4 +-
 server/service/integration_core_test.go            |   6 +
 server/service/integration_enterprise_test.go      |  23 +-

Assistant

I see a new file server/service/integration_android_software_test.go was added! Let's look at this:

Tool Call: git log --oneline -- server/service/integration_android_software_test.go
Status: Completed

Terminal:

ff0ba413b7 Android app self service: backend support (#34711)

Tool Call: Read file server/service/integration_android_software_test.go (lines 1-50)
Status: Completed

package service

import (
	"context"
	"fmt"
	"net/http"
	"net/url"
	"strings"

	"github.com/fleetdm/fleet/v4/server/fleet"
	"github.com/fleetdm/fleet/v4/server/mdm/android"
	android_service "github.com/fleetdm/fleet/v4/server/mdm/android/service"
	"github.com/fleetdm/fleet/v4/server/mdm/android/service/androidmgmt"
	"github.com/google/uuid"
	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
	"google.golang.org/api/androidmanagement/v1"
)

func (s *integrationMDMTestSuite) TestAndroidAppSelfService() {
	ctx := context.Background()
	t := s.T()

	appConf, err := s.ds.AppConfig(context.Background())
	require.NoError(s.T(), err)
	appConf.MDM.AndroidEnabledAndConfigured = false
	err = s.ds.SaveAppConfig(context.Background(), appConf)
	require.NoError(s.T(), err)
	s.setVPPTokenForTeam(0)

	t.Cleanup(func() {
		appConf, err := s.ds.AppConfig(context.Background())
		require.NoError(s.T(), err)
		appConf.MDM.AndroidEnabledAndConfigured = true
		err = s.ds.SaveAppConfig(context.Background(), appConf)
		require.NoError(s.T(), err)
	})

	// Adding android app before android MDM is turned on should fail
	var addAppResp addAppStoreAppResponse
	s.DoJSON(
		"POST",
		"/api/latest/fleet/software/app_store_apps",
		&addAppStoreAppRequest{AppStoreID: "com.should.fail", Platform: fleet.AndroidPlatform},
		http.StatusBadRequest,
		&addAppResp,
	)

	EnterpriseID := "LC02k5wxw7"
	EnterpriseSignupURL := "https://enterprise.google.com/signup/android/email?origin=android&thirdPartyToken=B4D779F1C4DD9A440"

Assistant

Now let's check what happens with databases in parallel tests. This new test was added in the same commit (ff0ba41) that added Android app self-service support. The problem is that when multiple tests run in parallel, they might be trying to use different database names, but the Android CreateNamedMySQLDS was always returning the same database (without using the name).

The error message shows server_mdm_android_tests_CreateNamedMySQLDS which looks like it's trying to use a database that was supposed to be created with that specific name. Let me search for where this naming pattern comes from:

Tool Call: Search files for regex server_mdm_android_tests
Status: Completed

No matches found

Assistant

Let me look at how database names are generated:

Tool Call: Read file server/datastore/mysql/testing_utils.go (lines 60-100)
Status: Completed

	// TODO: for some reason we never log datastore messages when running integration tests, why?
	//
	// Changes below assume that we want to follows the same pattern as the rest of the codebase.
	dslogger := log.NewLogfmtLogger(os.Stdout)
	if os.Getenv("FLEET_INTEGRATION_TESTS_DISABLE_LOG") != "" {
		dslogger = log.NewNopLogger()
	}

	// Use TestSQLMode which combines ANSI mode components with MySQL 8 strict modes
	// This ensures we catch data truncation errors and other strict behaviors during testing
	// Reference: https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html
	ds, err := New(*cfg, clock.NewMockClock(), Logger(dslogger), LimitAttempts(1), replicaOpt, SQLMode(common_mysql.TestSQLMode), WithFleetConfig(&tc))
	require.Nil(t, err)

	if opts.DummyReplica {
		setupDummyReplica(t, testName, ds, opts)
	}
	if opts.RealReplica {
		replicaOpts := &common_mysql.DBOptions{
			MinLastOpenedAtDiff: defaultMinLastOpenedAtDiff,
			MaxAttempts:         1,
			Logger:              log.NewNopLogger(),
			SqlMode:             common_mysql.TestSQLMode,
		}
		setupRealReplica(t, testName, ds, replicaOpts)
	}

	return ds
}

func setupDummyReplica(t testing.TB, testName string, ds *Datastore, opts *testing_utils.DatastoreTestOptions) {
	t.Helper()

	// create the context that will cancel the replication goroutine on test exit
	var cancel func()
	ctx := context.Background()
	if tt, ok := t.(*testing.T); ok {
		if dl, ok := tt.Deadline(); ok {
			ctx, cancel = context.WithDeadline(ctx, dl)
		} else {
			ctx, cancel = context.WithCancel(ctx)

Assistant

Let me look for the initializeDatabase function:

Tool Call: Search files for regex func initializeDatabase
Status: Completed

Found 1 matches:

Matches in fleet/server/datastore/mysql/testing_utils.go

func initializeDatabase › L363-368

func initializeDatabase(t testing.TB, testName string, opts *testing_utils.DatastoreTestOptions) *Datastore {
	_, filename, _, _ := runtime.Caller(0)
	schemaPath := path.Join(path.Dir(filename), "schema.sql")
	testing_utils.LoadSchema(t, testName, opts, schemaPath)
	return connectMySQL(t, testName, opts)
}

Tool Call: Search files for regex func LoadSchema
Status: Completed

Found 1 matches:

Matches in fleet/server/datastore/mysql/common_mysql/testing_utils/testing_utils.go

func LoadSchema › L111-121

func LoadSchema(t testing.TB, testName string, opts *DatastoreTestOptions, schemaPath string) {
	schema, err := os.ReadFile(schemaPath)
	if err != nil {
		t.Error(err)
		t.FailNow()
	}

	// execute the schema for the test db, and once more for the replica db if
	// that option is set.
	dbs := []string{testName}
	if opts.DummyReplica {

45 lines remaining in ancestor node. Read the file to see all.

Assistant

So the database name is created from the test name! Now let me look at how the tests are called to understand the database name pattern. The error shows server_mdm_android_tests_CreateNamedMySQLDS as the database name. Let's look at what test is calling this:

Tool Call: Search files for regex CreateNamedMySQLDS
Status: Completed

No matches found

Assistant

Let me look in the Android tests directory:

Tool Call: List the fleet/server/mdm/android/tests directory's contents
Status: Completed

Folders:

fleet/server/mdm/android/tests/enterprise

Files:

fleet/server/mdm/android/tests/README.md
fleet/server/mdm/android/tests/http.go
fleet/server/mdm/android/tests/integration_software_test.go
fleet/server/mdm/android/tests/testing_utils.go

Tool Call: Read file server/mdm/android/tests/testing_utils.go (lines 100-120)
Status: Completed

	AppConfig   fleet.AppConfig
	AppConfigMu sync.Mutex

	AndroidAPIClient android_mock.Client
	ProxyCallbackURL string
}

func (ts *WithServer) SetupSuite(t *testing.T, dbName string) {
	ts.DS.Datastore = CreateNamedMySQLDS(t, dbName)
	ts.CreateCommonDSMocks()

	ts.AndroidAPIClient = android_mock.Client{}
	ts.createCommonProxyMocks(t)

	logger := kitlog.NewLogfmtLogger(os.Stdout)
	activityModule := activities.NewActivityModule(&ts.DS.DataStore, logger)
	svc, err := service.NewServiceWithClient(logger, &ts.DS, &ts.AndroidAPIClient, "test-private-key", ts.DS.Datastore, activityModule)
	require.NoError(t, err)
	ts.Svc = svc

	ts.Server = runServerForTests(t, logger, &ts.FleetSvc, svc)

Assistant

Now I understand the problem! The issue is a race condition that started appearing when the Android app self-service test was added. Here's what's happening:

1. The Android tests have their own CreateNamedMySQLDS function that was supposed to create a database with a specific name but was ignoring the name parameter and always using the default database.

2. When the Android app self-service test was added on Thursday (commit ff0ba41), it started running in parallel with other tests in CI.

3. Different tests are trying to use different database names for isolation:

   • Some tests call CreateNamedMySQLDS with "server_mdm_android_tests_CreateNamedMySQLDS"
   • The Android test suite calls it with "androidSoftwareTestSuite"

4. But since the Android version was ignoring the name and always using the same database, when tests run in parallel, they're stepping on each other, causing the error about the database not being found.

The fix that has already been created (commit ce42562b4d) correctly addresses this by making the Android CreateNamedMySQLDS function actually use the name parameter by calling mysql.CreateNamedMySQLDS(t, name) instead of mysql.CreateMySQLDS(t).

Summary

The root cause of the database errors that started appearing on Thursday is:

1. The Android tests' CreateNamedMySQLDS function was incorrectly ignoring the database name parameter and always creating a default database instead of a named one.

2. A new Android integration test was added on Thursday (November 14) as part of the Android app self-service backend support (commit ff0ba41), which started running in parallel with other tests in CI.

3. This created a race condition where multiple tests were trying to use different database names for isolation, but the Android tests were all using the same database, causing conflicts and the "database not found" errors.

The fix has already been implemented in branch pass-through-name-on-android-testing-setup (commit ce42562b4d) which makes the Android CreateNamedMySQLDS function properly use the name parameter to create isolated databases for each test suite.