From 03475630ff57e90a66e345630ed54e91a7cda23c Mon Sep 17 00:00:00 2001 From: Andrew Skowronski Date: Thu, 23 Oct 2025 15:24:09 -0400 Subject: [PATCH 1/6] WIP Doc tweaks --- Analyzer/README.md | 2 +- Analyzer/SQLite/Parsers/SerializedFileParser.cs | 2 +- Documentation/analyze-examples.md | 4 +++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Analyzer/README.md b/Analyzer/README.md index 953501a..728c865 100644 --- a/Analyzer/README.md +++ b/Analyzer/README.md @@ -187,7 +187,7 @@ File, then rows are added to both the `objects` table and the `meshes` table. T Each supported Unity object type follows the same pattern: * A Handler class in the SQLite/Handlers (e.g. [MeshHandler.cs](./SQLite/Handler/MeshHandler.cs). -* The registration of the handler in the m_Handlers dictionary in [SQLiteWriter.cs](./SQLite/SQLiteWriter.cs). +* The registration of the handler in the m_Handlers dictionary in [SerializedFileSQLiteWriter.cs](./SQLite/SQLiteWriter.cs). * SQL statements defining extra tables and views associated with the type, e.g. [Mesh.sql](./SQLite/Resources/Mesh.sql). * A Reader class that uses RandomAccessReader to read properties from the serialized object. e.g. [Mesh.cs](./SerializedObjects/Mesh.cs). diff --git a/Analyzer/SQLite/Parsers/SerializedFileParser.cs b/Analyzer/SQLite/Parsers/SerializedFileParser.cs index 2a40f30..2773f1a 100644 --- a/Analyzer/SQLite/Parsers/SerializedFileParser.cs +++ b/Analyzer/SQLite/Parsers/SerializedFileParser.cs @@ -64,7 +64,7 @@ bool ShouldIgnoreFile(string file) private static readonly HashSet IgnoredExtensions = new() { ".txt", ".resS", ".resource", ".json", ".dll", ".pdb", ".exe", ".manifest", ".entities", ".entityheader", - ".ini", ".config" + ".ini", ".config", ".hash" }; bool ProcessFile(string file, string rootDirectory) diff --git a/Documentation/analyze-examples.md b/Documentation/analyze-examples.md index 11e5def..89ab091 100644 --- a/Documentation/analyze-examples.md +++ b/Documentation/analyze-examples.md @@ -4,7 +4,9 @@ This topic gives some examples of using the SQLite output of the UnityDataTools The command line arguments to invoke Analyze are documented [here](../UnityDataTool/README.md#analyzeanalyse). -The definition of the views, and some internal details about how Analyze is implemented, can be found [here](../Analyzer/README.md). +The definition of the views, and some internal details about how Analyze is implemented, can be found [here](../Analyzer/README.md). + +See [Addressables Build Reports](addressables-build-reports.md) for information about the tables and views created for Addressables build reports. ## Running Queries from the Command line From 17ea8df27db2e3a61aebbd86c794aab2b14524c0 Mon Sep 17 00:00:00 2001 From: Andrew Skowronski Date: Thu, 23 Oct 2025 17:12:30 -0400 Subject: [PATCH 2/6] Restore "assets" population and documentation updates The AssetBundleHandler got lost in recent code change so restore it. Add a few cross links to the new Addressables Build Layout parsing. Fix the section "Matching content back to the source asset" because it wasn't explaining that some source asset path are available in the build output (for AssetBundle explicit assets, and Player build scenes) --- Analyzer/README.md | 12 ++- .../SQLite/Handlers/AssetBundleHandler.cs | 96 +++++++++++++++++++ .../Writers/SerializedFileSQLiteWriter.cs | 1 + Documentation/analyze-examples.md | 13 +-- 4 files changed, 115 insertions(+), 7 deletions(-) create mode 100644 Analyzer/SQLite/Handlers/AssetBundleHandler.cs diff --git a/Analyzer/README.md b/Analyzer/README.md index 728c865..52a09af 100644 --- a/Analyzer/README.md +++ b/Analyzer/README.md @@ -187,8 +187,18 @@ File, then rows are added to both the `objects` table and the `meshes` table. T Each supported Unity object type follows the same pattern: * A Handler class in the SQLite/Handlers (e.g. [MeshHandler.cs](./SQLite/Handler/MeshHandler.cs). -* The registration of the handler in the m_Handlers dictionary in [SerializedFileSQLiteWriter.cs](./SQLite/SQLiteWriter.cs). +* The registration of the handler in the m_Handlers dictionary in [SerializedFileSQLiteWriter.cs](./SQLite/SerializedFileSQLiteWriter.cs). * SQL statements defining extra tables and views associated with the type, e.g. [Mesh.sql](./SQLite/Resources/Mesh.sql). * A Reader class that uses RandomAccessReader to read properties from the serialized object. e.g. [Mesh.cs](./SerializedObjects/Mesh.cs). It would be possible to extend the Analyze library to add additional columns for the existing types, or by following the same pattern to add additional types. The [dump](../UnityDataTool/README.md#dump) feature of UnityDataTool is a useful way to see the property names and other details of the serialization for a type. Based on that information, code in the Reader class can use the RandomAccessReader to retrieve those properties to bring them into the SQLite database. + +## Supporting Other File Formats + +Another direction of possible extension is to support analyzing additional file formats, beyond Unity SerializedFiles. + +This the approach taken to analyze Addressables Build Layout files, which are JSON files using the format defined in [BuildLayout.cs](./SQLite/Parsers/Models/BuildLayout.cs). + +Support for another file format could be added by deriving an additional class from SQLiteWriter and implementing a class derived from ISQLiteFileParser. Then follow the existing code structure convention to add new Commands (derived from AbstractCommand) and Resource .sql files to establish additional tables in the database. + +An example of another file format that could be useful to support, as the tool evolves, are the yaml [.manifest files](https://docs.unity3d.com/Manual/assetbundles-file-format.html), generated by BuildPipeline.BuildAssetBundles(). diff --git a/Analyzer/SQLite/Handlers/AssetBundleHandler.cs b/Analyzer/SQLite/Handlers/AssetBundleHandler.cs new file mode 100644 index 0000000..fa4b3d4 --- /dev/null +++ b/Analyzer/SQLite/Handlers/AssetBundleHandler.cs @@ -0,0 +1,96 @@ +using System; +using System.Text.RegularExpressions; +using Microsoft.Data.Sqlite; +using UnityDataTools.Analyzer.SerializedObjects; +using UnityDataTools.FileSystem.TypeTreeReaders; + +namespace UnityDataTools.Analyzer.SQLite.Handlers; + +public class AssetBundleHandler : ISQLiteHandler +{ + SqliteCommand m_InsertCommand; + private SqliteCommand m_InsertDepCommand; + private Regex m_SceneNameRegex = new Regex(@"([^//]+)\.unity"); + + public void Init(SqliteConnection db) + { + using var command = db.CreateCommand(); + command.CommandText = Resources.AssetBundle; + command.ExecuteNonQuery(); + + m_InsertCommand = db.CreateCommand(); + + m_InsertCommand.CommandText = "INSERT INTO assets(object, name) VALUES(@object, @name)"; + m_InsertCommand.Parameters.Add("@object", SqliteType.Integer); + m_InsertCommand.Parameters.Add("@name", SqliteType.Text); + + m_InsertDepCommand = db.CreateCommand(); + + m_InsertDepCommand.CommandText = "INSERT INTO asset_dependencies(object, dependency) VALUES(@object, @dependency)"; + m_InsertDepCommand.Parameters.Add("@object", SqliteType.Integer); + m_InsertDepCommand.Parameters.Add("@dependency", SqliteType.Integer); + } + + public void Process(Context ctx, long objectId, RandomAccessReader reader, out string name, out long streamDataSize) + { + var assetBundle = AssetBundle.Read(reader); + + foreach (var asset in assetBundle.Assets) + { + if (!assetBundle.IsSceneAssetBundle) + { + var fileId = ctx.LocalToDbFileId[asset.PPtr.FileId]; + var objId = ctx.ObjectIdProvider.GetId((fileId, asset.PPtr.PathId)); + m_InsertCommand.Transaction = ctx.Transaction; + m_InsertCommand.Parameters["@object"].Value = objId; + m_InsertCommand.Parameters["@name"].Value = asset.Name; + m_InsertCommand.ExecuteNonQuery(); + + for (int i = asset.PreloadIndex; i < asset.PreloadIndex + asset.PreloadSize; ++i) + { + var dependency = assetBundle.PreloadTable[i]; + var depFileId = ctx.LocalToDbFileId[dependency.FileId]; + var depId = ctx.ObjectIdProvider.GetId((depFileId, dependency.PathId)); + m_InsertDepCommand.Transaction = ctx.Transaction; + m_InsertDepCommand.Parameters["@object"].Value = objId; + m_InsertDepCommand.Parameters["@dependency"].Value = depId; + m_InsertDepCommand.ExecuteNonQuery(); + } + } + else + { + var match = m_SceneNameRegex.Match(asset.Name); + + if (match.Success) + { + var sceneName = match.Groups[1].Value; + var objId = ctx.ObjectIdProvider.GetId((ctx.SerializedFileIdProvider.GetId(sceneName), 0)); + m_InsertCommand.Transaction = ctx.Transaction; + m_InsertCommand.Parameters["@object"].Value = objId; + m_InsertCommand.Parameters["@name"].Value = asset.Name; + m_InsertCommand.ExecuteNonQuery(); + } + } + } + + name = assetBundle.Name; + streamDataSize = 0; + } + + public void Finalize(SqliteConnection db) + { + using var command = new SqliteCommand(); + command.Connection = db; + command.CommandText = "CREATE INDEX asset_dependencies_object ON asset_dependencies(object)"; + command.ExecuteNonQuery(); + + command.CommandText = "CREATE INDEX asset_dependencies_dependency ON asset_dependencies(dependency)"; + command.ExecuteNonQuery(); + } + + void IDisposable.Dispose() + { + m_InsertCommand?.Dispose(); + m_InsertDepCommand?.Dispose(); + } +} \ No newline at end of file diff --git a/Analyzer/SQLite/Writers/SerializedFileSQLiteWriter.cs b/Analyzer/SQLite/Writers/SerializedFileSQLiteWriter.cs index 56d5a40..3677012 100644 --- a/Analyzer/SQLite/Writers/SerializedFileSQLiteWriter.cs +++ b/Analyzer/SQLite/Writers/SerializedFileSQLiteWriter.cs @@ -35,6 +35,7 @@ public class SerializedFileSQLiteWriter : IDisposable { "Shader", new ShaderHandler() }, { "AudioClip", new AudioClipHandler() }, { "AnimationClip", new AnimationClipHandler() }, + { "AssetBundle", new AssetBundleHandler() }, { "PreloadData", new PreloadDataHandler() }, }; diff --git a/Documentation/analyze-examples.md b/Documentation/analyze-examples.md index 89ab091..67588d3 100644 --- a/Documentation/analyze-examples.md +++ b/Documentation/analyze-examples.md @@ -6,8 +6,6 @@ The command line arguments to invoke Analyze are documented [here](../UnityDataT The definition of the views, and some internal details about how Analyze is implemented, can be found [here](../Analyzer/README.md). -See [Addressables Build Reports](addressables-build-reports.md) for information about the tables and views created for Addressables build reports. - ## Running Queries from the Command line You can find data in the SQLite database by running SQL queries. @@ -222,9 +220,13 @@ This is a large subject, see [Comparing Builds](comparing-builds.md). ## Example: Matching content back to the source asset -UnityDataTool works on the output of a Unity build, which, by its very nature, only contains the crucial data needed to efficiently load built content in the Player. So it does not include any information about the assets and scenes in the project that was used to create that build. However you may want to match content back to the original source asset or scene. For example if the size of an AssetBundle has unexpectedly changed between builds then you may want to track down which source assets could be responsible for that change. Or you may want to confirm that some particular image has been included in the build. +UnityDataTool works on the output of a Unity build, which, by its very nature, only contains the crucial data needed to efficiently load built content in the Player. It does not include complete information about the assets and scenes in the project that was used to create that build. You may want to match content back to the original source asset or scene. For example if the size of an AssetBundle has unexpectedly changed between builds then you may want to track down which source assets could be responsible for that change. Or you may want to confirm that some particular image has been included in the build. + +For AssetBundles partial asset information can be found in the m_Containers list, inside the AssetBundle object. This records assets that were explicitly added to AssetBundles. In the database this can be found in the `assets` table. However, asset that are included in the build implicitly (because they are referenced from the explicitly added assets) will not be recorded anywhere in the AssetBundle content. -In many cases the source asset can be inferred based on your specific knowledge of your project, and how the build was configured. For example the level files in a Player build match the Scenes in the Build Profile Scene list. And the content of AssetBundles is driven from the assignment of specific assets to those AssetBundles (or Addressable groups). +Similarly for a player build the only paths populated in the `assets` table are the scenes from the Build Profile Scene List. The paths of the assets in the sharedAsset files is not recorded anywhere in the build output. + +In many cases the source asset can be inferred based on your specific knowledge of your project, and how the build was configured. For example the level files in a Player build match the Scenes in the Build Profile Scene list. And the content of AssetBundles is driven from the assignment of specific assets to those AssetBundles (or Addressable groups), along with assets they depend on. Also, in many cases the name of objects matches the file name of the asset. For example the Texture2D "red" object probably comes from a file named red.png somewhere in the project. @@ -237,5 +239,4 @@ Examples of alternative sources of build information: * The [BuildReport](https://docs.unity3d.com/ScriptReference/Build.Reporting.BuildReport.html) has detailed source information in the PackedAssets section. The [BuildReportInspector](https://github.com/Unity-Technologies/BuildReportInspector) is a useful way to view data from the BuildReport. * The Editor log reports a lot of information during a build. * Regular AssetBundle builds create [.manifest files](https://docs.unity3d.com/Manual/assetbundles-file-format.html), which contain information about the source assets and types. -* Addressable builds do not produce BuildReport files, nor .manifest files. But there is similar reporting, for example the [Build Layout Report](https://docs.unity3d.com/Packages/com.unity.addressables@2.4/manual/BuildLayoutReport.html). - +* Addressable builds do not produce BuildReport files, nor .manifest files. But UnityDataTools supports analyzing the [Addressables Build Reports](addressables-build-reports.md) and will populate the `addressables_build_explicit_assets` and `addressables_build_implicit_assets` tables. From fc373547211e4f5f39d72e8e389c1f495fd87d1c Mon Sep 17 00:00:00 2001 From: Andrew Skowronski <86242170+SkowronskiAndrew@users.noreply.github.com> Date: Thu, 23 Oct 2025 17:19:48 -0400 Subject: [PATCH 3/6] Update Documentation/analyze-examples.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- Documentation/analyze-examples.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/analyze-examples.md b/Documentation/analyze-examples.md index 67588d3..a61634d 100644 --- a/Documentation/analyze-examples.md +++ b/Documentation/analyze-examples.md @@ -222,7 +222,7 @@ This is a large subject, see [Comparing Builds](comparing-builds.md). UnityDataTool works on the output of a Unity build, which, by its very nature, only contains the crucial data needed to efficiently load built content in the Player. It does not include complete information about the assets and scenes in the project that was used to create that build. You may want to match content back to the original source asset or scene. For example if the size of an AssetBundle has unexpectedly changed between builds then you may want to track down which source assets could be responsible for that change. Or you may want to confirm that some particular image has been included in the build. -For AssetBundles partial asset information can be found in the m_Containers list, inside the AssetBundle object. This records assets that were explicitly added to AssetBundles. In the database this can be found in the `assets` table. However, asset that are included in the build implicitly (because they are referenced from the explicitly added assets) will not be recorded anywhere in the AssetBundle content. +For AssetBundles partial asset information can be found in the m_Containers list, inside the AssetBundle object. This records assets that were explicitly added to AssetBundles. In the database this can be found in the `assets` table. However, assets that are included in the build implicitly (because they are referenced from the explicitly added assets) will not be recorded anywhere in the AssetBundle content. Similarly for a player build the only paths populated in the `assets` table are the scenes from the Build Profile Scene List. The paths of the assets in the sharedAsset files is not recorded anywhere in the build output. From 3810f3d2da7385a0051b5fef9bc304c179c35a82 Mon Sep 17 00:00:00 2001 From: Andrew Skowronski Date: Thu, 23 Oct 2025 17:24:33 -0400 Subject: [PATCH 4/6] Fix link --- Analyzer/README.md | 6 +++--- Documentation/analyze-examples.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Analyzer/README.md b/Analyzer/README.md index 52a09af..574711b 100644 --- a/Analyzer/README.md +++ b/Analyzer/README.md @@ -186,10 +186,10 @@ important types like Meshes, Shaders, Texture2D and AnimationClips. For example File, then rows are added to both the `objects` table and the `meshes` table. The meshes table contains columns that only apply to Mesh objects, for example the number of vertices, indices, bones, and channels. The `mesh_view` is a view that joins the `objects` table with the `meshes` table, so that you can see all the properties of a Mesh object in one place. Each supported Unity object type follows the same pattern: -* A Handler class in the SQLite/Handlers (e.g. [MeshHandler.cs](./SQLite/Handler/MeshHandler.cs). -* The registration of the handler in the m_Handlers dictionary in [SerializedFileSQLiteWriter.cs](./SQLite/SerializedFileSQLiteWriter.cs). +* A Handler class in the SQLite/Handlers, e.g. [MeshHandler.cs](./SQLite/Handler/MeshHandler.cs). +* The registration of the handler in the m_Handlers dictionary in [SerializedFileSQLiteWriter.cs](./SQLite/Writers/SerializedFileSQLiteWriter.cs). * SQL statements defining extra tables and views associated with the type, e.g. [Mesh.sql](./SQLite/Resources/Mesh.sql). -* A Reader class that uses RandomAccessReader to read properties from the serialized object. e.g. [Mesh.cs](./SerializedObjects/Mesh.cs). +* A Reader class that uses RandomAccessReader to read properties from the serialized object, e.g. [Mesh.cs](./SerializedObjects/Mesh.cs). It would be possible to extend the Analyze library to add additional columns for the existing types, or by following the same pattern to add additional types. The [dump](../UnityDataTool/README.md#dump) feature of UnityDataTool is a useful way to see the property names and other details of the serialization for a type. Based on that information, code in the Reader class can use the RandomAccessReader to retrieve those properties to bring them into the SQLite database. diff --git a/Documentation/analyze-examples.md b/Documentation/analyze-examples.md index a61634d..4fb74f1 100644 --- a/Documentation/analyze-examples.md +++ b/Documentation/analyze-examples.md @@ -4,7 +4,7 @@ This topic gives some examples of using the SQLite output of the UnityDataTools The command line arguments to invoke Analyze are documented [here](../UnityDataTool/README.md#analyzeanalyse). -The definition of the views, and some internal details about how Analyze is implemented, can be found [here](../Analyzer/README.md). +The definition of the views, and some internal details about how Analyze is implemented, can be found [here](../Analyzer/README.md). ## Running Queries from the Command line From f1c7630aa09ecb02d18c8409b22d8c890b31873f Mon Sep 17 00:00:00 2001 From: Tim Thomas Date: Fri, 24 Oct 2025 14:24:20 -0500 Subject: [PATCH 5/6] add find implicit assets sql --- Documentation/addressables-build-reports.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/Documentation/addressables-build-reports.md b/Documentation/addressables-build-reports.md index 0883e0f..de05faf 100644 --- a/Documentation/addressables-build-reports.md +++ b/Documentation/addressables-build-reports.md @@ -114,6 +114,19 @@ You can analyze a directory with both asset bundles (*.bundle) and json files (* Once the data is in the database, you can run queries to analyze your Addressables build: + +#### Find all implicit assets for an explicit asset +```sql +-- Find implicitly included assets for a given explicit asset id +SELECT a.explicit_asset_id, b.id, b.asset_path, b.asset_path + FROM addressables_build_explicit_asset_internal_referenced_other_assets a, + addressables_build_data_from_other_assets b + WHERE a.internal_referenced_other_asset_rid = b.id + AND a.build_id = b.build_id; + AND a.explicit_asset_id = 5092 + AND a.build_id = 3; +``` + #### Find the cache name for an addressables bundle Addressables renames bundles to make it possible to do content updates. Internally bundles are still named by their internal hash and are cached based upon this name. If you want to lookup how a remote bundle will be cached in Unity's [cache](https://docs.unity3d.com/ScriptReference/Caching.html) you can use the addressables_build_cached_bundles view. ```sql @@ -165,4 +178,4 @@ group by a.id; SELECT name, start_time, duration, error FROM addressables_builds WHERE coalesce(error, '') != ''; -``` \ No newline at end of file +``` From b5351a8d2747aec8de54b098c752eb8512ae09f7 Mon Sep 17 00:00:00 2001 From: Tim Thomas Date: Fri, 24 Oct 2025 14:26:21 -0500 Subject: [PATCH 6/6] Fix addressables table name --- Documentation/analyze-examples.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/analyze-examples.md b/Documentation/analyze-examples.md index 4fb74f1..e3dd6f2 100644 --- a/Documentation/analyze-examples.md +++ b/Documentation/analyze-examples.md @@ -239,4 +239,4 @@ Examples of alternative sources of build information: * The [BuildReport](https://docs.unity3d.com/ScriptReference/Build.Reporting.BuildReport.html) has detailed source information in the PackedAssets section. The [BuildReportInspector](https://github.com/Unity-Technologies/BuildReportInspector) is a useful way to view data from the BuildReport. * The Editor log reports a lot of information during a build. * Regular AssetBundle builds create [.manifest files](https://docs.unity3d.com/Manual/assetbundles-file-format.html), which contain information about the source assets and types. -* Addressable builds do not produce BuildReport files, nor .manifest files. But UnityDataTools supports analyzing the [Addressables Build Reports](addressables-build-reports.md) and will populate the `addressables_build_explicit_assets` and `addressables_build_implicit_assets` tables. +* Addressable builds do not produce BuildReport files, nor .manifest files. But UnityDataTools supports analyzing the [Addressables Build Reports](addressables-build-reports.md) and will populate the `addressables_build_explicit_assets` and `addressables_build_data_from_other_assets` tables.