OAR enhancements, including support for recursive scan/loading and IW…

… OARs opt-in (#54)

* Changes to support filtered OAR loading, minus actual filtering.

* Added `scan iwoar`, without the database storage (yet).

* creator_assets storage, basic filtering of assets, root prim/objects

* Actually does the filtering, plus semantic changes on some commands.

saves creators on SCAN by default, doesn't remap uuids by default on load (unless --allow-reassign is used), a bit of cleanup on parameters.

* Oops, gotta filter the contents objects, not outer objects.

And void null reference exception

* Fixed null reference and return value for unfiltered case.

* Fixed counter for case where opt-in was unknown.

* work in progress. first attempt to reserialize modified objects.

* Changes to support filtered OAR loading, minus actual filtering.

* Added `scan iwoar`, without the database storage (yet).

* creator_assets storage, basic filtering of assets, root prim/objects

* Actually does the filtering, plus semantic changes on some commands.

saves creators on SCAN by default, doesn't remap uuids by default on load (unless --allow-reassign is used), a bit of cleanup on parameters.

* Oops, gotta filter the contents objects, not outer objects.

And void null reference exception

* Fixed null reference and return value for unfiltered case.

* Fixed counter for case where opt-in was unknown.

* work in progress. first attempt to reserialize modified objects.

* I didn't need this user asset remapping after all.

* Fixed the EOF error handling in TarArchiveReader to return null on EOF.

* Use more member variables for runtime context, inc m_inventorySerializer

* Track nested depth and replace asset on filtering.

* Doh. Put the Equals test back in to FilterContents, and disable depth.

The latter is just a test for now for tracing.

* Fixed the owner checks not to use the (old, wrong) serialized OwnerID

* Added ConsoleMode class and call to disable Quick Edit mode on consoles.

Was causing consoles to hang if someone clicked on one with the mouse. (I thought this was already coded, it's like a change was lost.) This implementation is probably cleaner than any previous one anyway.

* Added a runtime check to set the console mode only under Windows.

Should allow safe Windows execution, while also allowing a clean Linux startup again.

* Simple URL change in sample INI file

* NUnit to 3 series from NuGet

* Cleanup and fix LRUCache implementation and tests

Some modernization refactoring, documentation, and some bugfixing as revealed by the unit tests.

Removed the bool that Mike had added to LRUCache in favor of just checking the null status of the objectSizes object, like is done elsewhere in the code.

Had to fix the size accumulator not getting size amounts when objectSizes wasn't active: the rest of the code assumed it would be incremented by a size of 1 and it wasn't.

* Inline wait in unit test

This allows for cleaner reporting of aggregate exceptions.

* Cleaning up NREs exposed by updated tests

All because the tests are doing things for simplified setup that the code doesn't really support...

* Fixed remote presence tests

Needed the regions to actually be there so that they could be contacted and respond that they got the agent.

* Enable unit tests in Appveyor

* Document netsh commands needed to pass the unit tests

* Disable the error reports "Failure storing asset" due to already exists.

And "Not storing asset that already exists"

* Fixed inappropriate "ServerWeight is -1 after delta of 0" messages.

* Fixed incorrect nested foreach scaning objects, added counters.

* Factor ReplacePartWithDefaultPrim, filter sculpt/mesh assets.

* Add mesh to scan + counters, fix null reference on unrecognized assets

More error reporting for asset filters and failures

* Terrain texture filtering added.

No recognition of library assets yet though.

* Not InWorldz Library (this gets overridden by User's XML anyway.

* Don't filter Library assets.

* Coded but untested, support for coalesced Contents items (mutable!).

* For easier load iwoar testing, support 'all' variant on nuke command.

* Fixed breakage of OriginalBackupIDs tracking, new UUIDs.

* Let's substitute a plywood texture rather than white.

To avoid masking/hiding replacement cases and making them more difficult to spot.

* Cleaned up noisy logging (commented out for now).

* When reserializing items, ensure they have (correct) owner ID specified.

* Fix viewer textures, texture filtering, default faces, object owners, 

and lots of debugging

* Lowered the threshold for reporting "Low physics" to 25fps

* Added progress updates during load, and disabled debug messages.

* optional oar debugging ("debug oars {level}" command)

* Oar debugging, also fixed nuke to avoid attachments in wildcard case.

* Removed the special cases for mesh textures. 

All filtering is consistent now with plywood. There's a lot of plywood. :(

InWorldz/InWorldz.PhysxPhysics/PhysxScene.cs

@@ -460,7 +460,7 @@ private void UpdateFpsCalc()
             //Console.WriteLine("FPS: {0}", _currFps);
 
             //const float LOW_FPS_THRESHOLD = 54.0f;
-            const float LOW_FPS_THRESHOLD = 45.0f;
+            const float LOW_FPS_THRESHOLD = 25.0f;
             if (_currFps < LOW_FPS_THRESHOLD)
             {
                 m_log.WarnFormat("[InWorldz.PhysxPhysics] Low physics FPS {0}", _currFps);

InWorldz/InWorldz.RemoteAdmin/RemoteAdminPlugin.cs

@@ -336,6 +336,9 @@ public object LoadOARHandler(IList args, IPEndPoint remoteClient)
             String filename = (string)args[2];
             bool allowUserReassignment = Convert.ToBoolean(args[3]);
             bool skipErrorGroups = Convert.ToBoolean(args[4]);
+            bool debug = false;
+            if (args.Count > 5)
+                debug = Convert.ToBoolean(args[5]);
 
             m_log.Info("[RADMIN]: Received Load OAR Administrator Request");
 
@@ -345,7 +348,7 @@ public object LoadOARHandler(IList args, IPEndPoint remoteClient)
                 {
                     IRegionArchiverModule archiver = scene.RequestModuleInterface<IRegionArchiverModule>();
                     if (archiver != null)
-                        archiver.DearchiveRegion(filename, allowUserReassignment, skipErrorGroups);
+                        archiver.DearchiveRegion(filename, allowUserReassignment, skipErrorGroups, null);
                     else
                         throw new Exception("Archiver module not present for scene");
 
@@ -505,6 +508,9 @@ public object RegionRestoreHandler(IList args, IPEndPoint remoteClient)
             String filename = (string)args[2];
             bool allowUserReassignment = Convert.ToBoolean(args[3]);
             bool skipErrorGroups = Convert.ToBoolean(args[4]);
+            bool debug = false;
+            if (args.Count > 5)
+                debug = Convert.ToBoolean(args[5]);
 
             m_log.Info("[RADMIN]: Received Region Restore Administrator Request");
 
@@ -514,7 +520,7 @@ public object RegionRestoreHandler(IList args, IPEndPoint remoteClient)
                 {
                     IRegionArchiverModule archiver = scene.RequestModuleInterface<IRegionArchiverModule>();
                     if (archiver != null)
-                        archiver.DearchiveRegion(filename, allowUserReassignment, skipErrorGroups);
+                        archiver.DearchiveRegion(filename, allowUserReassignment, skipErrorGroups, null);
                     else
                         throw new Exception("Archiver module not present for scene");
 

OpenSim/Base/ConfigurationLoader.cs

@@ -243,7 +243,7 @@ public static IConfigSource DefaultConfig()
                 config.Set("user_source", String.Empty);
                 config.Set("asset_plugin", "OpenSim.Data.SQLite.dll");
                 config.Set("asset_source", "URI=file:Asset.db,version=3");
-                config.Set("LibraryName", "InWorldz Library");
+                config.Set("LibraryName", "Library");
                 config.Set("LibrariesXMLFile", string.Format(".{0}inventory{0}Libraries.xml", Path.DirectorySeparatorChar));
                 config.Set("AssetSetsXMLFile", string.Format(".{0}assets{0}AssetSets.xml", Path.DirectorySeparatorChar));
                 config.Set("dump_assets_to_file", false);

OpenSim/Base/OpenSim.cs

@@ -235,6 +235,10 @@ private void RegisterConsoleCommands()
                                           "debug crossings <level>",
                                           "Turn on crossings debugging (0 or 1)", Debug);
 
+            m_console.Commands.AddCommand("region", false, "debug oars",
+                                          "debug oars <level>",
+                                          "Turn on OAR debugging", Debug);
+
             m_console.Commands.AddCommand("region", false, "change region",
                                           "change region <region name>",
                                           "Change current console region", ChangeSelectedRegion);
@@ -260,9 +264,17 @@ private void RegisterConsoleCommands()
                                           "Save named prim to XML2", SavePrimsXml2);
 
             m_console.Commands.AddCommand("region", false, "load oar",
-                                          "load oar [--ignore-errors] <oar name>",
+                                          "load oar [--allow-reassign] [--ignore-errors] <oar name>",
                                           "Load a region's data from OAR archive", LoadOar);
 
+            m_console.Commands.AddCommand("region", false, "scan iwoar",
+                                          "scan iwoar <oar name>",
+                                          "Scan's a region's data for creator IDs of assets from an InWorldz OAR backup", ScanIWOar);
+
+            m_console.Commands.AddCommand("region", false, "load iwoar",
+                                          "load iwoar <oar name>",
+                                          "Load a region's data from an InWorldz OAR backup, filtering based on opt-in database", LoadIWOar);
+
             m_console.Commands.AddCommand("region", false, "save oar",
                                           "save oar <oar name> <store_assets>",
                                           "Save a region's data to an OAR archive",
@@ -341,8 +353,8 @@ private void RegisterConsoleCommands()
                                           "Collects information about objects posting updates", HandleShow);
 
             m_console.Commands.AddCommand("region", false, "nuke",
-                                          "nuke",
-                                          "Delete all objects owned by the specified UUID", HandleNuke);
+                                          "nuke ownerUUID | all",
+                                          "Delete all objects owned by the specified UUID or all users", HandleNuke);
 
             m_console.Commands.AddCommand("region", false, "blacklist object owner",
                                           "blacklist object owner",
@@ -974,6 +986,23 @@ protected void Debug(string module, string[] args)
 
                     break;
 
+                case "oars":
+                    if (args.Length > 2)
+                    {
+                        int newDebug;
+                        if (int.TryParse(args[2], out newDebug))
+                        {
+                            m_sceneManager.SetOARDebug(newDebug);
+                        }
+                        else
+                        {
+                            m_console.Error("packet debug should be 0..255");
+                        }
+                        m_console.Notice("New OAR debug: " + newDebug.ToString());
+                    }
+
+                    break;
+
                 default:
                     m_console.Error("Unknown debug");
                     break;
@@ -1218,19 +1247,24 @@ public override void HandleNuke(string mod, string[] cmd)
             if (showParams.Length > 0)
             {
                 UUID OwnerID;
-                if (UUID.TryParse(showParams[0], out OwnerID))
+                if (showParams[0] == "all")
+                {
+                    m_sceneManager.NukeObjectsOwnedBy(UUID.Zero);
+                }
+                else
+                if (UUID.TryParse(showParams[0], out OwnerID) && (OwnerID != UUID.Zero))
                 {
-                    m_console.Notice("Deploying nuke...");
                     m_sceneManager.NukeObjectsOwnedBy(OwnerID);
-                    m_console.Notice("Nuke complete.");
                 } else
                 {
-                    m_console.Notice("That does not look like a UUID.");
+                    m_console.Notice("That does not look like a valid UUID.");
+                    return;
                 }
+                m_console.Notice("Nuke complete.");
             }
             else
             {
-                m_console.Notice("You must specify the UUID of the owner.");
+                m_console.Notice("You must specify the UUID of the owner, or 'all'.");
             }
         }
 
@@ -1567,46 +1601,82 @@ protected void LoadXml2(string module, string[] cmdparams)
         }
 
         /// <summary>
-        /// Load a whole region from an opensim archive.
+        /// Scan's a region's data for creator IDs of assets from an InWorldz OAR backup
+        /// scan iwoar [--save] oarname
         /// </summary>
         /// <param name="cmdparams"></param>
-        protected void LoadOar(string module, string[] cmdparams)
+        protected void ScanIWOar(string module, string[] cmdparams)
         {
-            if (cmdparams.Length > 2)
+            string fileName;
+            bool saveCreators;
+            if (cmdparams.Length > 3 && cmdparams[2] == "--save")
             {
-                string fileName;
-                bool ignoreErrors = false;
-                if (cmdparams.Length > 3 && cmdparams[2] == "--ignore-errors")
-                {
-                    ignoreErrors = true;
-                    fileName = cmdparams[3];
-                }
-                else
-                {
-                    ignoreErrors = false;
-                    fileName = cmdparams[2];
-                }
-
-                try
-                {
-                    m_sceneManager.LoadArchiveToCurrentScene(fileName, true, ignoreErrors);
-                }
-                catch (FileNotFoundException)
-                {
-                    m_console.Error("Specified oar not found. Usage: load oar <filename>");
-                }
+                saveCreators = true;
+                fileName = cmdparams[3];
             }
             else
             {
-                try
-                {
-                    m_sceneManager.LoadArchiveToCurrentScene(DEFAULT_OAR_BACKUP_FILENAME, true, false);
-                }
-                catch (FileNotFoundException)
+                saveCreators = false;
+                fileName = cmdparams[2];
+            }
+
+            try
+            {
+                m_sceneManager.ScanSceneForCreators(fileName);
+            }
+            catch (FileNotFoundException)
+            {
+                m_console.Error("Specified oar not found. Usage: load oar <filename>");
+            }
+        }
+
+        /// <summary>
+        /// Load a whole region from an opensim archive with optional table name.
+        /// </summary>
+        /// <param name="cmdparams"></param>
+        protected void LoadOarWithOptions(string module, string[] cmdparams, string optionsTable)
+        {
+            string fileName = DEFAULT_OAR_BACKUP_FILENAME;
+            bool allowReassign = false; // should UUIDs for missing users get reassigned to master avatar
+            bool ignoreErrors = false;
+
+            // Skip "load oar" and "load iwoar" in cmdparams (start at 2).
+            for (int param = 2; param < cmdparams.Length; param++)
+            {
+                switch (cmdparams[param])
                 {
-                    m_console.Error("Default oar not found. Usage: load oar <filename>");
+                    case "--ignore-errors": ignoreErrors = true; break;
+                    case "--allow-reassign": allowReassign = true; break;
+                    default: fileName = cmdparams[param]; break;
                 }
             }
+
+            try
+            {
+                m_sceneManager.LoadArchiveToCurrentScene(fileName, allowReassign, ignoreErrors, optionsTable);
+            }
+            catch (FileNotFoundException)
+            {
+                m_console.Error("OAR file not found. Usage: load oar <filename>");
+            }
+        }
+
+        /// <summary>
+        /// Load a whole region from an opensim archive.
+        /// </summary>
+        /// <param name="cmdparams"></param>
+        protected void LoadOar(string module, string[] cmdparams)
+        {
+            LoadOarWithOptions(module, cmdparams, null);
+        }
+
+        /// <summary>
+        /// Load a region from an opensim archive with filtering based on opt-in status in a database table.
+        /// </summary>
+        /// <param name="cmdparams"></param>
+        protected void LoadIWOar(string module, string[] cmdparams)
+        {
+            LoadOarWithOptions(module, cmdparams, "iwopt");
         }
 
         /// <summary>
@@ -1658,7 +1728,7 @@ protected void LoadExplicitOar(string module, string[] cmdparams)
 
                 IRegionArchiverModule archiver = targetScene.RequestModuleInterface<IRegionArchiverModule>();
                 if (archiver != null)
-                    archiver.DearchiveRegion(cmdparams[3], false, false);
+                    archiver.DearchiveRegion(cmdparams[3], false, false, null);
             }
             catch (FileNotFoundException)
             {

OpenSim/Framework/Serialization/TarArchiveReader.cs

@@ -103,7 +103,7 @@ protected TarHeader ReadHeader()
 
             // If we've reached the end of the archive we'll be in null block territory, which means
             // the next byte will be 0
-            if (header[0] == 0)
+            if (header.Length < 1 || header[0] == 0)
                 return null;
 
             TarHeader tarHeader = new TarHeader();