New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write PackageManager Tests #534

Closed
ilexp opened this Issue May 13, 2017 · 12 comments

Comments

1 participant
@ilexp
Member

ilexp commented May 13, 2017

Summary

The PackageManager is a vital component of the Duality editor, as it is the primary way to install and update core, editor and plugins. However, there are no automated tests yet, which definitely should change before doing any major modifications to the system.

Analysis

  • There is nothing static in PackageManager and it accepts target directories to work in, so it should be possible to instantiate one in isolation during the test and then investigate its folders for expected file and file contents.
  • The test should take place with a temporary, local package repository containing only fake packages. No network connection should be required for the tests to be run. No actual NuGet repository reference.
  • Part of the test code will probably be using the NuGet API to create fake NuGet packages.
  • It may be beneficial to factor out some self-contained code parts out of PackageManager in the course of solve this issue.
  • Test candidates:
    • Installing a package
    • Uninstalling a package
    • Updating a package
    • Some special cases of the above where the package in question has dependencies
    • Issue #527
    • Others?
@ilexp

This comment has been minimized.

Member

ilexp commented May 13, 2017

Progress

  • Created a new packagemanager-tests-wip branch to work on this.
  • Extracted PackageSetup and PackageManagerEnvironment classes from PackageManager and re-wired its internals to use them.
  • Created two simple tests for checking non-existent package config files and empty package setups.

Immediate ToDo

  • Extract the whole "write an ApplyUpdate.xml file" deal out of PackageManager.
  • Introduce API to retrieve its data without writing it, so update list contents and serialization can be tested separately.
  • Write tests for:
    • Installing a package
    • Uninstalling a package
    • Updating a package
    • Some special cases of the above where the package in question has dependencies
    • Installing packages that is registered but not installed yet.
      • Specific version
      • Newest version
    • Uninstalling packages that are not registered but still installed.
    • Others?
  • Manually test a fresh package install and update locally to make sure everything is still fine before releasing anything.
    • If issues arise, write unit tests for them to verify they're fixed.
  • Merge back to master.
  • Trigger a binary release.
  • Merge to develop-3.0
  • Write a test that reproduces issue #527 and proceed with fixing it.
@ilexp

This comment has been minimized.

Member

ilexp commented May 14, 2017

Progress

  • Extracted PackageUpdateSchedule from PackageManager.
  • Wrote API docs for PackageUpdateSchedule, PackageSetup and PackageManagerEnvironment.
  • Fixed some places where PackageManager was still using some hardcoded relative path vs. what's exposed in the environment class.
  • Found that it is probably the better choice to use a leaky abstraction for the update schedule and let tests just read the schedule and parse XElement nodes, as opposed to introducing a full class hierarchy to characterize each element just for the tests.

Immediate ToDo

  • Write tests for:
    • Installing a package
    • Uninstalling a package
    • Updating a package
    • Some special cases of the above where the package in question has dependencies
    • Installing packages that is registered but not installed yet.
      • Specific version
      • Newest version
    • Uninstalling packages that are not registered but still installed.
    • Others?
  • Manually test a fresh package install and update locally to make sure everything is still fine before releasing anything.
    • If issues arise, write unit tests for them to verify they're fixed.
  • Merge back to master.
  • Trigger a binary release.
  • Merge to develop-3.0
  • Write a test that reproduces issue #527 and proceed with fixing it.
@ilexp

This comment has been minimized.

Member

ilexp commented May 14, 2017

Progress

  • Figured out how to mock a remote package repository and write tests using an arbitrary set of mock packages on that remote.
  • Wrote tests for:
    • QueryPackageInfo
    • QueryAvailablePackages

Immediate ToDo

  • Write tests for:
    • Installing a package
    • Uninstalling a package
    • Updating a package
    • Some special cases of the above where the package in question has dependencies
    • Installing packages that is registered but not installed yet.
      • Specific version
      • Newest version
    • Uninstalling packages that are not registered but still installed.
    • Others?
  • Manually test a fresh package install and update locally to make sure everything is still fine before releasing anything.
    • If issues arise, write unit tests for them to verify they're fixed.
  • Merge back to master.
  • Trigger a binary release.
  • Merge to develop-3.0
  • Write a test that reproduces issue #527 and proceed with fixing it.

@ilexp ilexp self-assigned this May 14, 2017

@ilexp

This comment has been minimized.

Member

ilexp commented May 20, 2017

Progress

  • Wrote a test for Installing a simple package without dependencies, covering precondition, info retrieval, install invocation, fired events, local setup changes and package update schedule.
  • Restructured some test code in preparation for more tests and test permutations.

Immediate ToDo

  • Extend the InstallPackage test (or create distinct tests that share code) to cover:
    • A Duality package without dependencies (covered)
    • A Duality package with Duality package dependencies.
    • A Duality package with non-Duality package dependencies.
    • A Duality package that is not a plugin, e.g. launcher, core, editor and similar.
  • Write tests for:
    • Uninstalling packages
    • Updating packages
    • Installing packages that is registered but not installed yet, i.e. "package restore".
      • Specific version
      • Newest version
    • Uninstalling packages that are not registered but still installed, i.e. "package removal".
    • Others?
  • Manually test a fresh package install and update locally to make sure everything is still fine before releasing anything.
    • If issues arise, write unit tests for them to verify they're fixed.
  • Merge back to master.
  • Trigger a binary release.
  • Merge to develop-3.0
  • Write a test that reproduces issue #527 and proceed with fixing it.
@ilexp

This comment has been minimized.

Member

ilexp commented May 21, 2017

Progress

  • Restructured InstallPackage test to allow for parameter-provided test case data and re-use all the setup and assertion code surrounding the install.
  • Added InstallPackage test cases for:
    • A Duality package without dependencies (was already covered)
    • A Duality package with Duality package dependencies.
    • A Duality package with non-Duality package dependencies.
    • A Duality package that is not a plugin, e.g. launcher, core, editor and similar.

Immediate ToDo

  • Add InstallPackage test cases for:
    • Duality and Non-Duality Packages including content structures / Resources
    • Packages including source code
  • Write tests for:
    • Uninstalling packages
    • Updating packages
    • Installing packages that is registered but not installed yet, i.e. "package restore".
      • Specific version
      • Newest version
    • Uninstalling packages that are not registered but still installed, i.e. "package removal".
    • Others?
  • Manually test a fresh package install and update locally to make sure everything is still fine before releasing anything.
    • If issues arise, write unit tests for them to verify they're fixed.
  • Merge back to master.
  • Trigger a binary release.
  • Merge to develop-3.0
  • Write a test that reproduces issue #527 and proceed with fixing it.
@ilexp

This comment has been minimized.

Member

ilexp commented May 21, 2017

Progress

  • Extended InstallPackage test cases to include content and source files.

Immediate ToDo

  • Write tests for:
    • Uninstalling packages
      • Without dependencies
      • With dependencies
    • Updating packages
      • Without dependencies
      • With dependencies
    • Installing packages that is registered but not installed yet, i.e. "package restore".
      • Specific version
      • Newest version
    • Uninstalling packages that are not registered but still installed, i.e. "package removal".
    • Others?
  • Manually test a fresh package install and update locally to make sure everything is still fine before releasing anything.
    • If issues arise, write unit tests for them to verify they're fixed.
  • Merge back to master.
  • Trigger a binary release.
  • Merge to develop-3.0
  • Write a test that reproduces issue #527 and proceed with fixing it.

Notes

For uninstall and update package tests, installing packages needs to be part of the testing setup. Since running the updater within the test is a bit flaky and not necessarily desired anyway, the test can simply perform the install and apply the update schedule copy operations manually before proceeding with the actual test code (uninstall / update).

Note that path mapping during install is already covered in InstallPackage tests, so there's no need to re-test special cases with packages being Duality, non-Duality, plugin or non-plugin packages. Just check with / without dependencies for a start.

@ilexp

This comment has been minimized.

Member

ilexp commented May 22, 2017

Progress

  • Refactored PackageManager tests to use a general-purpose package operation test case class to observe the effects of an install, uninstall, update or any combination of them to a local package setup.
  • Added equality operators and methods to the PackageName struct.
  • Exposed limited "apply update" functionality in PackageUpdateSchedule, which is used both in updater updates and tests.
  • Added pre-installed package setup to the InstallPackage test.
  • Cleaned up the InstallPackage test by moving most of the re-usable source code into different methods.
  • Added a PackageEventListener testing class that captures and asserts on PackageManager events.
  • The PackageManager will now check if a package installation would supersede an already installed older version of the same package, and uninstalls that one first if that's the case.
  • Added more diagnostic logs to PackageManager.
  • Added an internal re-routing of NuGet logs to the Duality editor log.
  • Modified PackageManager internal uninstall lock / queue to be default-off to avoid accidentally preventing a package from uninstalling. Instead, the uninstall lock / queue is only activated when uninstalling a Duality package with dependencies.
  • Wrote tests for uninstalling packages.

Immediate ToDo

  • Write tests for:
    • Updating packages
      • Without dependencies
      • With dependencies
    • Installing packages that is registered but not installed yet, i.e. "package restore".
      • Specific version
      • Newest version
    • Uninstalling packages that are not registered but still installed, i.e. "package removal".
    • Others?
  • Do some manual testing for package management and if issues arise, write unit tests for them to verify they're fixed.
    • Fresh install (binary download)
    • Package restore (with fully defined versions)
    • Installing a package with dependencies (Tilemaps sample?)
    • Uninstalling a package with dependencies (Tilemaps sample?)
    • Updating a package with dependencies (Tilemaps sample?)
    • Manual removal of package from config file, then start.
    • Manual addition of package to config file, then start.
    • Manual change of package version in config file, then start.
  • Merge back to master.
  • Trigger a binary release.
  • Merge to develop-3.0
  • Write a test that reproduces issue #527 and proceed with fixing it.

Notes

The number of PackageManager behavior changes slowly increases. Extend manual testing before release.

@ilexp

This comment has been minimized.

Member

ilexp commented May 23, 2017

Progress

  • Wrote tests for updating packages.
  • Wrote a first test for package restore.
  • Restructured a lot of the package manager testing code to make it easier to specify new package operation and restore test cases.
  • Fixed a PackageManager bug related to first package install.

Immediate ToDo

  • Write tests for:
    • Installing packages that is registered but not installed yet, i.e. "package restore".
      • Full restore with specific versions
      • Partial restore with specific versions
      • Partial restore with newest versions
      • Partial restore where one of the new installs will implicitly update a previously installed package
    • Uninstalling packages that are not registered but still installed, i.e. "package removal".
      • With Duality dependencies
      • With non-Duality dependencies
  • Do some manual testing for package management and if issues arise, write unit tests for them to verify they're fixed.
    • Fresh install (binary download)
    • Package restore (with fully defined versions)
    • Installing a package with dependencies (Tilemaps sample?)
    • Uninstalling a package with dependencies (Tilemaps sample?)
    • Updating a package with dependencies (Tilemaps sample?)
    • Manual removal of package from config file, then start.
    • Manual addition of package to config file, then start.
    • Manual change of package version in config file, then start.
  • Merge back to master.
  • Trigger a binary release.
  • Merge to develop-3.0
  • Write a test that reproduces issue #527 and proceed with fixing it.
@ilexp

This comment has been minimized.

Member

ilexp commented May 23, 2017

Progress

  • Wrote tests for:
    • Installing packages that is registered but not installed yet, i.e. "package restore".
      • Full restore with specific versions
      • Partial restore with specific versions
      • Partial restore with newest versions
      • Partial restore where one of the new installs will implicitly update a previously installed package
    • Uninstalling packages that are not registered but still installed, i.e. "package removal".
      • With Duality dependencies
      • With non-Duality dependencies
      • With shared dependencies
      • That other packages depend on
  • Replaced PackageManagers internal uninstall lock / queue with a custom dependency walk enforcing the custom rules that apply to Duality packages.
  • Improved PackageManager uninstall behavior in cases where packages are removed from the config file and uninstalled at startup.

Immediate ToDo

  • Extract the package dependency walking and counting algorithm from PackageManager into its own class.
  • Consider introducing a PackageInfo cache for packages with a fully specified ID and version, as their info should never change (?)
  • Fix that PackageManagerFrontend bug that introduces duplicates into the list view when scrolling while package update states are still being retrieved. Potentially move this to a distinct issue first, depending how big it actually is.
  • Do some manual testing for package management and if issues arise, write unit tests for them to verify they're fixed.
    • Fresh install (binary download)
    • Package restore (with fully defined versions)
    • Installing a package with dependencies (Tilemaps sample?)
    • Uninstalling a package with dependencies (Tilemaps sample?)
    • Updating a package with dependencies (Tilemaps sample?)
    • Manual removal of package from config file, then start.
    • Manual addition of package to config file, then start.
    • Manual change of package version in config file, then start.
  • Merge back to master.
  • Trigger a binary release.
  • Merge to develop-3.0
  • Write a test that reproduces issue #527 and proceed with fixing it.
@ilexp

This comment has been minimized.

Member

ilexp commented May 24, 2017

Progress

  • Extracted PackageDependencyWalker from PackageManager.
  • Extracted PackageCache from PackageManager.
  • Extended package info caching to cover all potentially remote queries and require an explicit cache clear to retrieve new information. This happens every time the package manager dialog is opened and could later be connected to an explicit "refresh" button, removing the default clear entirely.
  • No longer able to reproduce the list view scrolling bug, may have inadvertently fixed that, but keep an eye open for it.
  • Improved package dependency walk implementation to be faster, re-use more data and produce less garbage.
  • More PackageManager API docs.

Immediate ToDo

  • Do some manual testing for package management and if issues arise, write unit tests for them to verify they're fixed.
    • Fresh install (binary download)
    • Package restore (with fully defined versions)
    • Installing a package with dependencies (Tilemaps sample?)
    • Uninstalling a package with dependencies (Tilemaps sample?)
    • Updating a package with dependencies (Tilemaps sample?)
    • Manual removal of package from config file, then start.
    • Manual addition of package to config file, then start.
    • Manual change of package version in config file, then start.
    • Working without package management.
  • Merge back to master.
  • Trigger a binary release.
  • Merge to develop-3.0
  • Write a test that reproduces issue #527 and proceed with fixing it.
@ilexp

This comment has been minimized.

Member

ilexp commented May 24, 2017

Progress

  • PackageCache now operates solely on remote packages.
  • Fixed various smaller bugs that crept in during recent changes.
  • Manually tested the following cases:
    • Fresh install (binary download) ✔️
    • Package restore (with fully defined versions) ✔️
    • Installing a package with dependencies (Tilemaps sample?) ✔️
    • Uninstalling a package with dependencies (Tilemaps sample?) ✔️
    • Updating a package with dependencies (Tilemaps sample?) ✔️
    • Manual removal of package from config file, then start. ✔️
    • Manual addition of package to config file, then start. ✔️
    • Manual change of package version in config file, then start. ✔️
    • Working without package management. ✔️

Immediate ToDo

  • Merge back to master.
  • Trigger a binary release.
  • Test everything post-release and fix any remaining problems ASAP.
  • Merge to develop-3.0
  • Write a test that reproduces issue #527 and proceed with fixing it.
@ilexp

This comment has been minimized.

Member

ilexp commented May 25, 2017

Merged and released. Proceeding with issue #527 and closing this.

@ilexp ilexp closed this May 25, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment