Support for "packages" in Unity Solution Explorer

[SugoiDev]

Unity 2018 now supports packages.
As explained in #475, those packages have csproj created for them, and those projects are added to the sln, as if they were in the Assets folder.

The problem is that the Unity solution explorer doesn't see those, and we can't use, for example, "scroll from source" with it open.

---

A possible solution would be having more "headers" (top level foldouts) like the Unity3D one: one for each package.
This does not go against the idea of the Unity Solution Explorer (keep things clean, match the internal way of the Unity project, that is more folder-based than project-based, etc), since the packages are self-contained in their own folder, and are outside of the Assets folder (no duplication of displayed folders).

[SugoiDev]

Expanding a bit: some processing will be necessary.
The same way displaying the "Assets" folder for each assembly is redundant, the same can be said for the display of the "packages" folder.

See this example:

Instead of showing "packages" for each package, the explorer should show only the package itself.
Using the same idea that shows the custom assemblies' names to the right of the folder containing it, the package's asmdef files should be shown to the right of the package's folder.

Going further to make it clean, instead of showing the actual folder, the "DisplayName" property could be fetched from the package.json.

Then, instead of having something like packages\com.domain.somepackage@1.0.0 displayed, we would have SomePackage (SomePackage, SomePackage-editor).

(for a package with DisplayName set to SomePackage, and with two asmdef files named SomePackage and SomePackage-editor)

[xtmq]

Could anyone attach a sample solution with a couple of packages? I would take a look.

[SugoiDev]

Give me a few minutes.

[SugoiDev]

Alright, here it is.
It uses 2 packages from Unity (TextMeshPro and the Package Manager Ui), and 2 user packages (the combined shaman stuff and FullSerializer). You can open the package manager from Window -> Package Manager. User packages are also shown there.

To demonstrate package dependencies, FullSerializer has a dependency on Shaman.
This example project does not use the "normal" unity assemblies (Assembly-*). It only has a single asmdef file, besides the packages (which each have a single asmdef as well).

You should see 3 csproj when opening in Rider: the main project (EXAMPLE PROJECT), FullSerializer and Shaman.

The full Library folder is included. If you experience insanity, delete it before opening the Editor.

RU_PACKAGES_EXAMPLE_PROJECT.zip

[citizenmatt]

Thanks! I'm still not sure on how packages are meant to be used - are they intended to be treated like Assets? So anything in a package is essentially the same as something in the Assets subfolder? Are they user editable, or is the Packages folder to be treated as read-only?

And how would you expect to see the packages in the Unity Explorer view? I can see renaming the root node to "Assets", and then adding a second root node next to Assets called "Packages". Would we list all of a project's packages here, or just the ones that were installed locally into the Packages folder? If we're looking at all of them, are there going to be any "global" packages that include source? Presumably these should be read only?

[SugoiDev]

I'm also not totally sure, but at least a few things are a bit more certain.

are they intended to be treated like Assets

A package is a collection of source code and assemblies. Correct info from next comment: "A package is actually a collection of Unity assets, which may be only code and assemblies, "regular" assets, or a mix of them.". The source files will be compiled using the same preprocessor directives you have defined in your project at compile time.

Previously, you would have to keep those in sync when building your own DLLs for distribution (or your own use). For asset store packages supporting multiple Unity versions, this can get pretty insane very fast.

So anything in a package is essentially the same as something in the Assets subfolder?

Yes. But packages can't contain anything other than code and assemblies. They also aren't part of the AssetDatabase (so their source files won't have a UnityEngine.Object associated with them). (Actually, they can contain anything, but it won't be exposed to Unity, since the AssetDatabase doesn't know anything about those) Correct info from next comment: Packages are first-class citizens and work as if dropped in the assets folder. Further support (and project view improvements) is being worked on.

Are they user editable, or is the Packages folder to be treated as read-only?

Packages from the global repository (from Unity), are supposed to be treated as read-only. They are downloaded from unity's servers (for now, at least).

Current repositories: https://download.staging-packages.unity.com (stable, release) and https://staging-packages.unity.com (alpha, beta, experimental) Correct info from next comment: Repository is https://packages.unity.com (https://staging-packages.unity.com should not be used).

And how would you expect to see the packages in the Unity Explorer view? I can see renaming the root node to "Assets", and then adding a second root node next to Assets called "Packages".

That seems the best extension of the current view, yes.

Would we list all of a project's packages here, or just the ones that were installed locally into the Packages folder?

I think only the local packages (from the packages folder). Global packages don't have an associated csproj created for them. They are compiled and their references are injected by Unity automatically.

If we're looking at all of them, are there going to be any "global" packages that include source? Presumably these should be read only?

There are already "global" packages with source. But they should be treated as read-only, yes.
For example, the com.unity.package-manager-ui is full source code (has no compiled assemblies).

---

Please note that I don't have any "insider" info. This is just from my usage and exploration. I'm pretty excited about these features, so I'm trying to keep as up-to-date as possible in those matters. We do have insider info now. See next comment!

I believe the packages system will be pushed pretty much as is in the next releases, since it is already very functional. In the near future, I'm expecting to see a way to have asset store assets being installed as packages (with an intermediate step to copy over normal assets to the Assets/ folder, like prefabs and scriptable objects).

[mbargiel]

Hi SugoiDev! Nice detective work!

As an insider myself, I'd like to provide a bit more information, and correct some of your assumptions.

A package is a collection of source code and assemblies. The source files will be compiled using the same preprocessor directives you have defined in your project at compile time.
 
Previously, you would have to keep those in sync when building your own DLLs for distribution (or your own use). For asset store packages, supporting multiple Unity versions, this can get pretty insane very fast.

A package is actually a collection of Unity assets, which may be only code and assemblies, "regular" assets, or a mix of them.

Source files are compiled into separate assemblies using the new-ish .asmdef (see https://docs.unity3d.com/Manual/ScriptCompilationAssemblyDefinitionFiles.html), and any package code not associated to an .asmdef file will still be compiled into a separate package-specific assembly. All these assemblies are accessible to scripts under Assets (with the Editor restrictions applying accordingly) so all of this should not change how projects use that code.

Yes. But packages can't contain anything other than code and assemblies. They also aren't part of the AssetDatabase (so their source files won't have a UnityEngine.Object associated with them). (Actually, they can contain anything, but it won't be exposed to Unity, since the AssetDatabase doesn't know anything about those)

Well, as stated above, that's inaccurate.

In fact, the AssetDatabase does know about all of the assets in packages. The only difference is that these assets's path doesn't start with "Assets/" but "Packages//". Unfortunately, quite a lot of the Unity code expects assets to be in "Assets" and this is why there are usability issues like not being able to find assets in packages inside the Project Hierarchy view or in the Object Browser - yet. This is all being worked on at this time.

Packages from the global repository (from Unity), are supposed to be treated as read-only. They are downloaded from unity's servers (for now, at least).

This is right!

Current repositories: https://download.staging-packages.unity.com (stable, release) and https://staging-packages.unity.com (alpha, beta, experimental)

... but this isn't quite right. The current production repository is located at https://packages.unity.com, while https://staging-packages.unity.com is for internal validation, experimental packages, etc. Please note that we're still in the process of clarifying the proper process for all of this. For instance, the staging repository may be eliminated at some point, and preview packages (in various states of experimental, alpha, beta) will eventually be published to the production repository. (They will be clearly tagged and versioned as such.)

And how would you expect to see the packages in the Unity Explorer view? I can see renaming the root node to "Assets", and then adding a second root node next to Assets called "Packages".

This is exactly how we're going to present things in the Project Hierarchy view, and in the generated VS solutions.

 I think only the local packages (from the packages folder). Global packages don't have an associated csproj created for them. They are compiled and their references are injected by Unity automatically.

Yep, that's the intent. We are working on a way to make packages editable. I can't say too much about that at this time, but packages present in the Packages directory will be considered part of your project, and therefore fully editable. Other packages are considered external references and will not pollute the VS solution with csproj referencing readonly files. Of course, it's always possible to decompile managed assemblies using awesome tools like JetBrains dotPeek ;)

(Good job on finding out the "local package" feature, by the way! It won't be named like that though...)

There are already "global" packages with source. But they should be treated as read-only, yes.
For example, the com.unity.package-manager-ui is full source code (has no compiled assemblies).

Yes! In fact, we're pushing for package authors to ship their packages as source code, but in some cases, there will be a transition period, especially for packages shipping features that were previously Unity extensions.

Providing packages with source code has a bunch of advantages, including:

• packages can use UNITY_* preprocessor defines (or even other defines),
• debugging is easier/more efficient,
• eventually, when packages are made editable, they can be edited/fixed/customized in-place for your project.

There are downsides, however. For instance, because of the compilation steps, it may take a bit longer to open a project. However the split into multiple assemblies minimizes this overhead, and modifying scripts in Assets will not cause package assemblies to be recompiled.

In any case, stay tuned for official announcements about the package manager! We're really excited with the reception so far, and we're working really hard to address issues and add all these new features!

[SugoiDev]

@mbargiel Ah, thanks a lot for all the clarifications.
Also, thanks for correcting the urls. I mixed them up!

Now, if I may, I'll sidetrack this issue just a bit to ask for more info on a point that interests me a lot!

In fact, the AssetDatabase does know about all of the assets in packages...

That's incredible! Is the AssetDatabase already aware of package assets, or is it a future plan?
I intercept asset opening, and couldn't get an object or an instanceID for assets (source files) in a "local" package. Also couldn't get references to scripts in packages aren't to be stored my serialized fields (references to MonoScript objects).

Actually, I had #475 because package script files had no valid instanceID/object (so an OnOpenAsset method had no way of retrieve an object using AssetDatabase.getAssetPath(instanceID)).

I noticed that TextMeshPro has an additional step where you need to run a menu command for it to copy over its regular assets to the Assets/ folder. Will those steps be eliminated in the future (or was it already and I failed to notice)?

[mbargiel]

@SugoiDev The AssetDatabase is already aware of assets in packages. In theory, you could already hack references to these assets by using their fileID and GUIDs... ugh, not terribly user-friendly.

I'm not sure why you've been having these problems honestly. Assets in packages should have instance IDs the same as project assets do. However not all AssetDatabase methods will properly consider all assets so some APIs may not work as expected, yet. One thing I'm fairly certain of is that you can use the AssetPathToGUID, GUIDToAssetPath and LoadAssetAtPath methods.

Regarding TMP, this has not yet been solved. I can't tell you whether this is going to change or not, but one thing is sure, packages fetched from external repositories will always be immutable, so any mutable assets like Settings or Samples will still need to be imported into the project, for the foreseeable future. I think the process has been made fairly seamless in the case of TMP. If you have any issues with it, please don't hesitate to post on the Unity forum about it.

[SugoiDev]

@mbargiel

I'm not sure why you've been having these problems honestly

I used GetAssetPath with the instanceID provided by a method decorated with the OnOpenAsset attribute. If I remember correctly, the instanceID was always -1 or 0, not sure now. Maybe that callback doesn't yet support packages?

Since I'm opening script files, I just get the file path from the stacktrace that I already have from the Unity log callback for free.
I'll check again and post in the forums if it is still not working.

If you have any issues with it, please don't hesitate to post on the Unity forum about it.
No issue here. It works.

I imagine that "Editor Default Resources" and "Gizmos" will have to be copied over as well, yes?
Many asset store packages use those folders so I'm expecting to see some turbulence with that when asset store devs begin transitioning their assets into packages. Same with hard-coded "Assets" string trimming and length.

---

I'll stop with the hijack of this issue now and will take any more questions I might have to the forums. Thanks for taking the time to reply to us here!

[SugoiDev]

@mbargiel Created a thread on the forums. It seems none of those methods are working with "local" package files on my end.

[citizenmatt]

@mbargiel Thanks for the very thorough answers, very helpful. From this, I'd say that we're still in wait-and-see territory about adding new package facing features. We'll definitely be keeping an eye on this, though, and will add support as and when it becomes appropriate.

[van800]

@xtmq Please simplify editing Packages/manifest.json.
This file is not inside Assets, so I need to switch to Solution Explorer with Show All Files to edit it.

[van800]

New Unity versions have packages in the view together with Assets: