New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.NET Core paket and dotnet/sdk #2875

Open
enricosada opened this Issue Oct 30, 2017 · 10 comments

Comments

Projects
None yet
4 participants
@enricosada
Copy link
Collaborator

enricosada commented Oct 30, 2017

Paket depends now on .NET framework (win) and mono (unix/mac).

This is really good because the normal paket.exe console app can run xplat if mono/.net is installed.

For some scenario, like .net core development (or fable) with new dotnet/sdk (new csproj/fsproj), the mono/.net shouldnt be needed.

To remove that deps of paket in these scenario there are some points to address:

  • xplat paket.exe console app, based on .net core
  • boostrapping process (from clean repo)
    • minimal set of files (and size) needed to be committed on repo
    • self updating of boostrapping process
    • binaries sources: nuget/github
    • chicken/egg. how to boostrap the boostrapper
  • execute paket from command line, xplat. with good dev UX
    • magic mode
  • auto-update of paket
    • frequent release schedule -> minimize size/download
    • binaries sources: nuget/github
  • minimize UX changes for users
  • maintenance.
    • try to avoid two version (net/netcore) because old sdk project can also use .net core based paket
  • tradeoff downloadsize vs dependencies

Atm ihmo boostrapping and auto-updates are the real issue

Current Plan

  • make paket console app to .net core so works the same as now if published as SCD (wip #2918)
  • implement dotnetcli tools, per repo ( ref #2919 )
  • modify build.fsx to put in Paket nuget both paket .net core FDD (in tools/netcoreapp2.0) and paket .net fw (in tools/net451)
  • reuse paket SCD itself as boostrapper, or use a dotnetclitool, to add paket in PATH

Notes

notes about console deployment options on .net core

Options are:

  • SCD, self contained deployment
    • bigger, one release for platform (unix/win/macosx), require just .net core runtime 2.0 DEPS installed
  • FDD, framework dependent deployment
    • smaller size
    • require .net core 2.0 runtime installed
    • execution is dotnet paket.dll, so require a wrapper script to execute for UX (paket.cmd/paket.sh)
    • where is dotnet? is the right version installed?
  • DotnetCliTool
    • just a proj (few lines of text) is required
    • update based on just nuget.org
    • require restore
    • require .net core sdk installed
    • run as dotnet paket
    • require execution from same dir of proj, so need wrapper scripts (paket.cmd/paket.sh)

File size:

DotnetCliTool FDD SCD
File Size 0,1 MB 1MB 60 MB
Requirements .net sdk + runtime + runtime deps (120MB) .net core runtime + runtime deps (60mb) .net core runtime deps (native pkg)

notes about paket, dotnet/sdk, .net core

paket is a strange tool, because atm for restore process, need to be there BEFORE the restore.
or at least a target file to override default nuget implementation in dotnet/sdk
instead old sdk require the exe to restore/update projs

But lots of challenges are shared with normal tools in .net ecosystem

There is no builtin per repo tools. or a way to script (like an fsx).
AFAIK the most similar to per repo tools are the cli tools or the FDD.
AFAIK the most similar to script is an msbuild target + optional task in assembly

How now .net sdk support external tools? or custom execute code xplat.

  • Dotnet Cli tools (per project)
  • msbuild targets
  • built it on machine. dotnet new + override Program.cs + dotnet publish

Other package manager expect the dev sdk installed, and use that to boostrap:

Possible solutions

My current ideas, not in preference order.

plan A

support tools at repo level natively in dotnet/sdk. ref dotnet/cli#4723

PRO:

  • native support in dotnet/sdk
  • less maintenance by paket team
  • no boostrapper needed. the sdk is the boostrapper.

CONS:

  • native support in dotnet/sdk
    • require new version of dotnet/sdk. probably at least 2.2 or 3.x
    • will not work on older VS version or just msbuild (but VS now bundle .net core sdk...)
  • depends on dotnet/sdk team
    • slower to patch/workaround, based on .net team schedule not paket team
  • require .net sdk installed, but native paket app can be download if needed, that's just for boostrap
  • cannot replace current paket for older sdk

plan B

Why: paket deps doesnt change a lot. paket does.

like now, using native app (SCD) but smarter update mechanism

  • paket published as SCD and FDD
  • current paket.boostrapper.exe as SCD
  • SCD are big, need a smart update mechanism
    1. first usage download latest native console binaries (per OS)
    2. updating is just xplat paket FDD, so paket.dll and non-bcl managed deps (paket.core.dll, etc)
    3. if not possible the minimal update, do step 1
  • boostrapping process start with a dotnet msbuild (target + dll), to just download paket.boostrapper.exe

PRO:

  • pretty much like now
  • can replace current boostrapping
    • new deps will be just .net core 2 runtime deps, not mono/.net
  • boostrapping can also download the net version like now, if want to support that too

CONS:

  • maintenance on paket team
  • paket specific :( a missed opportunity
  • download paket native boostrapper is big. but can be cached.

plan C

Why: support and enhance .net cli tools. use that support for paket itself.

Add support of repo level tools to paket and .net cli tools. Use a built paket native (SCD) to boostrap paket (replace previous paket.boostrapper.exe).
The chicked/egg solution.

  • paket as packaged console app (like a .net cli tool, ex dotnet-fable), so small and easy to pack/version
  • boostrapper (paket native):
    • is paket, with a paket.depencies who contains paket-cli as tool
  • add .net cli tools support to paket
    • the .net cli tool (like dotnet fable) just invoke dotnet with args and a file with deps. paket generate these args/files needed
    • generate a shims/script wrapper, so a dotnet hello tool can be invoked as hello. and paket as paket
  • boostrapping based on dotnet msbuild to just download native paket per OS

PRO:

  • cli tools can be in also used paket.deps/lock. per repo.
  • boostrapper is not a special case, is just paket built as native console app
  • minimal size.
  • enable other tools (good). like dotnet fable as fable. will work in whole repo not just the proj dir.

CONS:

  • download paket native boostrapper is big. but can be cached.
  • boostrapper paket schedule <> paket schedule, so doesnt need to update often. more maintance.
  • atm cli tools are from nuget feed. github fallback need to be implemented (paket download nupkg from github to a dir -> add that dir as feed -> restore)

Conclusions

  • plan A: .net oss way, but slower to implement and require at least .net sdk vNext version as deps.
  • plan B: upgrade current boostrapping mechanism
  • plan C: remove current bootstrapper, replace with paket who support .net cli tools

Atm i prefer plan C, while is longer than B to implement, add a lot more value with less maintenance long term.
Plan A is ok, but doesnt help paket in medium term, because cannot replace current way.

TODO

as a note, i volounteer to implement the choosen solution. if someone want to help, will be nice, because tasks can be split

@forki forki closed this Oct 30, 2017

@forki forki reopened this Oct 30, 2017

@forki

This comment has been minimized.

Copy link
Member

forki commented Oct 30, 2017

Is there a way to put bootstrapper directly into sdk?

@enricosada

This comment has been minimized.

Copy link
Collaborator

enricosada commented Oct 30, 2017

Is there a way to put bootstrapper directly into sdk?

yes that's another way. two options afaik:

bundled in sdk, as app/package

Technically, yes. we have done that for fsharp.net.sdk, and put inside sdk is just a packaging thing (cli/sdk build script restore a pinned package and unzip in release dir).
But depends if .net cli/sdk team want to do that.
For example with fsharp.net.sdk, was just possibile to put in dotnet cli, but not in nuget packages distribuited in the sdk installed with VS because of policy. so it will had worked with dotnet sdk (the cli) but not in msbuild bundled with VS. This was strange because some package were there, like newtonsoft or others by aspnet deps, but that it.

Issue is, if there is an error in boostrapper, you need another release of sdk to fix that (and just sdk team can schedule that).

as Sdk attribute

Another possibility, the real issue, is if Sdk attribute can be extended, like by restored packages.
that is the real extensibility paket need afaik.
So will be possibile to use like Sdk="Paket.Sdk;Microsoft.NET.Sdk".
Is somewhat tracked in Microsoft/msbuild#1493 but is now stalled afaik, because noone in MS need that (they can just continue to bundle things..)

Both mean will work only with new msbuild/cli/sdks versions.. and given the churn in sdk tooling and the need for stability, ihmo is not going to be implemented soon..

@enricosada enricosada referenced this issue Nov 28, 2017

Closed

[WIP] .net core paket #2918

5 of 8 tasks complete
@enricosada

This comment has been minimized.

Copy link
Collaborator

enricosada commented Nov 28, 2017

ok, dotnet cli is adding tools (sdk scooped) in v2.2 (ref dotnet/cli#8067 )

we can use that later for boostrapper. meanwhile need another options for that

And paket can manage tools too, because should be easy. adding per repo tools (who can be used to self-update itself, if the boostrapper is just paket)

the tools are just .net core fdd app published in the tools/netcoreapp2.1 dir of a nupkg.

ref a nuspec

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
    <metadata>
        <id>global.tool.console.demo</id>
        <version>1.0.4</version>
        <description>test app</description>
        <authors>testauthor</authors>
    </metadata>
    <files>
        <file src="bin\Release\netcoreapp2.1\publish\*.*" target="tools\netcoreapp2.1\" /> 
        <file src="DotnetToolsConfig.xml" target="tools\DotnetToolsConfig.xml" /> 
    </files>
</package>

and an xmlfile to read the entry points, so commands can be renamed and multiple

<?xml version="1.0" encoding="utf-8" ?>
<DotNetCliTool>
  <Commands>
    <Command Name="demo" EntryPoint="consoledemo.dll" Runner="dotnet" />
  </Commands>
</DotNetCliTool>
@enricosada

This comment has been minimized.

Copy link
Collaborator

enricosada commented Dec 13, 2017

For interested ppl, there is a demo version on a ready to use repository ( more info #2918 (comment) ).
is ok to beta test (just the boostrapper require mono atm, but is temporary)

based on the PR wip, will be updated

@enricosada

This comment has been minimized.

Copy link
Collaborator

enricosada commented Jan 23, 2018

By @0x53A in #2418 (comment)

MSBuild will add a package-based SDK resolving mechanism.

MSBuild spec: Microsoft/msbuild#1493
NuGet work item: NuGet/Home#5220

There is nothing to do at the moment - but as soon as NuGet actually implements this stuff, we should also start so we don't hang behind.

From the empty checklists in both issues, I would expect this to land in 16.0 maybe.

The important question is how the resolving works when we don't control the host (VS or Rider, or VS Code, ...).
Can we pass-in our paket resolver, or will VS always hardcoded use the NuGet resolver?

@halcwb

This comment has been minimized.

Copy link
Contributor

halcwb commented Mar 9, 2018

Just a sanity check. Currently it is not possible to use only dotnetcore with Paket? You need to have .net framework or mono installed?

@forki

This comment has been minimized.

Copy link
Member

forki commented Mar 9, 2018

yes to run paket you still need that. but @enricosada is working on it...

@enricosada

This comment has been minimized.

Copy link
Collaborator

enricosada commented May 3, 2018

ok, new WIP of install flow (maintaingn bootstrapper or not) in #3183 based on sdk tools ( .net core sdk >= 2.1.300 ).

@aggieben

This comment has been minimized.

Copy link

aggieben commented Sep 13, 2018

Any changes to this recently?

@enricosada

This comment has been minimized.

Copy link
Collaborator

enricosada commented Sep 14, 2018

@aggieben yes, latest (just today, i was working on it these days), is in https://github.com/enricosada/paket-netcore-testing-as-tool/

should be somewhat done (minus known bugs). please review the PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment