New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to publish multiple projects at the same time. #4825

Open
joelharkes opened this Issue Nov 23, 2016 · 21 comments

Comments

Projects
None yet
10 participants
@joelharkes

joelharkes commented Nov 23, 2016

When i publish with dotnet cli, the output is:

publish: Published to ../output/project1
Published 1/1 projects successfully

Which indicates i could publish multiple projects at te same time (why else say: 1/1)

so i tried this.

Steps to reproduce

dotnet publish project1 project2 -o ../output

Expected behavior

I expect 2 projects to be published in this output folder (each in there own folder, having the same name as the project folder).

Actual behavior

Unrecognized command or argument project2

Environment data

.NET Command Line Tools (1.0.0-preview2-1-003155)

Product Information:
 Version:            1.0.0-preview2-1-003155
 Commit SHA-1 hash:  d7b0190bd4

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.14393
 OS Platform: Windows
 RID:         win10-x64

@TheRealPiotrP TheRealPiotrP added this to the Discussion milestone Nov 24, 2016

@TheRealPiotrP

This comment has been minimized.

Show comment
Hide comment
@TheRealPiotrP

TheRealPiotrP Nov 24, 2016

Collaborator

@blackdwarf I agree with @joelharkes's assertion that the experience in preview2.1 was confusing. Thoughts? Note, this is a bug on the PJ CLI so it may not apply directly, but what is our intent on multi-project [sln?] publish?

Collaborator

TheRealPiotrP commented Nov 24, 2016

@blackdwarf I agree with @joelharkes's assertion that the experience in preview2.1 was confusing. Thoughts? Note, this is a bug on the PJ CLI so it may not apply directly, but what is our intent on multi-project [sln?] publish?

@joelharkes

This comment has been minimized.

Show comment
Hide comment
@joelharkes

joelharkes Nov 24, 2016

@piotrpMSFT I think multi publish should be considered.

Our architecture is made up of 2 kestrel console apps and 3 other console applications.
now i have to execute 5 times dotnet publish. 5 times going over library projects and all executing in synchronous order (otherwise i don't know if it will run into locking problems with trying to build the same project). This is a lot of overhead. if this is managed by the CLI, projects could be run asynchronously, speeding up the build performance with a big factor.

joelharkes commented Nov 24, 2016

@piotrpMSFT I think multi publish should be considered.

Our architecture is made up of 2 kestrel console apps and 3 other console applications.
now i have to execute 5 times dotnet publish. 5 times going over library projects and all executing in synchronous order (otherwise i don't know if it will run into locking problems with trying to build the same project). This is a lot of overhead. if this is managed by the CLI, projects could be run asynchronously, speeding up the build performance with a big factor.

@TheRealPiotrP

This comment has been minimized.

Show comment
Hide comment
@TheRealPiotrP

TheRealPiotrP Nov 28, 2016

Collaborator

@blackdwarf will propose the intention change here.

Collaborator

TheRealPiotrP commented Nov 28, 2016

@blackdwarf will propose the intention change here.

@blackdwarf

This comment has been minimized.

Show comment
Hide comment
@blackdwarf

blackdwarf Dec 21, 2016

Contributor

For publish, I don't think publishing from a solution falls naturally because it is highly dependent on the solution architecture and the overall type of projects that are contained within. For some solutions, like the one @joelharkes mentions, it may make sense. For others that I saw and discussed as I was thinking about this, the expected behavior would be exactly the opposite, that is, publish would be expected not to work on SLN files as that would produce an output that would not be wanted.

There are also many nuances that need to be thought of when thinking on this, mostly w.r.t higher-level tooling that will also consume this target and how does running that target on the solution works for their UX-es and capabilities.

So, for V1 at least, I don't see us supporting this behavior.

Contributor

blackdwarf commented Dec 21, 2016

For publish, I don't think publishing from a solution falls naturally because it is highly dependent on the solution architecture and the overall type of projects that are contained within. For some solutions, like the one @joelharkes mentions, it may make sense. For others that I saw and discussed as I was thinking about this, the expected behavior would be exactly the opposite, that is, publish would be expected not to work on SLN files as that would produce an output that would not be wanted.

There are also many nuances that need to be thought of when thinking on this, mostly w.r.t higher-level tooling that will also consume this target and how does running that target on the solution works for their UX-es and capabilities.

So, for V1 at least, I don't see us supporting this behavior.

@neman

This comment has been minimized.

Show comment
Hide comment
@neman

neman Mar 24, 2017

This is can be easy be done with powershell script.
You just have to create for each project build/publish script, and then execute it within loop.
This way I build/publish/create nuget/push nuget with one command for more than 30 projects in one solution.

This code could be in some PublishMultipleProjects.ps1 script.
You can extend this by adding parms what to include and so on...

Push-Location $PSScriptRoot     
$files = get-childitem -Recurse -Include "Build.ps1"   
foreach ($file in $files)    
{   
    # Execute each Build.ps1 script   
    & $file   
}   
Push-Location $PSScriptRoot

neman commented Mar 24, 2017

This is can be easy be done with powershell script.
You just have to create for each project build/publish script, and then execute it within loop.
This way I build/publish/create nuget/push nuget with one command for more than 30 projects in one solution.

This code could be in some PublishMultipleProjects.ps1 script.
You can extend this by adding parms what to include and so on...

Push-Location $PSScriptRoot     
$files = get-childitem -Recurse -Include "Build.ps1"   
foreach ($file in $files)    
{   
    # Execute each Build.ps1 script   
    & $file   
}   
Push-Location $PSScriptRoot
@joelharkes

This comment has been minimized.

Show comment
Hide comment
@joelharkes

joelharkes Mar 28, 2017

@neman That is exactly what you want to avoid for 2 main reasons:

  • They ported back to msbuild so you could use an msbuild file for this instead of powershell
  • This is either: Slow because its all executed after each other or gets into locking issues because trying to build multiple projects at once.

I made a script that builds the dependency trees for all projects and then executes builds in parallel (Promises nodejs) according to the builds. I got a major decrease in build/publish time (on a multi core environment).

Too bad though our build server agents only use 1 processor per agent but, but locally for developers its still better this way.

joelharkes commented Mar 28, 2017

@neman That is exactly what you want to avoid for 2 main reasons:

  • They ported back to msbuild so you could use an msbuild file for this instead of powershell
  • This is either: Slow because its all executed after each other or gets into locking issues because trying to build multiple projects at once.

I made a script that builds the dependency trees for all projects and then executes builds in parallel (Promises nodejs) according to the builds. I got a major decrease in build/publish time (on a multi core environment).

Too bad though our build server agents only use 1 processor per agent but, but locally for developers its still better this way.

@dasMulli

This comment has been minimized.

Show comment
Hide comment
@dasMulli

dasMulli Mar 28, 2017

Contributor

I've managed to bring down build+test time by using msbuild scripts like this (even in non-parallel builds): https://gist.github.com/dasMulli/69f5303aa79a8cd4060e44891c90fd2d

Essentially, making a single build.proj file that calls publish on others via <MSBuild/> tasks (=> dotnet msbuild build.proj.

Contributor

dasMulli commented Mar 28, 2017

I've managed to bring down build+test time by using msbuild scripts like this (even in non-parallel builds): https://gist.github.com/dasMulli/69f5303aa79a8cd4060e44891c90fd2d

Essentially, making a single build.proj file that calls publish on others via <MSBuild/> tasks (=> dotnet msbuild build.proj.

@joelharkes

This comment has been minimized.

Show comment
Hide comment
@joelharkes

joelharkes Mar 28, 2017

@dasMulli looks good, but ContinueOnError="ErrorAndContinue" this seems very dangerous/error-prone? are you sure dependencies are always build before projects higher in the tree are build?

joelharkes commented Mar 28, 2017

@dasMulli looks good, but ContinueOnError="ErrorAndContinue" this seems very dangerous/error-prone? are you sure dependencies are always build before projects higher in the tree are build?

@dasMulli

This comment has been minimized.

Show comment
Hide comment
@dasMulli

dasMulli Mar 28, 2017

Contributor

I've used ErrorAndContinue to make sure it at least tries to test all test projects. The target will still return errors but ensures that all tasks are executed. Dependencies are always built before individual projects and msbuild should already handle concurrency situations inside a single msbuild invocation. For lots of test projects, it's probably even faster to build a solution and then running tests in parallel using VSTestNoBuild=true as additional parameter.

Contributor

dasMulli commented Mar 28, 2017

I've used ErrorAndContinue to make sure it at least tries to test all test projects. The target will still return errors but ensures that all tasks are executed. Dependencies are always built before individual projects and msbuild should already handle concurrency situations inside a single msbuild invocation. For lots of test projects, it's probably even faster to build a solution and then running tests in parallel using VSTestNoBuild=true as additional parameter.

@joelharkes

This comment has been minimized.

Show comment
Hide comment
@joelharkes

joelharkes Apr 24, 2017

We are planning our migration to csproj.
I seem to have the same problem here.

I want to publish 5 startup projects at the same time not needing to rebuild library projects more than once. But still there seems no possiblity?

if i dotnet publish solution.sln it puts all the 5 projects into 1 folder.

If i run dotnet publish project1.csproj it publishes 1 project but first builds all the project refernces.

Could there be an option made to either specify separate publish projects in the solution file? Or an option: --no-build like on dotnet pack to publish a project without rebuilding it?

joelharkes commented Apr 24, 2017

We are planning our migration to csproj.
I seem to have the same problem here.

I want to publish 5 startup projects at the same time not needing to rebuild library projects more than once. But still there seems no possiblity?

if i dotnet publish solution.sln it puts all the 5 projects into 1 folder.

If i run dotnet publish project1.csproj it publishes 1 project but first builds all the project refernces.

Could there be an option made to either specify separate publish projects in the solution file? Or an option: --no-build like on dotnet pack to publish a project without rebuilding it?

@joelharkes

This comment has been minimized.

Show comment
Hide comment
@joelharkes

joelharkes Apr 24, 2017

@blackdwarf is there anything on the roadmap for this feature? it could greatly speed up our build process.

joelharkes commented Apr 24, 2017

@blackdwarf is there anything on the roadmap for this feature? it could greatly speed up our build process.

@blackdwarf

This comment has been minimized.

Show comment
Hide comment
@blackdwarf

blackdwarf Apr 25, 2017

Contributor

@joelharkes sorry, but I'm not with Microsoft anymore so I really don't know what the roadmap is like these days. @livarcocc @richlander and @bleroy could help with this though, I would think.

FWIW, when we took a look at publishing based on the SLN file, I remember that there was a big split on whether the behavior could be made sane enough for most use cases to work. Some people were very for it (similar to dotnet restore or dotnet build), while others thought it was exactly the wrong thing to do. Add to it the options for redirecting output (--output) and you have a nice can 'o' worms there.

Your idea of having a --no-build option on dotnet publish does sound useful, so I guess a separate issue could be made for that, but that would also have to go into the dotnet/sdk repo IINM.

Contributor

blackdwarf commented Apr 25, 2017

@joelharkes sorry, but I'm not with Microsoft anymore so I really don't know what the roadmap is like these days. @livarcocc @richlander and @bleroy could help with this though, I would think.

FWIW, when we took a look at publishing based on the SLN file, I remember that there was a big split on whether the behavior could be made sane enough for most use cases to work. Some people were very for it (similar to dotnet restore or dotnet build), while others thought it was exactly the wrong thing to do. Add to it the options for redirecting output (--output) and you have a nice can 'o' worms there.

Your idea of having a --no-build option on dotnet publish does sound useful, so I guess a separate issue could be made for that, but that would also have to go into the dotnet/sdk repo IINM.

@gandarez

This comment has been minimized.

Show comment
Hide comment
@gandarez

gandarez Jan 18, 2018

having the same behaviour here, I have two projects to be published and "dot net" puts both on the same page 👎

gandarez commented Jan 18, 2018

having the same behaviour here, I have two projects to be published and "dot net" puts both on the same page 👎

@livarcocc

This comment has been minimized.

Show comment
Hide comment
@livarcocc

livarcocc Jan 18, 2018

Member

cc @KathleenDollard

@gandarez are you publishing a solution or publishing with -o?

Member

livarcocc commented Jan 18, 2018

cc @KathleenDollard

@gandarez are you publishing a solution or publishing with -o?

@gandarez

This comment has been minimized.

Show comment
Hide comment
@gandarez

gandarez Jan 18, 2018

@livarcocc
I'm publishing through vsts using -o

gandarez commented Jan 18, 2018

@livarcocc
I'm publishing through vsts using -o

@KathleenDollard

This comment has been minimized.

Show comment
Hide comment
@KathleenDollard

KathleenDollard Jan 18, 2018

This issue is still open, meaning we haven't said no. But we also do not have it on the roadmap as far as I know. I believe it needs some thought to get right.

@ganderez - I am a bit confused about your scenario. Can you send more information so we can figure out if this is a CLI issue or a VSTS issue?

KathleenDollard commented Jan 18, 2018

This issue is still open, meaning we haven't said no. But we also do not have it on the roadmap as far as I know. I believe it needs some thought to get right.

@ganderez - I am a bit confused about your scenario. Can you send more information so we can figure out if this is a CLI issue or a VSTS issue?

@gandarez

This comment has been minimized.

Show comment
Hide comment
@gandarez

gandarez Jan 19, 2018

@KathleenDollard
The solution contains two web api core projects and I'm using dot net publish to publish them. But the cli put every single project to the same root output directory.
The MSBUILD does not have that behaviour, it groups each project in its respective folder.

gandarez commented Jan 19, 2018

@KathleenDollard
The solution contains two web api core projects and I'm using dot net publish to publish them. But the cli put every single project to the same root output directory.
The MSBUILD does not have that behaviour, it groups each project in its respective folder.

@dasMulli

This comment has been minimized.

Show comment
Hide comment
@dasMulli

dasMulli Jan 19, 2018

Contributor

@gandarez when you say "The MSBUILD does not have that behaviour", how have you been using msbuild? the msbuild equivalent of dotnet publish -o foo would be msbuild /t:Publish /p:PublishDir=foo. If you have been using /p:DeployOnBuild=true with a publish profile, that is a different way of publishing specific to asp.net and asp.net core and may behave differently.
It should also be noted that if you pass a relative path to the -o parameter, it is always interpreted relative to the individual csproj file, not relative to the path the dotnet publish command is invoked from.

Contributor

dasMulli commented Jan 19, 2018

@gandarez when you say "The MSBUILD does not have that behaviour", how have you been using msbuild? the msbuild equivalent of dotnet publish -o foo would be msbuild /t:Publish /p:PublishDir=foo. If you have been using /p:DeployOnBuild=true with a publish profile, that is a different way of publishing specific to asp.net and asp.net core and may behave differently.
It should also be noted that if you pass a relative path to the -o parameter, it is always interpreted relative to the individual csproj file, not relative to the path the dotnet publish command is invoked from.

@livarcocc

This comment has been minimized.

Show comment
Hide comment
@livarcocc

livarcocc May 11, 2018

Member

@gandarez can you answer @dasMulli questions above so that we can understand your scenario better?

Member

livarcocc commented May 11, 2018

@gandarez can you answer @dasMulli questions above so that we can understand your scenario better?

@housten

This comment has been minimized.

Show comment
Hide comment
@housten

housten Jun 26, 2018

One solution that seems simple to me would be to have a variable that references the current project being built/published so one can have dotnet publish -o foo\$(Build.CurrentProject) or msbuild /t:Publish /p:PublishDir=foo\$(Build.CurrentProject) if one wants separate folders for each publish in a solution.
Or maybe this already exists? I am not seeing it in the build variables documentation at any rate.

housten commented Jun 26, 2018

One solution that seems simple to me would be to have a variable that references the current project being built/published so one can have dotnet publish -o foo\$(Build.CurrentProject) or msbuild /t:Publish /p:PublishDir=foo\$(Build.CurrentProject) if one wants separate folders for each publish in a solution.
Or maybe this already exists? I am not seeing it in the build variables documentation at any rate.

@avodovnik

This comment has been minimized.

Show comment
Hide comment
@avodovnik

avodovnik Aug 30, 2018

Has there been any progress on this issue, or do we know of any other way around this?

avodovnik commented Aug 30, 2018

Has there been any progress on this issue, or do we know of any other way around this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment