Allow multiple repositories to be monitored and updated

[bleroy]

The scenario is the following: the application consists of multiple independent repositories (not submodules). In my specific case, the application is the top-level repo, and there's a second content repository in a subdirectory.

Currently, only one repo is handled, but to allow for this scenario, it should be possible to configure multiple ones.

Solution provided by David Ebbo: https://weblogs.asp.net/bleroy/automatic-deployment-of-multiple-repositories-to-azure

[bleroy]

Current workaround: use the Kudu diagnostics console to manually update the second repo when there are changes.

[ahmelsayed]

If I understand this correctly, you should be able to achieve that with a custom deployment script.

from Kudu UI, click on Tools -> Download deployment script. or you can manually get them from D:\home\site\deployments\tools

you will have 2 files, .deployment and deploy.cmd. You will need to check these files in the root of your main site's repo.
Then open deploy.cmd there will be a section entitled

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: Deployment
:: ----------

that probably ends with :: . KuduSync. This is the logic for deploying from your git repo.
you can add something like this after KuduSync is done

if EXIST %HOME%\site\wwwroot\myOtherRepo (
   pushd %HOME%\site\wwwroot\myOtherRepo
   git pull
   popd
) ELSE (
   git clone <link fot repo>.git %HOME%\site\wwwroot\myOtherRepo
)

Now every time you do a git deployment, that will execute and update the other repos in your site. You can add as many as you want and in any layout you want.

[bleroy]

mmm, no, completely broke my site (now only showing "You do not have permission to view this directory or page."). I haven't been able to determine why yet.

[bleroy]

Removing the custom logic from the deployment script restored the site. Back to square one.

[davidebbo]

Start by including the unmodified custom deployment files to isolate. Also, try comparing the web.config in the working vs non-working case. In theory, simply committing the default deployment scripts that the site is already using should be a no-op.

[bleroy]

Yes, it is: when I removed the custom logic, the site was back up. Here's the custom code I added to the script:

 if EXIST %HOME%\site\wwwroot\sites\default\.git (
   pushd %HOME%\site\wwwroot\sites\default
   git pull
   popd
)

[ahmelsayed]

when you had that logic in there, did you see any output from git in the deployment logs for that step?

[bleroy]

No.

[davidebbo]

If the script only breaks with those additional lines, then there must be something wrong there. Echo lines in various places, and selectively adding lines should help you isolate. e.g. start with an empty logic block in the if statement, etc...

[bleroy]

I'll have to set-up a new test environment then, I don't want to pollute my repos or take the site down. I'll keep you posted.

[bleroy]

Seems like manually doing a git pull from the second repository's directory in the Kudu diagnostics console also kills the site (returns "You do not have permission to view this directory or page.") on all requests. Restarting the site doesn't fix it. The only fix that works is to push a new change onto the main repository.
But wait, there's worse: when you do a git pull, it copies the contents of the second repository into the root of the site.
So here's what I think is happening: whenever a git pull command is executed, no matter where in the file system under the site, it goes through the motions of the main repo's update, and the inner repo's changes end up at the root, and mess up the whole site. Only by forcing the main repo to re-pull and reset the site can you restore the site.
Did you try to reproduce this?

[davidebbo]

I'm not able to repro this with simple repos. e.g. I used https://github.com/KuduApps/NodeHelloWorldNoConfig as the Kudu repo, and https://github.com/KuduApps/HelloWorld as the second repo in wwwroot\HelloWord. Can you try with those exact repos?

Note that when you do a pull on your second repo from the console, there should be nothing specific to Azure/Kudu. You only dealing with git, and whatever behavior you get is what git is doing.

If you are stuck, you may need to provide more precise info about your scenario, and the exact steps. e.g. when you do a pull, are you actually pulling new commits into the second repo or is the repo already up to date? Having precise steps will help, as there is too much room for interpretation from a high level description.

[bleroy]

Yes, I am pulling actual new commits, otherwise it's a no-op. Did you add a hook to the second repository (which we'd obviously need in order for pushes to master to trigger the deployment script)?

[davidebbo]

You didn't mention a hook for the second repo. What kind of hook is it? I don't see why this would be needed if you have logic in the main repo's deployment script to pull the 2nd repo when the main one deploys.

But this emphasizes my previous point: there are just too many aspects of this that are not capture precisely enough, and having an unambiguous sequence of steps that leads to the situation would probably save a lot of back and forth :)

[bleroy]

Well, this would be completely useless if changes on the second repo didn't trigger anything, right? If you have to manually go into the console and manually do a git pull, what's the point? Do you mean that I would need to do a dummy commit on the first repo in order for the second one to update? I did mention that in email, but you're right that I didn't on this thread.

I'll give you my repro steps, but I'm not sure this will help more than what you've already done, because there are many more moving parts in my case. I'm sorry I haven't had time yet to narrow down the problem and only have observed it in the much more complex environment of my real site. It may be worth trying to just reproduce the first repo's hook on the second repo, then push a change on repo2 and git pull it from the console.

Here's my overly complicated repro:

Deploy from your own fork of https://github.com/DecentCMS/DecentCMS
From Kudu console:
cd site\wwwroot
npm install
mkdir sites
cd sites
git clone your own fork of https://github.com/DecentCMS/DecentConsulting.git
cd ..
grunt install_runtime
mkdir sites\default\logs

At this point, you should have a running site.

Reproduce the first repository's hook onto the second repo.

Make and push a change to your fork of the second repo, then git pull from sites/default in the diagnostics console. The site should now be down, and you should have junk from the second repo at the root of the site.

[davidebbo]

This whole approach based on what @ahmelsayed suggested above is based on deployment of the second repo happening as a result of pushing to the first. This may not be the behavior that you want, but let's first try to explain the strange behavior that you get when you try that. Then we can explore alternative approaches.

[davidebbo]

What exactly does "Reproduce the first repository's hook onto the second repo" mean? Are you copying the post-receive hook file from one repo's .git folder to the other?

[bleroy]

I don't think there's anything that doesn't work as expected in what @ahmelsayed suggested above, but it doesn't correspond to any useful scenario.

Copy the settings of the first hook onto the second one. Unless you see another way to trigger the deployment script (or even better, another script) from a push on the second repo. As you probably guessed, I don't care how it's implemented, only about being able to push to the second repo and see changes being automatically integrated into the site as expected.

[davidebbo]

The confusion was that I thought you were trying what Ahmed suggested and were hitting issues, when in fact you were experimenting with different approaches.

I'm still not clear what you mean for the hook, as the term is quite over loaded. Here, it could mean:

1. the git hooks that any git repo has in its .git\hooks folder. There are lots of different hooks in there.
2. the github WebHooks

Given that you wrote "manually doing a git pull from the second repository's directory in the Kudu diagnostics console also kills the site", I assume you are talking 1., since 2. would have no effect at all when simply pulling from GitHub. Can you confirm?

[bleroy]

I meant 2, Github webhooks: when you go to the repository's page on Github, go into settings, then webhooks and services. When you configure an Azure web site to get its contents from a repo, it creates a hook on the Github side automatically. The URL for that hook is, in the case of Azure, hitting decentnode.scm.azurewebsites.net/deploy with a querystring and some credentials. All those parameters can be copied to a new hook on the second repository. Making sense?

So no, pulling from Github has no effect on those, but pushing does. That's the idea: that the Azure site takes changes when a new changeset is pushed.

[davidebbo]

Ok, so I think we agree that the statement "manually doing a git pull from the second repository's directory in the Kudu diagnostics console also kills the site" was not correct. The thing that broke your site was the triggering of the GitHub hook, and it will not happen in this case (I can explain in more details why this breaks your site, but at this point it's moot).

Moving on, some (hopefully) good news: I found a completely different way to make this work, by sort-of 'abusing' our WebJobs feature. Please try the following:

• Using Kudu console, create a D:\home\site\wwwroot\App_Data\jobs\triggered\DeploySubrepo folder
• In there, create a run.cmd file containing the following (adjust the folder and repo as needed):

cd /d %HOME%\site\wwwroot\DecentConsulting
git pull https://github.com/DecentCMS/DecentConsulting.git

• On your GitHub repo for the second repo, create a WebHook that looks similar to the one on your primary repo, with the path changed from /deploy?scmType=GitHub to /api/triggeredwebjobs/DeploySubrepo/run.

The idea is that a push to your second repo will trigger the WebJob, which will in turn pull your second repo inside your site. I have tried it on a simple case, so I know it works!

If you need to debug the WebJob invocation, go to /azurejobs on the scm URL (i.e. under Kudu). You'll see all the invocations, and their log.

This is definitely a new way to use WebJobs, so you can be a pioneer. It's slightly quirky to set up, once it's done it should just work.

[bleroy]

Yes, that's correct (as you can see, I'm trying to sort this out progressively). Thanks for this new idea. I just deployed it and it works flawlessly.

[davidebbo]

Back link to your post! https://weblogs.asp.net/bleroy/automatic-deployment-of-multiple-repositories-to-azure