Skip to content
This repository has been archived by the owner on Oct 18, 2018. It is now read-only.

Cache Coherence locally in ci-test. #418

Merged
merged 1 commit into from
Mar 16, 2016
Merged

Cache Coherence locally in ci-test. #418

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
40 changes: 39 additions & 1 deletion makefile.shade
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ use namespace='System.Text'
use namespace='System.Text.RegularExpressions'
use namespace='System.Threading.Tasks'
use import="BuildEnv"
use import="Files"
use import="Json"

functions
Expand Down Expand Up @@ -86,7 +87,7 @@ var buildTarget = "compile"

#verify-all .pull .change-default-build-target-to-verify .build-all

#ci-test .pull .sync-commits .remove-src-folders .change-default-build-target-to-verify .build-all
#ci-test .pull .sync-commits .remove-src-folders .cache-coherence .change-default-build-target-to-verify .build-all

#ci-build
@{
Expand Down Expand Up @@ -203,6 +204,40 @@ var buildTarget = "compile"
}
}

#cache-coherence
@{
var coherenceCacheDir = Path.Combine(BASE_DIR, Environment.GetEnvironmentVariable("COHERENCE_CACHE_DIR") ?? ".coherence-cache");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the build dir should not be relative to the BASE_DIR if the COHERENCE_CACHE_DIR is set. On the CI we'll probably set it to a global location so that builds share it and they don't have to redownload it.

var dropsShare = Environment.GetEnvironmentVariable("ASPNETCI_DROPS_SHARE");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be a default for this in code. It's hard to run it locally otherwise because you need to search through code to figure out what vars to set


if (!string.IsNullOrWhiteSpace(dropsShare))
{
var coherenceShare = Path.Combine(dropsShare, "Coherence", BUILD_BRANCH);
var latestBuild = Directory.EnumerateDirectories(coherenceShare)
.Select(d => Path.GetFileName(d))
.Where(d => { int _; return int.TryParse(d, out _); })
.OrderByDescending(d => d)
.First();
var targetDir = Path.Combine(coherenceCacheDir, latestBuild);

Log.Info("Latest Coherence build is " + latestBuild + ".");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

space space space space space

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol brain damage


if (!Directory.Exists(targetDir))
{
var latestBuildShare = Path.Combine(coherenceShare, latestBuild);

Log.Info("Caching Coherence build " + latestBuild + " at " + targetDir + ".");
CopyFolder(latestBuildShare, targetDir, overwrite: true);
}
else
{
Log.Info("Coherence build " + latestBuild + " already cached at " + targetDir + ".");
}

Environment.SetEnvironmentVariable("NUGET_VOLATILE_FEED_ASPNETVNEXT", Path.Combine(targetDir, "build"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caching the nupkgs as-is is not super nice for perf especially given the directory has close to 600 nupkgs. You should probably expand it.

Environment.SetEnvironmentVariable("NUGET_VOLATILE_FEED_EXTERNAL", Path.Combine(dropsShare, "latest-packages", "external", BUILD_BRANCH));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not set this up in the CI? It's easier to remove this in case we need to not use volatile?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be a way to overwrite this with an env var but we should move away from configurations that don't have defaults in code because it makes it hard to reproduce builds locally.

Btw, where are this variable and the one above used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pranavkm We don't want those variables set unless caching Coherence was successful.

@victorhurdugaci https://github.com/aspnet/KoreBuild/blob/f693dea4c9199e63bb8ae1f544f9b39861dcd073/build/shade/_use-volatile-feed.shade As we've discussed offline, we should come up with a clearer way to go about this.

}
}

#change-default-build-target-to-verify
- buildTarget = "verify";

Expand Down Expand Up @@ -614,6 +649,9 @@ macro name='Exec' program='string' commandline='string' workingdir='string'
macro name='NuGetPackagesAdd' sourcePackagesDir='string' targetPackagesDir='string'
nuget-packages-add

macro name="CopyFolder" sourceDir='string' outputDir='string' overwrite='bool'
copy

functions
@{
static IDictionary<string, string> GetCommitsToSync()
Expand Down