[Unified validations] Update the validation database entities

[loic-sharma]

Background

Update the validation database as per the spec: https://github.com/NuGet/Engineering/blob/master/Server.Specs/OrchestratorUnifiedValidations.md#new-database-schema

Part of https://github.com/NuGet/Engineering/issues/3576

How to review

1. Please review the full changes
2. Please review the customizations to the migration: 22838c9
3. Please review the migration's generated SQL below.

Migration's generated SQL...

-- Create the new PackageValidationResults table
CREATE TABLE [dbo].[PackageValidationResults] (
    [Key] [bigint] NOT NULL IDENTITY,
    [Type] [nvarchar](128) NOT NULL,
    [Data] [nvarchar](max) NOT NULL,
    [PackageValidationSetKey] [bigint] NOT NULL,
    [PackageValidationKey] [uniqueidentifier],
    CONSTRAINT [PK_dbo.PackageValidationResults] PRIMARY KEY ([Key])
)
CREATE INDEX [IX_PackageValidationSetKey] ON [dbo].[PackageValidationResults]([PackageValidationSetKey])
CREATE INDEX [IX_PackageValidationKey] ON [dbo].[PackageValidationResults]([PackageValidationKey])

-- Add the new columns to the PackageValidationSets table
ALTER TABLE [dbo].[PackageValidationSets] ADD [PackageContentType] [nvarchar](128)
ALTER TABLE [dbo].[PackageValidationSets] ADD [Expiration] [datetime2](7)
ALTER TABLE [dbo].[PackageValidationSets] ADD [ValidationProperties] [nvarchar](max)

-- Remove the not-null constraints on the PackageValidationSets table's PackageKey, PackageId,
-- and PackageNormalizedVersion columns
DECLARE @var0 nvarchar(128)
SELECT @var0 = name
FROM sys.default_constraints
WHERE parent_object_id = object_id(N'dbo.PackageValidationSets')
AND col_name(parent_object_id, parent_column_id) = 'PackageKey';
IF @var0 IS NOT NULL
    EXECUTE('ALTER TABLE [dbo].[PackageValidationSets] DROP CONSTRAINT [' + @var0 + ']')

ALTER TABLE [dbo].[PackageValidationSets] ALTER COLUMN [PackageKey] [int] NULL

DECLARE @var1 nvarchar(128)
SELECT @var1 = name
FROM sys.default_constraints
WHERE parent_object_id = object_id(N'dbo.PackageValidationSets')
AND col_name(parent_object_id, parent_column_id) = 'PackageId';
IF @var1 IS NOT NULL
    EXECUTE('ALTER TABLE [dbo].[PackageValidationSets] DROP CONSTRAINT [' + @var1 + ']')

ALTER TABLE [dbo].[PackageValidationSets] ALTER COLUMN [PackageId] [nvarchar](128) NULL

DECLARE @var2 nvarchar(128)
SELECT @var2 = name
FROM sys.default_constraints
WHERE parent_object_id = object_id(N'dbo.PackageValidationSets')
AND col_name(parent_object_id, parent_column_id) = 'PackageNormalizedVersion';
IF @var2 IS NOT NULL
    EXECUTE('ALTER TABLE [dbo].[PackageValidationSets] DROP CONSTRAINT [' + @var2 + ']')

ALTER TABLE [dbo].[PackageValidationSets] ALTER COLUMN [PackageNormalizedVersion] [nvarchar](64) NULL

-- Add foreign keys to PackageValidationResults
ALTER TABLE [dbo].[PackageValidationResults] ADD CONSTRAINT [FK_dbo.PackageValidationResults_dbo.PackageValidations_PackageValidationKey] FOREIGN KEY ([PackageValidationKey]) REFERENCES [dbo].[PackageValidations] ([Key])

ALTER TABLE [dbo].[PackageValidationResults] ADD CONSTRAINT [FK_dbo.PackageValidationResults_dbo.PackageValidationSets_PackageValidationSetKey] FOREIGN KEY ([PackageValidationSetKey]) REFERENCES [dbo].[PackageValidationSets] ([Key]) ON DELETE CASCADE

INSERT [dbo].[__MigrationHistory]([MigrationId], [ContextKey], [Model], [ProductVersion])
VALUES (N'202102022155232_AddGenericValidations', N'NuGet.Services.Validation.ValidationMigrationsConfiguration', <redacted>)

How I verified this

1. I created a copy of the DEV validation database
2. I applied the migration
3. I created a "generic" validation set using my seed program (see below)
4. I rolled back the migration
5. I verified the "generic" validation set
6. I re-applied the migration
7. I created a second "generic" validation set using my seed program
8. I verified both "generic" validation sets were as expected

Seed program...

using System;
using System.Collections.Generic;
using System.Data.Entity;
using System.Linq;
using NuGet.Services.Validation;

namespace SeedGenericValidatdions
{
    class Program
    {
        private const string ConnectionString = @"Data Source=(localdb)\mssqllocaldb; Initial Catalog=nuget-dev-validation; Integrated Security=True; MultipleActiveResultSets=True";
        static void Main(string[] args)
        {
            var validationTrackingId = Guid.NewGuid();

            Console.WriteLine($"Creating validation set '{validationTrackingId}'...");

            using (var ctx = new ValidationEntitiesContext(ConnectionString))
            {
                var genericValidationSet = new PackageValidationSet
                {
                    ValidationTrackingId = validationTrackingId,
                    PackageKey = null,
                    PackageId = null,
                    PackageNormalizedVersion = null,
                    PackageETag = null,
                    PackageContentType = "VsCodeExtensionV1",
                    Created = DateTime.UtcNow,
                    Updated = DateTime.UtcNow,
                    Expiration = DateTime.UtcNow.AddDays(1),
                    ValidatingType = ValidatingType.Generic,
                    ValidationSetStatus = ValidationSetStatus.InProgress,
                    ValidationProperties = "{}",

                    PackageValidations = new List<PackageValidation>
                    {
                        new PackageValidation
                        {
                            Type = "CvsContentScan",
                            Started = null,
                            ValidationStatus = ValidationStatus.NotStarted,
                            ValidationStatusTimestamp = DateTime.UtcNow,
                        }
                    }
                };

                ctx.PackageValidationSets.Add(genericValidationSet);
                ctx.SaveChanges();
            }

            Console.WriteLine($"Created validation set");

            Console.WriteLine($"Adding validation results...");

            using (var ctx = new ValidationEntitiesContext(ConnectionString))
            {
                var validationSet = ctx.PackageValidationSets.Single(s => s.ValidationTrackingId == validationTrackingId);
                var validationStep = ctx.PackageValidations.Single(s => s.PackageValidationSet.ValidationTrackingId == validationTrackingId);

                var result1 = new PackageValidationResult
                {
                    Type = "Diagnostic",
                    Data = @"{ ""Message"": ""Diagnostic on the validation SET!"" }",

                    PackageValidationSet = validationSet
                };

                var result2 = new PackageValidationResult
                {
                    Type = "Diagnostic",
                    Data = @"{ ""Message"": ""Diagnostic on the validation STEP!"" }",

                    PackageValidationSet = validationSet,
                    PackageValidation = validationStep
                };

                ctx.PackageValidationResults.Add(result1);
                ctx.PackageValidationResults.Add(result2);
                ctx.SaveChanges();
            }

            Console.WriteLine($"Added validation results");

            Console.WriteLine($"Verifying data...");

            using (var ctx = new ValidationEntitiesContext(ConnectionString))
            {
                var validationSet = ctx
                    .PackageValidationSets
                    .Include(s => s.PackageValidationResults)
                    .Single(s => s.ValidationTrackingId == validationTrackingId);

                var validationResults = ctx.PackageValidationResults
                    .Where(s => s.PackageValidationSet.ValidationTrackingId == validationTrackingId)
                    .Include(s => s.PackageValidationSet)
                    .Include(s => s.PackageValidation)
                    .ToList();

                Assert(validationSet.PackageValidationResults.Count() == 2, "The validation set should have two results.");
                Assert(validationResults.Count() == 2, "There should be two validation results.");

                Assert(validationResults.Count(r => r.PackageValidationKey.HasValue) == 1, "Only one validation set should be linked to a validation step.");
                Assert(validationResults.Count(r => r.PackageValidation != null) == 1, "Only one validation set should be linked to a validation step.");

                var stepResult = validationResults.Single(r => r.PackageValidation != null);
                var setResult = validationResults.Single(r => r.PackageValidation == null);

                Assert(stepResult.Data.Contains("validation STEP"), "Unexpected validation step result data");
                Assert(setResult.Data.Contains("validation SET"), "Unexpected validation set result data");
            }

            Console.WriteLine($"Verified data");
        }

        private static void Assert(bool condition, string message)
        {
            if (!condition)
            {
                var originalColor = Console.ForegroundColor;

                Console.ForegroundColor = ConsoleColor.Red;
                Console.Write("ERROR: ");
                Console.WriteLine(message);
                Console.ForegroundColor = originalColor;
            }
        }
    }
}

[loic-sharma]

Please make sure to review these changes: bc9fc85

Please also review the SQL generated by this migration. You can find this in the pull request's description.

[joelverhagen]

Good stuff. Nice test pass. I'm a little sad the schema is "weakening" with more nullable but I understand the desire to pick the "right" level of engineering effort. It'll probably be fine.

[loic-sharma]

@agr I checked on PROD validation database which constraints would be dropped by the generated migration. It turns out no constraints would be affected, so the constraint drops would no-op.

SQL script to find constraints that would be dropped...

SELECT name
FROM sys.default_constraints
WHERE parent_object_id = object_id(N'dbo.PackageValidationSets')
AND col_name(parent_object_id, parent_column_id) = 'PackageKey';

SELECT name
FROM sys.default_constraints
WHERE parent_object_id = object_id(N'dbo.PackageValidationSets')
AND col_name(parent_object_id, parent_column_id) = 'PackageId';

SELECT name
FROM sys.default_constraints
WHERE parent_object_id = object_id(N'dbo.PackageValidationSets')
AND col_name(parent_object_id, parent_column_id) = 'PackageNormalizedVersion';

[loic-sharma]

This matches the Gallery DB's migration timeout: https://github.com/NuGet/NuGetGallery/blob/4bd46fcf6f366b5343f21c4cef1abdfe67d06c27/src/NuGetGallery/Migrations/MigrationsConfiguration.cs#L16