Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add migrations for new block editor #7910

Merged
merged 7 commits into from Apr 14, 2020
Merged

Add migrations for new block editor #7910

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f78e4fc
Add migrations for new block editor
benjaminc Apr 7, 2020
d6c95bb
Handle color picker data in stacked content
benjaminc Apr 7, 2020
432ce80
Merge branch 'v8/feature/block-editor-list' into v8/feature/block-edi…
benjaminc Apr 8, 2020
ff8ef00
Incorporate latest block list editor changes, update migration for ch…
benjaminc Apr 8, 2020
e7be4bd
Merge branch 'v8/feature/block-editor-list' into v8/feature/block-edi…
benjaminc Apr 8, 2020
133d307
Handle invalid data type references
benjaminc Apr 8, 2020
e8a31ef
Remove code duplicated from PR #7957
benjaminc Apr 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/Umbraco.Core/Migrations/Upgrade/UmbracoPlan.cs
View file
Open in desktop
Expand Up @@ -194,6 +194,10 @@ protected void DefinePlan()


To<MissingDictionaryIndex>("{a78e3369-8ea3-40ec-ad3f-5f76929d2b20}");

// to 8.7.0...
To<StackedContentToBlockList>("{DFA35FA2-BFBB-433F-84E5-BD75940CDDF6}");
To<ConvertToElements>("{DA434576-3DEF-46D7-942A-CE34D7F7FB8A}");
//FINAL
}
}
Expand Down
78 changes: 78 additions & 0 deletions src/Umbraco.Core/Migrations/Upgrade/V_8_7_0/ConvertToElements.cs
View file
Open in desktop
@@ -0,0 +1,78 @@
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System.Collections.Generic;
using System.Linq;
using Umbraco.Core.Persistence;
using Umbraco.Core.Persistence.Dtos;

namespace Umbraco.Core.Migrations.Upgrade.V_8_7_0
{
public class ConvertToElements : MigrationBase
Shazwazza marked this conversation as resolved.
Show resolved Hide resolved
{
public ConvertToElements(IMigrationContext context) : base(context)
{
}

public override void Migrate()
{
// Get all document type IDs by alias
var docTypes = Database.Fetch<ContentTypeDto>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not just do Database.Fetch<ContentTypeDto>().ToDictionary(x => x.Alias, x => x.NodeId) ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had written an app a couple years back that did a lot of list to dictionary conversions, and it was performing horribly. After doing using VS's performance analyzer, we found that the dictionary resize operations are extremely expensive, especially as the list grows to 1,000+ elements. If you size the dictionary appropriately at the start and then just insert into it, it was lightning fast. Ever since then, I've always tried to make it a habit to size the dictionary appropriately and then insert into it. Since the ToDictionary extension is on the IEnumerable interface, it can't do that, and so just iterates and adds to the dictionary. If they created a ToDictionary that was an extension on the ICollection interface, that would be ideal, but without that I just create my own, properly sized dictionary and add to it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough! Might be worth making an ext method for this then based on ICollection, not required for this PR but in the future. And good to know! Wonder if there are places in our codebase where we are creating large dictionaries with ToDictionary based on collections, might have a look today and see.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided to create a PR for changing some of our internal usages of ToDictionary and discovered we should be using ToLookup in various places too instead of GroupBy/ToDictionary which is much slower. I've left this open for the community to review but please have a look i fyou have time #7969

var docTypeMap = new Dictionary<string, int>(docTypes.Count);
docTypes.ForEach(d => docTypeMap[d.Alias] = d.NodeId);

// Find all Nested Content or Block List data types
var dataTypes = GetDataTypes(Constants.PropertyEditors.Aliases.BlockList);

// Find all document types listed in each
var elementTypeIds = dataTypes.SelectMany(d => GetDocTypeIds(d.Configuration, docTypeMap)).ToList();

// Find all compositions those document types use
var parentElementTypeIds = Database.Fetch<ContentType2ContentTypeDto>(Sql()
.Select<ContentType2ContentTypeDto>()
.From<ContentType2ContentTypeDto>()
.WhereIn<ContentType2ContentTypeDto>(c => c.ChildId, elementTypeIds)
).Select(c => c.ParentId);

elementTypeIds = elementTypeIds.Union(parentElementTypeIds).ToList();

// Convert all those document types to element type
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any chance this can break people's sites? Like if potentially they were using a doc type as both an element type and a doc type?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is, and I thought about that further when I was writing the migrations that I did for #7939 where I decided that if there were any element types already (i.e. this wasn't a v7 to v8 migration), then I wouldn't mess with retyping any document types. I'll port that change back to here, to minimize the effect on anyone upgrading from a previous v8 version. We are dealing just with those upgrading Stacked Content, so likely they are still coming straight from v7 anyway, but it would be a good quick check.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool so since this is already merged i supposed we will just wait until you have a little bit of time to push a new PR, just let us know :) I'll add some notes to the code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as a follow-up to this, I ended up pulling the ConvertToElement migration in PR #8336 since the NestedContent portion is PR'd in #7957 and without the StackedContent to BlockEditorList migration, there is nothing else to convert.

foreach (var docType in docTypes)
{
if (!elementTypeIds.Contains(docType.NodeId)) continue;

docType.IsElement = true;
Database.Update(docType);
}
}

private List<DataTypeDto> GetDataTypes(params string[] aliases)
{
var sql = Sql()
.Select<DataTypeDto>()
.From<DataTypeDto>()
.WhereIn<DataTypeDto>(d => d.EditorAlias, aliases);

return Database.Fetch<DataTypeDto>(sql);
}

private IEnumerable<int> GetDocTypeIds(string configuration, Dictionary<string, int> idMap)
{
if (configuration.IsNullOrWhiteSpace() || configuration[0] != '{') return Enumerable.Empty<int>();

var obj = JObject.Parse(configuration);
if (obj["blocks"] is JArray blArr)
{
var arr = blArr.ToObject<BlockConfiguration[]>();
return arr.Select(i => idMap.TryGetValue(i.Alias, out var id) ? id : 0).Where(i => i != 0);
}

return Enumerable.Empty<int>();
}

public class BlockConfiguration
{
[JsonProperty("contentTypeAlias")]
public string Alias { get; set; }
}
}
}
98 changes: 80 additions & 18 deletions src/Umbraco.Core/Migrations/Upgrade/V_8_7_0/StackedContentToBlockList.cs
View file
Open in desktop
Expand Up @@ -20,6 +20,7 @@ public StackedContentToBlockList(IMigrationContext context) : base(context)

public override void Migrate()
{
// Convert all Stacked Content properties to Block List properties, both in the data types and in the property data
var refreshCache = Migrate(GetDataTypes("Our.Umbraco.StackedContent"), GetKnownDocumentTypes());

// if some data types have been updated directly in the database (editing DataTypeDto and/or PropertyDataDto),
Expand All @@ -38,7 +39,7 @@ private List<DataTypeDto> GetDataTypes(string alias)
return Database.Fetch<DataTypeDto>(sql);
}

private Dictionary<Guid, string> GetKnownDocumentTypes()
private Dictionary<Guid, KnownContentType> GetKnownDocumentTypes()
{
var sql = Sql()
.Select<ContentTypeDto>(r => r.Select(x => x.NodeDto))
Expand All @@ -47,12 +48,40 @@ private List<DataTypeDto> GetDataTypes(string alias)
.On<ContentTypeDto, NodeDto>(c => c.NodeId, n => n.NodeId);

var types = Database.Fetch<ContentTypeDto>(sql);
var map = new Dictionary<Guid, string>(types.Count);
types.ForEach(t => map[t.NodeDto.UniqueId] = t.Alias);
return map;
var typeMap = new Dictionary<int, ContentTypeDto>(types.Count);
Shazwazza marked this conversation as resolved.
Show resolved Hide resolved
types.ForEach(t => typeMap[t.NodeId] = t);

sql = Sql()
.Select<ContentType2ContentTypeDto>()
.From<ContentType2ContentTypeDto>();
var joins = Database.Fetch<ContentType2ContentTypeDto>(sql);
// Find all relationships between types, either inherited or composited
var joinLk = joins
.Union(types
.Where(t => typeMap.ContainsKey(t.NodeDto.ParentId))
.Select(t => new ContentType2ContentTypeDto { ChildId = t.NodeId, ParentId = t.NodeDto.ParentId }))
.ToLookup(j => j.ChildId, j => j.ParentId);

sql = Sql()
.Select<PropertyTypeDto>(r => r.Select(x => x.DataTypeDto))
.From<PropertyTypeDto>()
.InnerJoin<DataTypeDto>()
.On<PropertyTypeDto, DataTypeDto>(c => c.DataTypeId, n => n.NodeId)
.WhereIn<DataTypeDto>(d => d.EditorAlias, new[] { Constants.PropertyEditors.Aliases.NestedContent, Constants.PropertyEditors.Aliases.ColorPicker });
var props = Database.Fetch<PropertyTypeDto>(sql);
// Get all nested content and color picker property aliases by content type ID
var propLk = props.ToLookup(p => p.ContentTypeId, p => p.Alias);

var knownMap = new Dictionary<Guid, KnownContentType>(types.Count);
types.ForEach(t => knownMap[t.NodeDto.UniqueId] = new KnownContentType
Shazwazza marked this conversation as resolved.
Show resolved Hide resolved
{
Alias = t.Alias,
StringToRawProperties = propLk[t.NodeId].Union(joinLk[t.NodeId].SelectMany(r => propLk[r])).ToArray()
});
return knownMap;
}

private bool Migrate(IEnumerable<DataTypeDto> dataTypesToMigrate, Dictionary<Guid, string> knownDocumentTypes)
private bool Migrate(IEnumerable<DataTypeDto> dataTypesToMigrate, Dictionary<Guid, KnownContentType> knownDocumentTypes)
{
var refreshCache = false;

Expand All @@ -73,16 +102,16 @@ private bool Migrate(IEnumerable<DataTypeDto> dataTypesToMigrate, Dictionary<Gui
return refreshCache;
}

private BlockListConfiguration UpdateConfiguration(DataTypeDto dataType, Dictionary<Guid, string> knownDocumentTypes)
private BlockListConfiguration UpdateConfiguration(DataTypeDto dataType, Dictionary<Guid, KnownContentType> knownDocumentTypes)
{
var old = JsonConvert.DeserializeObject<StackedContentConfiguration>(dataType.Configuration);
var config = new BlockListConfiguration
{
Blocks = old.ContentTypes?.Select(t => new BlockListConfiguration.BlockConfiguration
{
Alias = knownDocumentTypes[t.IcContentTypeGuid],
Alias = knownDocumentTypes.TryGetValue(t.IcContentTypeGuid, out var ct) ? ct.Alias : null,
Label = t.NameTemplate
}).ToArray(),
}).Where(c => c.Alias != null).ToArray(),
Shazwazza marked this conversation as resolved.
Show resolved Hide resolved
UseInlineEditingAsDefault = old.SingleItemMode == "1" || old.SingleItemMode == bool.TrueString
};

Expand All @@ -96,7 +125,7 @@ private BlockListConfiguration UpdateConfiguration(DataTypeDto dataType, Diction
return config;
}

private void UpdatePropertyData(DataTypeDto dataType, BlockListConfiguration config, Dictionary<Guid, string> knownDocumentTypes)
private void UpdatePropertyData(DataTypeDto dataType, BlockListConfiguration config, Dictionary<Guid, KnownContentType> knownDocumentTypes)
{
// get property data dtos
var propertyDataDtos = Database.Fetch<PropertyDataDto>(Sql()
Expand All @@ -115,7 +144,7 @@ private void UpdatePropertyData(DataTypeDto dataType, BlockListConfiguration con
}


private bool UpdatePropertyDataDto(PropertyDataDto dto, BlockListConfiguration config, Dictionary<Guid, string> knownDocumentTypes)
private bool UpdatePropertyDataDto(PropertyDataDto dto, BlockListConfiguration config, Dictionary<Guid, KnownContentType> knownDocumentTypes)
{
var model = new SimpleModel();

Expand All @@ -141,11 +170,11 @@ private void UpdateDataType(DataTypeDto dataType)
private class BlockListConfiguration
{

[ConfigurationField("blocks", "Available Blocks", "views/propertyeditors/blocklist/prevalue/blocklist.elementtypepicker.html", Description = "Define the available blocks.")]
[JsonProperty("blocks")]
benjaminc marked this conversation as resolved.
Show resolved Hide resolved
public BlockConfiguration[] Blocks { get; set; }


[ConfigurationField("validationLimit", "Amount", "numberrange", Description = "Set a required range of blocks")]
[JsonProperty("validationLimit")]
public NumberRange ValidationLimit { get; set; } = new NumberRange();

public class NumberRange
Expand All @@ -159,6 +188,15 @@ public class NumberRange

public class BlockConfiguration
{
[JsonProperty("backgroundColor")]
public string BackgroundColor { get; set; }

[JsonProperty("iconColor")]
public string IconColor { get; set; }

[JsonProperty("thumbnail")]
public string Thumbnail { get; set; }

[JsonProperty("contentTypeAlias")]
public string Alias { get; set; }

Expand All @@ -170,11 +208,16 @@ public class BlockConfiguration

[JsonProperty("label")]
public string Label { get; set; }

[JsonProperty("editorSize")]
public string EditorSize { get; set; }
}

[ConfigurationField("useInlineEditingAsDefault", "Inline editing mode", "boolean", Description = "Use the inline editor as the default block view")]
[JsonProperty("useInlineEditingAsDefault")]
public bool UseInlineEditingAsDefault { get; set; }

[JsonProperty("maxPropertyWidth")]
public string MaxPropertyWidth { get; set; }
}

private class StackedContentConfiguration
Expand Down Expand Up @@ -202,18 +245,31 @@ private class SimpleModel
[JsonProperty("data")]
public List<JObject> Data { get; } = new List<JObject>();

public void AddDataItem(JObject obj, Dictionary<Guid, string> knownDocumentTypes)
public void AddDataItem(JObject obj, Dictionary<Guid, KnownContentType> knownDocumentTypes)
{
if (!Guid.TryParse(obj["key"].ToString(), out var key)) throw new ArgumentException("Could not find a valid key in the data item");
if (!Guid.TryParse(obj["icContentTypeGuid"].ToString(), out var ctGuid)) throw new ArgumentException("Could not find a valid content type GUID in the data item");
if (!knownDocumentTypes.TryGetValue(ctGuid, out var ctAlias)) throw new ArgumentException($"Unknown content type GUID '{ctGuid}'");
if (!Guid.TryParse(obj["key"].ToString(), out var key)) key = Guid.NewGuid();
if (!Guid.TryParse(obj["icContentTypeGuid"].ToString(), out var ctGuid)) ctGuid = Guid.Empty;
if (!knownDocumentTypes.TryGetValue(ctGuid, out var ct)) ct = new KnownContentType { Alias = ctGuid.ToString() };

obj.Remove("key");
obj.Remove("icContentTypeGuid");

var udi = new GuidUdi(Constants.UdiEntityType.Element, key).ToString();
obj["udi"] = udi;
obj["contentTypeAlias"] = ctAlias;
obj["contentTypeAlias"] = ct.Alias;

if (ct.StringToRawProperties != null && ct.StringToRawProperties.Length > 0)
{
// Nested content inside a stacked content item used to be stored as a deserialized string of the JSON array
// Now we store the content as the raw JSON array, so we need to convert from the string form to the array
foreach (var prop in ct.StringToRawProperties)
{
var val = obj[prop];
var value = val?.ToString();
if (val != null && val.Type == JTokenType.String && !value.IsNullOrWhiteSpace())
obj[prop] = JsonConvert.DeserializeObject<JToken>(value);
}
}

Data.Add(obj);
Layout.Refs.Add(new SimpleLayout.SimpleLayoutRef { Udi = udi });
Expand All @@ -231,5 +287,11 @@ public class SimpleLayoutRef
}
}
}

private class KnownContentType
{
public string Alias { get; set; }
public string[] StringToRawProperties { get; set; }
}
}
}
2 changes: 2 additions & 0 deletions src/Umbraco.Core/Umbraco.Core.csproj
View file
Open in desktop
Expand Up @@ -131,6 +131,8 @@
<Compile Include="Migrations\Upgrade\V_8_0_0\Models\ContentTypeDto80.cs" />
<Compile Include="Migrations\Upgrade\V_8_0_0\Models\PropertyDataDto80.cs" />
<Compile Include="Migrations\Upgrade\V_8_0_0\Models\PropertyTypeDto80.cs" />
<Compile Include="Migrations\Upgrade\V_8_7_0\ConvertToElements.cs" />
<Compile Include="Migrations\Upgrade\V_8_7_0\StackedContentToBlockList.cs" />
<Compile Include="Models\Blocks\IBlockEditorDataHelper.cs" />
<Compile Include="Models\InstallLog.cs" />
<Compile Include="Persistence\Repositories\IInstallationRepository.cs" />
Expand Down