Skip to content

Commit

Permalink
fix BigStringEmbedded
Browse files Browse the repository at this point in the history
  • Loading branch information
@olmobrutall
olmobrutall committed Aug 29, 2020
1 parent 5ae9569 commit 726165e
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 8 deletions.
4 changes: 3 additions & 1 deletion Signum.Engine/Basics/ExceptionLogic.cs
View file
Expand Up @@ -74,7 +74,9 @@ static ExceptionEntity GetEntity(Exception ex)
entity.ThreadId = Thread.CurrentThread.ManagedThreadId;
entity.ApplicationName = Schema.Current.ApplicationName;
entity.HResult = ex.HResult;

entity.Form = new BigStringEmbedded();
entity.QueryString = new BigStringEmbedded();
entity.Session = new BigStringEmbedded();

entity.Environment = CurrentEnvironment;
try
Expand Down
14 changes: 7 additions & 7 deletions Signum.Entities/Basics/Exception.cs
View file
Expand Up @@ -46,7 +46,7 @@ public string ExceptionMessage

public int ExceptionMessageHash { get; private set; }

BigStringEmbedded stackTrace;
BigStringEmbedded stackTrace = new BigStringEmbedded();
[NotifyChildProperty]
public BigStringEmbedded StackTrace
{
Expand Down Expand Up @@ -98,16 +98,16 @@ public BigStringEmbedded StackTrace
public string? UserHostName { get; set; }

[NotifyChildProperty]
public BigStringEmbedded Form { get; set; }
public BigStringEmbedded Form { get; set; } = new BigStringEmbedded();

[NotifyChildProperty]
public BigStringEmbedded QueryString { get; set; }
public BigStringEmbedded QueryString { get; set; } = new BigStringEmbedded();

[NotifyChildProperty]
public BigStringEmbedded Session { get; set; }
public BigStringEmbedded Session { get; set; } = new BigStringEmbedded();

[NotifyChildProperty]
public BigStringEmbedded Data { get; set; }
public BigStringEmbedded Data { get; set; } = new BigStringEmbedded();

public int HResult { get; internal set; }

Expand Down
5 changes: 5 additions & 0 deletions Signum.Entities/Basics/OperationLog.cs
View file
Expand Up @@ -8,6 +8,11 @@ namespace Signum.Entities.Basics
[Serializable, EntityKind(EntityKind.System, EntityData.Transactional), TicksColumn(false), InTypeScript(Undefined = false)]
public class OperationLogEntity : Entity
{
public OperationLogEntity()
{
RebindEvents();
}

[ImplementedByAll]
public Lite<IEntity>? Target { get; set; }

Expand Down

2 comments on commit 726165e

@olmobrutall
Copy link
Collaborator Author

@olmobrutall olmobrutall commented on 726165e Aug 31, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Introducing BigStringEmbedded

Signum Framework has a few tables used for logging: OperationLog, ViewLog, Exception, EmailMessage, etc...

When stored in the database, this tables tend to grow big, because big chunks of text are saved for each entity, and many entities are created every day.

BigStringEmbedded is an abstraction in Framework to store this big entities in the database first.

   [Serializable]
   public class BigStringEmbedded : EmbeddedEntity
   {
       [DbType(Size = int.MaxValue)]
       public string? Text { get; set; }
   }

but by starting BigStringLogic (in Extensions) and adding the BigStringMixin to BitStringEmbedded (yes! remember mixins can be added to embedded entities now!), you can move this big strings to the file system / azure storage.

    [Serializable]
    public class BigStringMixin : MixinEntity
    {
        BigStringMixin(ModifiableEntity mainEntity, MixinEntity? next)
            : base(mainEntity, next)
        {
        }

        public FilePathEmbedded? File { get; set; }
    }

Some numbers

In a real application where OperationLogs where a problem

  Before After BigString in Azure Storage
Total Database size 37,93 GB 3,62 GB
OperationLog table 34,89 GB 0,59 GB
Daily Full Backup 5,05 GB 0,9 GB

I've also Shrinked the mdf file from 50 GB to 10GB.

The funny thing is that, in Azure Storage, the 34GB have become just a few hundred MBs.

Looks like storing this big strings in the database is a very bad idea, so update you applications to BigStringEmbedded!

How to update`

Just get the latest version of Framework and Extensions and synchronize and you will see that columns like StackTrace have been renamed to StackTrace_Text. If you want you can keep it like this and you won't see any important change.

But if you want to get move the logs check this southwind commit: signumsoftware/southwind@d1bf677

Basically:

  • Call BigStringLogic.Start(sb) in your Start method.
  • This will require that you associate the mixin MixinDeclarations.Register<BigStringEmbedded, BigStringMixin>();
  • Also you need to call BigString.Register / BigString.RegisterAll for each type with a BigStringEmbedded. This method defines where each field is going to be stored. There are 4 modes, some requiring a FileTypeSymbol:
    public enum BigStringMode 
    {
        /// <summary>
        /// Only Text column in database
        /// </summary>
        Database, 
        /// <summary>
        /// Only File column in database pointing to a path in the file system / blob storage
        /// </summary>
        FileSystem, 
        /// <summary>
        /// Migrating from Text -> File
        /// </summary>
        Migrating_FromDatabase_ToFileSystem,
        /// <summary>
        /// Migrating from File -> Text
        /// </summary>
        Migrating_FromFileSystem_ToDatabase,
    }

  • So in order to migrate and preserve the logs, I recommend to use BigString.Migrating_FromDatabase_ToFileSystem, sync (could take some minutes).

  • Then make a CSharp migration that calls:

BigStringLogic.MigrateBigString<ExceptionEntity>();
BigStringLogic.MigrateBigString<OperationLogEntity>();
BigStringLogic.MigrateBigString<ViewLogEntity>();
BigStringLogic.MigrateBigString<EmailMessageEntity>();
...

This will save every single log and move the BigStrings to the file system / blob storage. This could take a long time (hours), so better do it at night.

That's it! Enjoy a small and lean database!

@MehdyKarimpour
Copy link
Contributor

@MehdyKarimpour MehdyKarimpour commented on 726165e Aug 31, 2020 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.