Excessive memory usage

[yreynhout]

I know that's a little vague but if you insert lots (>100000) of events in a loop using sql persistence, I notice lots of memory is used and not freed up. I'm not sure it's an actual issue, just weird that it keeps growing. At first I thought it was the tracker, but that seems gone from the source. I must be doing something wrong ...

I observed this behavior using a compiled version of 403e4ce

Code I used during which the behavior was observed (.NET 4.0):

    using System;
    using System.Collections.Generic;
    using System.Diagnostics;
    using System.Threading.Tasks;
    using EventStore;

    namespace PerfWise {
      class Program {
        static void Main(string[] args) {
          var store = Wireup.Init().
            UsingSqlPersistence("eStore").InitializeDatabaseSchema().
            UsingServiceStackJsonSerialization().Build();

          var thousandSchedulesWith5YearsWorthOfData = 365*5*1000;
          var watch = Stopwatch.StartNew();
          var start = DateTime.Now;
          Parallel.For(
            0,
            thousandSchedulesWith5YearsWorthOfData,
            new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount },
            i => {
              Console.WriteLine("Writing {0}", i);
              var streamId = Guid.NewGuid();
              using (var stream = store.CreateStream(streamId)) {
                stream.Add(new EventMessage {
                  Body = new ScheduleDayCreatedEvent(streamId, 1, Guid.NewGuid(), DateTime.Today.AddDays(i),
                    new[] {
                           new ScheduleDaySessionData(1, TimeSpan.Parse("08:00:00"), TimeSpan.Parse("12:00:00"), "Morning", "Morn", 1.1, Guid.NewGuid(), 
                             new [] {
                                      new ScheduleDayProgramData(1, TimeSpan.Parse("08:00:00"), TimeSpan.Parse("12:00:00"), 2, Guid.NewGuid())
                                    }
                             ), 
                          new ScheduleDaySessionData(2, TimeSpan.Parse("14:00:00"), TimeSpan.Parse("17:00:00"), "Afternoon", "Aft", 1.2, Guid.NewGuid(), 
                             new [] {
                                      new ScheduleDayProgramData(1, TimeSpan.Parse("14:00:00"), TimeSpan.Parse("17:00:00"), 1.5, Guid.NewGuid())
                                    }
                             ), 
                  })
                });
                stream.CommitChanges(Guid.NewGuid());
              }
            });
          watch.Stop();
          var end = DateTime.Now - start;
          Console.WriteLine("Done in (stopwatch) {0}ms or (datetime) {1}ms.", watch.ElapsedMilliseconds, end.TotalMilliseconds);
          Console.ReadLine();
        }
      }

      public class ScheduleDayCreatedEvent : IEvent {
        public Guid ScheduleDayId { get; private set; }
        public long Version { get; private set; }
        public Guid ScheduleId { get; private set; }
        public DateTime Date { get; private set; }
        public ICollection<ScheduleDaySessionData> Sessions { get; private set; }

        public ScheduleDayCreatedEvent(Guid scheduleDayId, long version, Guid scheduleId, DateTime date, ICollection<ScheduleDaySessionData> sessions) {
          ScheduleDayId = scheduleDayId;
          Version = version;
          ScheduleId = scheduleId;
          Date = date;
          Sessions = sessions;
        }
      }

      public class ScheduleDaySessionData {
        public int SessionSequence { get; private set; }
        public TimeSpan From { get; private set; }
        public TimeSpan To { get; private set; }
        public string Name { get; private set; }
        public string Abbreviation { get; private set; }
        public double MaximumFillPercentage { get; private set; }
        public Guid LocationId { get; private set; }
        public ICollection<ScheduleDayProgramData> Programs { get; private set; }

        public ScheduleDaySessionData(int sessionSequence, TimeSpan from, TimeSpan to, string name, string abbreviation, double maximumFillPercentage, Guid locationId, ICollection<ScheduleDayProgramData> programs) {
          SessionSequence = sessionSequence;
          From = from;
          To = to;
          Name = name;
          Abbreviation = abbreviation;
          MaximumFillPercentage = maximumFillPercentage;
          LocationId = locationId;
          Programs = programs;
        }
      }

      public class ScheduleDayProgramData {
        public int ProgramSequence { get; private set; }
        public TimeSpan From { get; private set; }
        public TimeSpan To { get; private set; }
        public double MaximumFillPercentage { get; private set; }
        public Guid ProgramId { get; private set; }

        public ScheduleDayProgramData(int programSequence, TimeSpan from, TimeSpan to, double maximumFillPercentage, Guid programId) {
          ProgramSequence = programSequence;
          From = from;
          To = to;
          MaximumFillPercentage = maximumFillPercentage;
          ProgramId = programId;
        }
      }

      public interface IEvent {}
    }

[yreynhout]

Okay, I managed to setup a memory profiling session using dotTrace's memory profiler. The TrackStream seems to have moved into the OptimisticPipelineHook, which is the culprit of all this. Next to a maximum for the number of commits tracked per stream, maybe there should be a maximum number of streams tracked. There should be a choice of tracking or not tracking in the WireUp extensions,in essence more control over the pipeline using e.g. an IPipelineHookBuilder. This way there would be choice between the built-in builder or a custom builder. There's a good example of this pipeline building in WSE3 (yeah, I know, don't ask).

[joliver]

Could you resubmit this as a new issue? For some reason it got closed like you said...

Also, I'm looking at potentially using a bloom filter to keep memory usage down.

[yreynhout]

Sure, I'll do that.
I also forked your repo to take a stab at providing a pipeline builder.

Regards,
Yves.

Verstuurd vanaf mijn iPhone

Op 8-apr.-2011 om 05:14 heeft joliverreply@reply.github.com het volgende geschreven:

Could you resubmit this as a new issue? For some reason it got closed like you said...

Also, I'm looking at potentially using a bloom filter to keep memory usage down.

Reply to this email directly or view it on GitHub:
#34 (comment)