Slow performance on developer instance

[lukee48]

Sorry if this is the wrong place to ask

We are looking to migrate to a documentstore and are currently trialing a few different providers, mongo, raven ect

We setup a free tier instance but this was painfully slow trying to query from our AWS lambdas (Ireland)

We changed our instance to a developer instance since we could then host the cluster in Ireland also. Still painfully slow, we have to increase our lambda timeouts to 300 seconds just do get it working.

Ideally we don't want to have to go onto more expensive instances while we evaluate, our company is a pain at signing of the slightest thing

Are we doing anything wrong? Are the developer instances suppose to be this slow?

[arekpalinski]

Hi @lukee48

Can you elaborate more on the slowness that you experience? When do you see that (requests handling, indexing) and what performance numbers are we talking about?

The developer instances aren't supposed to be slow although we cannot expect much performance from the smallest instances (e.g. Dev10 has only 0.5GB of memory, note that there is also operating system running there). They are meant for very low workload. That Dev tier instances are burstable:

https://ravendb.net/docs/article-page/6.0/csharp/cloud/cloud-instances#a-development-cloud-server
https://ravendb.net/docs/article-page/6.0/csharp/cloud/cloud-overview#burstable-instances

and the smallest ones have pretty small CPU credits budget. If you did some operations that could it consume it within small span time then the instance need some time to reclaim that. You can check the CPU credits in Metrics tab of your instance in RavenDB Cloud portal.

Another thing you might want to check are the following resources about using RavenDB from AWS lambda (CC @kamranayub):

#16547
https://ravendb.net/docs/article-page/5.4/csharp/start/guides/aws-lambda/overview

In particular, maybe you're hitting a cold start issue in AWS lambda:
https://ravendb.net/docs/article-page/5.4/csharp/start/guides/aws-lambda/deployment#document-store-lifetime-and-cold-starts

[kamranayub]

Yes, @lukee48 please check out the AWS guides and let us know if that helps. With our templates, I was not seeing my test Lambdas that slow, but there is a small cold start cost.

[lukee48]

@arekpalinski @kamranayub

So we did some digging into this

We adding logging and confirmed it is not directly related to cold starts (But they don't help)

To init the Raven Store takes about 10 seconds. Executing the indexes takes around 20 seconds.

Baring in mind, this is a database with no data in it and only has two simple indexes.

Are these expected times for a low dev instance? Or are we doing something fundamentally wrong?

[lukee48]

` services.AddSingleton(provider =>
{
var certPrivateKey = Environment.GetEnvironmentVariable("RAVEN_PRIVATE_KEY");
var certPublicKeyFilePath = Environment.GetEnvironmentVariable("RAVEN_PUBLIC_KEY_PATH");
var databaseUrl = Environment.GetEnvironmentVariable("RAVEN_DATABASE_URL");

            var certPem = File.ReadAllText(certPublicKeyFilePath);
            
            var intermediateCert = X509Certificate2.CreateFromPem(certPem, certPrivateKey);
            var cert = new X509Certificate2(intermediateCert.Export(X509ContentType.Pfx));
            intermediateCert.Dispose();

            var store = new DocumentStore
            {
                Database = "trial",
                Urls = new []{databaseUrl},
                Certificate = cert
            };

            store.Conventions.UseOptimisticConcurrency = true;
            store.Conventions.IdentityPartsSeparator = '-';

            Console.WriteLine("init store");
            
            store.Initialize();
            
            Console.WriteLine("Store initiated");

            store.ExecuteIndex(new TestIndexOne());
            store.ExecuteIndex(new TestIndexTwo());
            
            Console.WriteLine("Indexes Ran");
            
            return store;
        }); `

[arekpalinski]

Just to make sure, by saying:

To init the Raven Store takes about 10 seconds

you mean that the execution of this line is taking 10 seconds:

store.Initialize();

?

This is purely client side code and it doesn't interact with the server:

ravendb/src/Raven.Client/Documents/DocumentStore.cs

Lines 242 to 275 in 0d38559

	public override IDocumentStore Initialize()
	{
	if (Initialized)
	return this;
	AssertValidConfiguration();
	RequestExecutor.ValidateUrls(Urls, Certificate);
	if (Certificate != null && Certificate.HasPrivateKey == false)
	throw new InvalidOperationException(
	$"The supplied {Certificate.FriendlyName} certificate contains no private key. " +
	"Constructing the certificate with the 'X509KeyStorageFlags.MachineKeySet' flag may solve this problem.");
	try
	{
	if (Conventions.AsyncDocumentIdGenerator == null) // don't overwrite what the user is doing
	{
	var generator = new AsyncMultiDatabaseHiLoIdGenerator(this);
	_asyncMultiDbHiLo = generator;
	Conventions.AsyncDocumentIdGenerator = (database, entity) => generator.GenerateDocumentIdAsync(database, entity);
	}
	Conventions.Freeze();
	Initialized = true;
	}
	catch (Exception)
	{
	Dispose();
	throw;
	}
	return this;
	}

On the first attempt to send any request to the server the connection will be established. So in your case it will happen on store.ExecuteIndex(new TestIndexOne()); which is just sending an index definition to the server. 20 seconds for that sounds crazy.

You can check the actual request durations on the server by using Traffic Watch view (RavenDB Studio -> Manage Server -> Traffic Watch).

[lukee48]

#17601 (comment)

[lukee48]

Ok, interesting, you can see our cloudwatch logs here that lineup with the code I added above, with the logs

Interestingly though, yes, everything looks healthy in our Traffic Watch.

[ayende]

Index creation can be expensive, we need to scan all the types in the assembly, figuring out Linq, etc.
You don't need them as a Lambda, no.

[lukee48]

Preferably all our services would be serverless.

So for any service that required indexes, they shouldn't be a lambda? Is that correct?

[ayende]

No, you can do something separate, spawn a thread to create indexes as needed.
Or do that as part of the deployment. Most of the time, they don't change.

[lukee48]

Sorry for all the messages, just so I can make sure I have understood correctly

1. We can execute our indexes in our pipelines.
2. We can then deploy our lambda that can query the indexes without having to execute them, meaning we don't get that initial index creation performance issue

Does that sound right?

[kamranayub]

@lukee48 Yeah, that's right. For example, what I do is have a single background jobs worker that includes that index execution line, but my web app and serverless functions don't.

At deployment time, everything gets deployed so once the worker boots up, it updates any indexes. The background worker is continuous, so it doesn't matter how long it takes to do everything on initialization. And indexes are only ever changed within the code, so any updates are always part of a CI/CD deployment.

Technically, this can mean that the serverless functions or web app are querying indexes that may not exist yet or haven't been updated. But in practice, that isn't much of an issue, since requests can be retried.

This could be good guidance to add to the serverless deployment guides.