Skip to content

Loading…

Massive only inserts first 4k of text #37

Closed
wants to merge 1 commit into from

2 participants

@codeimpossible

I've run into an issue with inserting large amounts of text (greater than 4000 characters) with Massive. Massive sets the length for all string parameters to 4000 without checking to see if their length is longer. I added a check for this and have posted a link to a console app I wrote to test the bug and the fix as well.

To run the test app, create a db on a local sqlexpress install named "BlogExample", then run the console app.

http://dl.dropbox.com/u/6291954/MassiveTest.zip

@datachomp

Jared, you are creating a poison cache scenario with that fix.

In addition, the variable used is of nvarchar type by default, so you are really passing 8000 bytes worth which is really close to blowing up a row when you exceed that. If you have modified your version to use varchar() Then you should be able to set it to varchar(8000) when > 4000 to avoid plan cache poison. If > 8000 you can consider changing to varchar(max) type, but bear in mind that if this is mostly used for SQL Server CE... that it might struggle with that datatype.

@codeimpossible

Ouch! I didn't know that the query plan caching depended on param sizes. Sounds like I need to do some reading and a lot more testing before submitting a fix for this. I'll close the pull request. Thanks for the info!

@datachomp

Ha, don't feel bad. All of microsoft did it with Linq to Sql and EF until .Net 4 :)

@codeimpossible

So, how would I use Massive to handle inserting a large blob of text? If I changed my patch to up the length in stages as you mentioned, and I'm talking about going against an nvarchar(N) or nvarchar(max) column here (not using varchars), that should keep the cache under control correct?

@datachomp

What is the datatype you are trying to insert into? Yes, having hard defined lengths on the parameters passed to SQL will control the cache. Meaning, with nvarchar(4000) or nvarchar(max) when > 4000 2 plans will be created. The other way of a dynamic length param could pollute your plan cache (which is much smaller than your data cache) with thousands of plans for the same type of operations ie(nvarchar(4000), nvarchar(4001)....). Not sure if you care or not, but remember that when you insert over 8060 and trigger a row-overflow that you can get some interesting performance penalties with the page pointers.

@codeimpossible

First, thanks for the reply - i don't know a lot about sql query plans (caching or otherwise) and this really helping me understand whats going on.

Second, I'm trying to insert a lot of text (blog posts, error logs) into an nvarchar(max) column. Is this something that Massive just isn't supposed to handle?

@datachomp

I suppose it depends on your perspective if it is supposed to handle it or not. If you using it for SQL Server CE.... ha then it is very much not meant to handle it. Since you are not on SQL Server CE, have a nvarchar(max) datatype in your DB then I completely think it is in your best interest to modify it to nvarchar(max) when > nvarchar(4000). I would say that your situation is likely more the exception than the rule... in which case you can make a deliberate and surgical change to your version to fit your scenario. Then give yourself an open source high five.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 14, 2011
This page is out of date. Refresh to see the latest.
Showing with 1 addition and 1 deletion.
  1. +1 −1 Massive.cs
View
2 Massive.cs
@@ -41,7 +41,7 @@ public static class ObjectExtensions {
}
//from DataChomp
if (item.GetType() == typeof(string))
- p.Size = 4000;
+ p.Size = 4000 > ((string)item).Length ? 4000 : ((string)item).Length;
}
cmd.Parameters.Add(p);
}
Something went wrong with that request. Please try again.