Not able to store data above 2GB! Help! #149

Closed
sherry-ummen opened this Issue Jun 30, 2015 · 17 comments

Comments

Projects
None yet
3 participants
@sherry-ummen

Hi,

Tried to stores some quite large data. and I reached to a point of 2Gb of file sixe after that the process just hung. When I stop it and start again then i get StackOverFlowexception from EJB.cs

internal static bool _ejdbopen(IntPtr db, string path, int mode) {
IntPtr pptr = Native.NativeUtf8FromString(path); //UnixMarshal.StringToHeap(path, Encoding.UTF8);
try {
return _ejdbopen(db, pptr, mode);
} finally {
Marshal.FreeHGlobal(pptr); //UnixMarshal.FreeHeap(pptr);
}
}

At this particular line: return _ejdbopen(db, pptr, mode);

@sherry-ummen sherry-ummen changed the title from Not able to store data above 2GB! to Very Urgent!! Not able to store data above 2GB! Jun 30, 2015

@akaustel

This comment has been minimized.

Show comment
Hide comment
@akaustel

akaustel Jul 1, 2015

What file system do you use? Could that be the limit? I'm fairly sure that I have been able to handle data sets larger than 2GB.

akaustel commented Jul 1, 2015

What file system do you use? Could that be the limit? I'm fairly sure that I have been able to handle data sets larger than 2GB.

@sherry-ummen

This comment has been minimized.

Show comment
Hide comment
@sherry-ummen

sherry-ummen Jul 1, 2015

Windows 8.1 NTFS

C# Driver

What I tried:

var stopwatch = new Stopwatch();
stopwatch.Start();
//int COLLECTIONNAME = 1;
foreach (var index in Enumerable.Range(1, 1000000000)) {
var customer = BSONDocument.ValueOf(new {
Id = "FOOBAR" + index,
name = "FOOBAR" + index,
date = DateTime.Now.ToLocalTime().Millisecond,
records = Enumerable.Range(1, 100000).ToArray(),
rep_modified = "M",
split = false,
type = 0,
user = "ADMI",
version = "A"
});
jb.Save("parrots", customer);

        }
        stopwatch.Stop();
        Console.WriteLine("Speed: " + stopwatch.ElapsedMilliseconds);

Sent from Windows Mail

From: akaustel
Sent: ‎Wednesday‎, ‎July‎ ‎1‎, ‎2015 ‎9‎:‎30‎ ‎PM
To: Softmotions/ejdb
Cc: Sherry Ummen

What file system do you use? Could that be the limit? I'm fairly sure that I have been able to handle data sets larger than 2GB.


Reply to this email directly or view it on GitHub.

Windows 8.1 NTFS

C# Driver

What I tried:

var stopwatch = new Stopwatch();
stopwatch.Start();
//int COLLECTIONNAME = 1;
foreach (var index in Enumerable.Range(1, 1000000000)) {
var customer = BSONDocument.ValueOf(new {
Id = "FOOBAR" + index,
name = "FOOBAR" + index,
date = DateTime.Now.ToLocalTime().Millisecond,
records = Enumerable.Range(1, 100000).ToArray(),
rep_modified = "M",
split = false,
type = 0,
user = "ADMI",
version = "A"
});
jb.Save("parrots", customer);

        }
        stopwatch.Stop();
        Console.WriteLine("Speed: " + stopwatch.ElapsedMilliseconds);

Sent from Windows Mail

From: akaustel
Sent: ‎Wednesday‎, ‎July‎ ‎1‎, ‎2015 ‎9‎:‎30‎ ‎PM
To: Softmotions/ejdb
Cc: Sherry Ummen

What file system do you use? Could that be the limit? I'm fairly sure that I have been able to handle data sets larger than 2GB.


Reply to this email directly or view it on GitHub.

@akaustel

This comment has been minimized.

Show comment
Hide comment
@akaustel

akaustel Jul 1, 2015

Have you set the large option?

bool large; /**< Large collection. It can be larger than 2GB. Default false */

I do not know how to set it in C#, but I guess you can figure that out.

akaustel commented Jul 1, 2015

Have you set the large option?

bool large; /**< Large collection. It can be larger than 2GB. Default false */

I do not know how to set it in C#, but I guess you can figure that out.

@sherry-ummen

This comment has been minimized.

Show comment
Hide comment
@sherry-ummen

sherry-ummen Jul 1, 2015

Yes, Did that

EJDBCollectionOptionsN co = new EJDBCollectionOptionsN();
co.large = true;
co.compressed = false;
jb.EnsureCollection("parrots", co);
var stopwatch = new Stopwatch();
stopwatch.Start();
foreach (var index in Enumerable.Range(1, 1000000000)) {
var customer = BSONDocument.ValueOf(new {
Id = "FOOBAR" + index,
name = "FOOBAR" + index,
date = DateTime.Now.ToLocalTime().Millisecond,
records = Enumerable.Range(1, 100000).ToArray(),
rep_modified = "M",
split = false,
type = 0,
user = "ADMI",
version = "A"
});
jb.Save("parrots", customer);
}
stopwatch.Stop();
Console.WriteLine("Speed: " + stopwatch.ElapsedMilliseconds);

But still after reavhing to the point of 2Gb the process just hangs.

Yes, Did that

EJDBCollectionOptionsN co = new EJDBCollectionOptionsN();
co.large = true;
co.compressed = false;
jb.EnsureCollection("parrots", co);
var stopwatch = new Stopwatch();
stopwatch.Start();
foreach (var index in Enumerable.Range(1, 1000000000)) {
var customer = BSONDocument.ValueOf(new {
Id = "FOOBAR" + index,
name = "FOOBAR" + index,
date = DateTime.Now.ToLocalTime().Millisecond,
records = Enumerable.Range(1, 100000).ToArray(),
rep_modified = "M",
split = false,
type = 0,
user = "ADMI",
version = "A"
});
jb.Save("parrots", customer);
}
stopwatch.Stop();
Console.WriteLine("Speed: " + stopwatch.ElapsedMilliseconds);

But still after reavhing to the point of 2Gb the process just hangs.

@akaustel

This comment has been minimized.

Show comment
Hide comment
@akaustel

akaustel Jul 2, 2015

Did you remove the database file or see to that the large option really was enabled. It might not be updated on an already existing db file.

akaustel commented Jul 2, 2015

Did you remove the database file or see to that the large option really was enabled. It might not be updated on an already existing db file.

@sherry-ummen

This comment has been minimized.

Show comment
Hide comment
@sherry-ummen

sherry-ummen Jul 2, 2015

Yes I always create a new database. How can I check from the file that it has that flag ON?

Thanks

Sent from my Windows Phone


From: akaustelmailto:notifications@github.com
Sent: ‎02-‎07-‎2015 10:42
To: Softmotions/ejdbmailto:ejdb@noreply.github.com
Cc: Sherry Ummenmailto:sherry.ummen@outlook.com
Subject: Re: [ejdb] Very Urgent!! Not able to store data above 2GB! (#149)

Did you remove the database file or see to that the large option really was enabled. It might not be updated on an already existing db file.


Reply to this email directly or view it on GitHub:
#149 (comment)

Yes I always create a new database. How can I check from the file that it has that flag ON?

Thanks

Sent from my Windows Phone


From: akaustelmailto:notifications@github.com
Sent: ‎02-‎07-‎2015 10:42
To: Softmotions/ejdbmailto:ejdb@noreply.github.com
Cc: Sherry Ummenmailto:sherry.ummen@outlook.com
Subject: Re: [ejdb] Very Urgent!! Not able to store data above 2GB! (#149)

Did you remove the database file or see to that the large option really was enabled. It might not be updated on an already existing db file.


Reply to this email directly or view it on GitHub:
#149 (comment)

@akaustel

This comment has been minimized.

Show comment
Hide comment
@akaustel

akaustel Jul 2, 2015

The method DBMeta seems familiar from the nodejs API. It returns a BSONDocument.

Found in https://github.com/Softmotions/ejdb-csharp/blob/master/Ejdb.DB/EJDB.cs:

public class EJDB : IDisposable {
    ...
    public BSONDocument DBMeta
    ...

akaustel commented Jul 2, 2015

The method DBMeta seems familiar from the nodejs API. It returns a BSONDocument.

Found in https://github.com/Softmotions/ejdb-csharp/blob/master/Ejdb.DB/EJDB.cs:

public class EJDB : IDisposable {
    ...
    public BSONDocument DBMeta
    ...
@akaustel

This comment has been minimized.

Show comment
Hide comment
@akaustel

akaustel Jul 2, 2015

Please consider rephrasing your title. Some might read it as arrogant and disregard your plea for help altogether.

akaustel commented Jul 2, 2015

Please consider rephrasing your title. Some might read it as arrogant and disregard your plea for help altogether.

@sherry-ummen sherry-ummen changed the title from Very Urgent!! Not able to store data above 2GB! to Not able to store data above 2GB! Help! Jul 2, 2015

@sherry-ummen

This comment has been minimized.

Show comment
Hide comment
@sherry-ummen

sherry-ummen Jul 2, 2015

I have changed the title as you suggested! Thanks

I have changed the title as you suggested! Thanks

@sherry-ummen

This comment has been minimized.

Show comment
Hide comment
@sherry-ummen

sherry-ummen Jul 2, 2015

now I tried to check the properties as you said and it is actually a 'large' file. Here is the snapshot
image

To show whats happenign I prepared to videos:

  1. https://www.youtube.com/watch?v=Y_98eRAwbgs&feature=youtu.be
    This one if you see its writting to the file and when it reaches to 2Gb it automatically becomes 1Gb may be some clearing EJDB does and then again after reaching 2GB it just hang their forever

  2. https://www.youtube.com/watch?v=tAe-y9VehjY&feature=youtu.be
    this shows that when you open the same hung database then we get Stackover flow exception.

now I tried to check the properties as you said and it is actually a 'large' file. Here is the snapshot
image

To show whats happenign I prepared to videos:

  1. https://www.youtube.com/watch?v=Y_98eRAwbgs&feature=youtu.be
    This one if you see its writting to the file and when it reaches to 2Gb it automatically becomes 1Gb may be some clearing EJDB does and then again after reaching 2GB it just hang their forever

  2. https://www.youtube.com/watch?v=tAe-y9VehjY&feature=youtu.be
    this shows that when you open the same hung database then we get Stackover flow exception.

@adamansky

This comment has been minimized.

Show comment
Hide comment
@adamansky

adamansky Jul 3, 2015

Member

Hello!! Thank you for that detailed report! I will investigate problem within few days, hope fix it ASAP

Member

adamansky commented Jul 3, 2015

Hello!! Thank you for that detailed report! I will investigate problem within few days, hope fix it ASAP

@sherry-ummen

This comment has been minimized.

Show comment
Hide comment
@sherry-ummen

sherry-ummen Jul 3, 2015

Thank you for your response. I had almost lost faith on this project :).

Btw it would be nice if you could comment on this also #150

Thanks

Thank you for your response. I had almost lost faith on this project :).

Btw it would be nice if you could comment on this also #150

Thanks

@adamansky

This comment has been minimized.

Show comment
Hide comment
@adamansky

adamansky Jul 8, 2015

Member

Ok, I'm suspecting this problem related to the maximal size of MapViewOfFile address space. 2GB on 32bit systems? (correct me if I'm wrong) See tchdb.c#L5689). TokyoCabinet's TCHDB windows port uses mmaping of whole database file. Looking for hooks to bypass it.

Member

adamansky commented Jul 8, 2015

Ok, I'm suspecting this problem related to the maximal size of MapViewOfFile address space. 2GB on 32bit systems? (correct me if I'm wrong) See tchdb.c#L5689). TokyoCabinet's TCHDB windows port uses mmaping of whole database file. Looking for hooks to bypass it.

@sherry-ummen

This comment has been minimized.

Show comment
Hide comment
@sherry-ummen

sherry-ummen Jul 10, 2015

mine is a 64bit system

mine is a 64bit system

@sherry-ummen

This comment has been minimized.

Show comment
Hide comment
@sherry-ummen

sherry-ummen Jul 10, 2015

does you current change fix the problem? somewhow I can test this?

does you current change fix the problem? somewhow I can test this?

adamansky added a commit that referenced this issue Jul 10, 2015

adamansky added a commit that referenced this issue Jul 10, 2015

@adamansky

This comment has been minimized.

Show comment
Hide comment
@adamansky

adamansky Jul 10, 2015

Member

Fixed
v1.2.9 released http://ejdb.org/doc/install/windows.html
Please reopen issue if you have troubles.

Member

adamansky commented Jul 10, 2015

Fixed
v1.2.9 released http://ejdb.org/doc/install/windows.html
Please reopen issue if you have troubles.

@sherry-ummen

This comment has been minimized.

Show comment
Hide comment
@sherry-ummen

sherry-ummen Jul 11, 2015

NIce! Thanks its works. I just tested by insertting 10Gb of data

NIce! Thanks its works. I just tested by insertting 10Gb of data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment