Append has become quite slow

[Karl1155]

When i run the test it takes seconds ! to append 100 records. VO appends 100 or 1000 records in less than 0,1 seconds.

FUNCTION Start( ) AS VOID
LOCAL i AS DWORD 
LOCAL aFields AS ARRAY
LOCAL cDbf, cPfad AS STRING  
LOCAL f AS FLOAT 
                     
    RddSetDefault( "dbfcdx" )
    

	cPfad := "D:\TEST\"	

	cDBF := cPfad + "Foo.dbf"


	aFields := { { "LAST" , "C" , 10 , 0 } } 
    
//	SetExclusive ( TRUE )  
	// X# 2,94 secs  append + fieldput
	//    2,49 secs  append only	
	      
	SetExclusive ( FALSE )   
	// X# 5,54 secs  append + fieldput
	//    2,50 secs  append only	
	
	
	? DbCreate( cDBF , AFields) 
	? DbUseArea( ,,cDBF )	

	f := Seconds()
		
	FOR i := 1 UPTO 100  
		DbAppend() 
		FieldPut ( 1 , PadL ( i , 10 , "0" ) )
						
	NEXT 

	? "Elapsed:" , Seconds() - f
	
	DbCloseArea()
	
RETURN 

Karl-Heinz

[cpyrgas]

Hi Karl-Heinz,

Could it be you are including also the time needed to load and initialize the rdd, macro compiler etc? I tested your code in my very slow 10 year old laptop and it only takes 0.25 secs to complete the appends

[wriedmann]

Hi Chris, hi Karl-Heinz,
on my machine it takes 1.06 seconds for the first run, and when I execute the test a second time in the same application it takes 1.09 seconds.
Using Exclusive := true the time is down to 0.56 seconds.
Wolfgang

[Karl1155]

Guys,

tried it on two Win10 machines and the append speed is acceptable. So it seems to be a problem with win8.1 64-Bit. Karl (Faller) mentioned the same speed issues and If i remember correctly he uses Win8.1 too. I´ll ask him if he still faces the same problems.

Karl-Heinz

[Karl1155]

Guys,

i installed the earlier build 2.1.0.0 (Bandol GA 2.1) - XSharpSetup210aFox.exe. This build appends 1000 records in less than 0,1 seconds, no matter how the setexclusive setting is. It seems that the RDD changes introduced in the build 2.2.0.0 (Bandol GA 2.2) are responsible for my Win8.1 append problems.

Fixed locking problems in the DBFCDX RDD that were causing problems when opening files shared between multiple apps but also between multiple threads. The RDD now should properly detect that the CDX was updated by another process or thread.

But that doesn't really explain why there are now such huge time differences between Win10 and Win8.1 ?

Karl-Heinz

[RobertvanderHulst]

Karl-Heinz,
I see the speed difference too on a test machine. I will look into this, but any changes that I will have to make will not be included in this weeks build.
Robert

[Karl1155]

Thanks Robert,
no problem to wait, important is that the problem is now known.
Karl-Heinz

[cpyrgas]

Indeed there seems to be an important general problem now. The following takes only a couple seconds to finish in my PC, while in X# it takes several minutes!

Note that this is only pure dbf, no indexes involved at all. Could it just be the speed of the FileStream class and need to move to WIn32 API for file write, as we had done in vulcan as well?

FUNCTION Start() AS INT
LOCAL aStruct AS ARRAY
LOCAL cDbf AS STRING
LOCAL i AS INT

RddSetDefault("DBFCDX")

cDbf := "c:\temp\testdbf"
FErase(cDbf + ".dbf")
FErase(cDbf + ".cdx")
FErase(cDbf + ".fpt")

aStruct := {{"NFIELD" , "N" , 10 , 0}}
FOR i := 1 UPTO 50
	AAdd(aStruct , {"CFIELD" + AsString(i) , "C" , 30 , 0})
NEXT
//AAdd(aStruct , {"MFIELD" , "M" , 10 , 0})

DbCreate(cDbf, aStruct)

DbUseArea(TRUE,"DBFCDX",cDbf,"alias1",TRUE,FALSE)

// does not really make a difference, but opening another instance just to make sure it is opened shared
DbUseArea(TRUE,"DBFCDX",cDbf,"alias2",TRUE,FALSE)
? alias1->Eof()
? alias2->Eof()

DbGoBottom()
? FieldGet(1), Eof()
wait "Press enter to start speed test"
FOR	i := 1 UPTO 100000
	DbAppend()
	FieldPut(1,i)
NEXT

? alias1->RecCount()
? alias2->RecCount()
DbCloseAll()
wait
RETURN 0

[RobertvanderHulst]

I'll look at this after releasing the current build.

[cpyrgas]

OK, so ignore my private message :)

[RobertvanderHulst]

Guys,
I have done some research to see what has changed, and I see that I have added a call to FFlush() to the method that writes the current record to disk.
It I remove that then the speed is faster.
This was changed in Dec 2019 to fix some locking issues. I will remove it and we'll have to see what the effect will be.
I checked the Vulcan DBF RDD and it has a similar flush in the __DbfWrite method.
However the call to FFlush() calls the FileStream:Flush() overload with TRUE parameter, which probably causes the delay.
I will change the code to call the parameter-less Flush() on the filestream to fix this.

[wriedmann]

Hi Robert,
if I remember correctly there was an issue appended records not being visible to other threads/processes, and then they overwrote them.
I can search the test code that Andreas used to reproduce the issue.

[RobertvanderHulst]

Wolfgang,
No need to send that example. I still have it and will test it after making the change.

Robert

[wriedmann]

Thank you very much, Robert!
Your work is invaluable to us.
Wolfgang