Skip to content

Conversation

@Baunsgaard
Copy link
Contributor

Frame Cbind is implemented naively via appending rows one at a time and executing single threaded.

Current execution of a 64kx2k matrix of:

m = read($1)
for(i in 1:$2){
    m2 = rbind(m,m)
}
print(toString(m2[1:10]))
SystemDS Statistics:
Total elapsed time:		50.224 sec.
Total compilation time:		0.385 sec.
Total execution time:		49.840 sec.
Cache hits (Mem/Li/WB/FS/HDFS):	11/0/0/0/1.
Cache writes (Li/WB/FS/HDFS):	0/6/0/0.
Cache times (ACQr/m, RLS, EXP):	0.122/0.002/0.002/0.000 sec.
HOP DAGs recompiled (PRED, SB):	0/0.
HOP DAGs recompile time:	0.000 sec.
Total JIT compile time:		0.459 sec.
Total JVM GC count:		0.
Total JVM GC time:		0.0 sec.
Heavy hitter instructions:
 #  Instruction  Time(s)  Count
 1  append        49.813      5
 2  toString       0.012      1

There are optimizations available in parallelization and appending a block at a time, both going to be implemented in this PR.

@Baunsgaard
Copy link
Contributor Author

Baunsgaard commented Jan 5, 2023

After:

SystemDS Statistics:
Total elapsed time:		0.489 sec.
Total compilation time:		0.255 sec.
Total execution time:		0.234 sec.
Cache hits (Mem/Li/WB/FS/HDFS):	11/0/0/0/1.
Cache writes (Li/WB/FS/HDFS):	0/6/0/0.
Cache times (ACQr/m, RLS, EXP):	0.095/0.002/0.002/0.000 sec.
HOP DAGs recompiled (PRED, SB):	0/0.
HOP DAGs recompile time:	0.000 sec.
Total JIT compile time:		0.502 sec.
Total JVM GC count:		0.
Total JVM GC time:		0.0 sec.
Heavy hitter instructions:
 #  Instruction  Time(s)  Count
 1  append         0.211      5
 2  createvar      0.010      7
 3  rightIndex     0.005      1
 4  toString       0.005      1
 5  assignvar      0.001      3

a 236x speedup !

@Baunsgaard Baunsgaard marked this pull request as ready for review January 5, 2023 13:53
@Baunsgaard Baunsgaard changed the title [??] Frame Append cBind optimizations [SYSTEMDS-3484] Frame Append optimizations Jan 5, 2023
@Baunsgaard Baunsgaard changed the title [SYSTEMDS-3484] Frame Append optimizations [SYSTEMDS-3484] Frame Append Optimizations Jan 5, 2023
@Baunsgaard Baunsgaard closed this in 3f0a6f8 Jan 5, 2023
@Baunsgaard Baunsgaard deleted the FrameAppend branch January 24, 2023 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant