compression bug fix#46
Conversation
|
|
||
| // set .z.zd to control how data gets compressed | ||
| setcompression:{[compression] if[ 3= count compression; | ||
| .lg.o[`compression;"setting compression level to (",(";" sv string compression),")"]; |
There was a problem hiding this comment.
Can you fix up the indentation here. Make it consistent with the rest of the file
| removecompression:{[compression] if[ 3= count compression; | ||
| .z.zd:16 0 0; | ||
| .lg.o[`compression;"resetting compression level to (",(";" sv string 16 0 0),")"]]} | ||
|
|
There was a problem hiding this comment.
Could you make "set compression" and removecompression a single function? (or remove could be a projection of set). So basically if count=3, then set it. if count = 0 (empty list), then unset it
| [.lg.o[`sort;"sorting on slave sort", string .z.p]; | ||
| {[x] .sort.sorttab[x];if[gc;.gc.run[]]} peach tablist,'.Q.par[dir;pt;] each tablist; | ||
| {[x] setcompression[compression];.sort.sorttab[x];removecompression[compression];if[gc;.gc.run[]]} peach tablist,'.Q.par[dir;pt;] each tablist; | ||
| ]; |
There was a problem hiding this comment.
This isn't going to work. The sorting here is done on a remote process, which might have different compression parameters defined. You should pass the compression settings to the remote process (as an extra parameter to the function)
There was a problem hiding this comment.
Oh ok, I had been setting values for "compression" in the sort/sortslaves config files. Instead, should the "compression" value set in the wdb config file get used in all 3 cases ?
There was a problem hiding this comment.
Yes - it should just be set in one place and it should drive the other processes.
There was a problem hiding this comment.
I've fixed that problem now, but I've had to add a compression parameter into a few functions (informsortandreload, endofdaysort, endofdaymerge, endofdaysortdate) so that it feeds through to the sort/sortslave processes. Is this ok or is there a better way to do it ?
| merge[dir;pt;;mergelimits] peach flip (key tablist;value tablist);]; | ||
| removecompression[compression]; | ||
| [.lg.o[`merge;"merging on master"]; | ||
| merge[dir;pt;;mergelimits] each flip (key tablist;value tablist)]]; |
There was a problem hiding this comment.
I think you need to wrap this block in set/remove statements? This is the bit that does the merging on the master process
| [.lg.o[`merge;"merging on slave"]; | ||
| setcompression[compression]; | ||
| merge[dir;pt;;mergelimits] peach flip (key tablist;value tablist);]; | ||
| removecompression[compression]; |
There was a problem hiding this comment.
I don't think this will work. setcompression will be executed locally, whereas merge will be executed remotely. You need to wrap the merge statement in a lambda and set/remove compression either side of the merge function call (the same as you do with the other peach statement)
…pression/setcommpression into 1 function
|
|
||
| endofdaysortdate:{[dir;pt;tablist] | ||
| // set .z.zd to control how data gets compressed | ||
| setcompression:{[compression] if[3=count compression;.lg.o[`compression;$[compression~16 0 0;"resetting";"setting"]," compression level to (",(";" sv string compression),")"];.z.zd:compression]} |
There was a problem hiding this comment.
could you split this across lines?
you could also add a function of
resetcompression:{setcompression 16 0 0}
which might be nice
…/merger * aquaq/master: compression bug fix (DataIntellectTech#46) Datareplay util (DataIntellectTech#48)
* tag 'v3.1.0': Version num (DataIntellectTech#52) Update version.txt (DataIntellectTech#51) Add kafka to docs (DataIntellectTech#50) kafka for l64 (DataIntellectTech#49) compression bug fix (DataIntellectTech#46) Datareplay util (DataIntellectTech#48)
2 participants
No description provided.