gtools-0.6.17 (2017-09-17); fixes #15

Bug fixes * Fixes #15 which was introduced trying to fix #15
mcaceresb · Sep 18, 2017 · cb76dc2 · cb76dc2
1 parent 8477936
commit cb76dc2
Show file tree

Hide file tree

Showing 25 changed files with 180 additions and 130 deletions.
diff --git a/README.md b/README.md
@@ -11,7 +11,7 @@ _Gtools_ is a Stata package that provides a fast implementation of
 common group commands like collapse and egen using C plugins for a
 massive speed improvement.
 
-`version 0.6.16 13Sep2017`
+`version 0.6.17 17Sep2017`
 Builds: Linux [![Travis Build Status](https://travis-ci.org/mcaceresb/stata-gtools.svg?branch=develop)](https://travis-ci.org/mcaceresb/stata-gtools),
 Windows (Cygwin) [![Appveyor Build status](https://ci.appveyor.com/api/projects/status/2bh1q9bulx3pl81p/branch/develop?svg=true)](https://ci.appveyor.com/project/mcaceresb/stata-gtools)
 

diff --git a/build/changelog.md b/build/changelog.md
@@ -1,6 +1,16 @@
 Change Log
 ==========
 
+## gtools-0.6.16 (2017-09-13); addresses #7
+
+### Bug fixes
+
+* Improves the issues raised by https://github.com/mcaceresb/stata-gtools/issues/7
+  Now the commands only fail if Stata hits the `matsize` limit (internally, the plugin
+  no longer uses the `subinstr` hack to go from locals to mata string matrices and uses
+  `tokens` instead, which is more appropriate; further, the plugin tries to set matsize
+  to at least the number of variables and gives a verbose error if it fails.)
+
 ## gtools-0.6.15 (2017-09-12); fixes #14, #9
 
 ### Bug fixes

diff --git a/build/gcollapse.ado b/build/gcollapse.ado
@@ -1,4 +1,4 @@
-*! version 0.6.16 13Sep2017 Mauricio Caceres Bravo, mauricio.caceres.bravo@gmail.com
+*! version 0.6.17 17Sep2017 Mauricio Caceres Bravo, mauricio.caceres.bravo@gmail.com
 *! -collapse- implementation using C for faster processing
 
 capture program drop gcollapse
@@ -1457,26 +1457,35 @@ if ( "`c(os)'" == "Windows" ) {
     }
 }
 
-* The legacy versions segfault if they are not loaded First
-cap program drop __gtools_plugin
-cap program __gtools_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_legacy.plugin"')
+* The legacy versions segfault if they are not loaded first (Unix only)
+if ( `"`:di lower("`c(os)'")'"' == "unix" ) {
+    cap program drop __gtools_plugin
+    cap program __gtools_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_legacy.plugin"')
 
-cap program drop __gtoolsmulti_plugin
-cap program __gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi_legacy.plugin"')
-
-* But we only want to use them when multi-threading fails normally
-cap program drop gtoolsmulti_plugin
-cap program gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi.plugin"')
-if ( _rc ) {
-    cap program drop gtools_plugin
-    program gtools_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_legacy.plugin"')
+    cap program drop __gtoolsmulti_plugin
+    cap program __gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi_legacy.plugin"')
 
+    * But we only want to use them when multi-threading fails normally
     cap program drop gtoolsmulti_plugin
-    cap program gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi_legacy.plugin"')
+    cap program gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi.plugin"')
+    if ( _rc ) {
+        cap program drop gtools_plugin
+        program gtools_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_legacy.plugin"')
+
+        cap program drop gtoolsmulti_plugin
+        cap program gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi_legacy.plugin"')
+    }
+    else {
+        cap program drop gtools_plugin
+        program gtools_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'.plugin"')
+    }
 }
 else {
     cap program drop gtools_plugin
     program gtools_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'.plugin"')
+
+    cap program drop gtoolsmulti_plugin
+    cap program gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi.plugin"')
 }
 
 * This is very inelegant, but I have debugging fatigue, and this seems to work.

diff --git a/build/gcollapse.sthlp b/build/gcollapse.sthlp
@@ -1,5 +1,5 @@
 {smcl}
-{* *! version 0.6.16 13Sep2017}{...}
+{* *! version 0.6.17 17Sep2017}{...}
 {viewerdialog gcollapse "dialog gcollapse"}{...}
 {vieweralsosee "[R] gcollapse" "mansection R gcollapse"}{...}
 {viewerjumpto "Syntax" "gcollapse##syntax"}{...}

diff --git a/build/gegen.ado b/build/gegen.ado
@@ -1,4 +1,4 @@
-*! version 0.6.16 13Sep2017 Mauricio Caceres Bravo, mauricio.caceres.bravo@gmail.com
+*! version 0.6.17 17Sep2017 Mauricio Caceres Bravo, mauricio.caceres.bravo@gmail.com
 *! implementation of by-able -egen- functions using C for faster processing
 
 /*
@@ -644,29 +644,36 @@ if ( "`c(os)'" == "Windows" ) {
     }
 }
 
-* The legacy versions segfault if they are not loaded First
-cap program drop __gtools_plugin
-cap program __gtools_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_legacy.plugin"')
+* The legacy versions segfault if they are not loaded first (Unix only)
+if ( `"`:di lower("`c(os)'")'"' == "unix" ) {
+    cap program drop __gtools_plugin
+    cap program __gtools_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_legacy.plugin"')
 
-cap program drop __gtoolsmulti_plugin
-cap program __gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi_legacy.plugin"')
-
-* But we only want to use them when multi-threading fails normally
-cap program drop gtoolsmulti_plugin
-cap program gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi.plugin"')
-if ( _rc ) {
-    cap program drop gtools_plugin
-    program gtools_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_legacy.plugin"')
+    cap program drop __gtoolsmulti_plugin
+    cap program __gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi_legacy.plugin"')
 
+    * But we only want to use them when multi-threading fails normally
     cap program drop gtoolsmulti_plugin
-    cap program gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi_legacy.plugin"')
+    cap program gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi.plugin"')
+    if ( _rc ) {
+        cap program drop gtools_plugin
+        program gtools_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_legacy.plugin"')
+
+        cap program drop gtoolsmulti_plugin
+        cap program gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi_legacy.plugin"')
+    }
+    else {
+        cap program drop gtools_plugin
+        program gtools_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'.plugin"')
+    }
 }
 else {
     cap program drop gtools_plugin
     program gtools_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'.plugin"')
-}
 
-* This is very inelegant, but I have debugging fatigue, and this seems to work.
+    cap program drop gtoolsmulti_plugin
+    cap program gtoolsmulti_plugin, plugin using(`"gtools_`:di lower("`c(os)'")'_multi.plugin"')
+}
 
 * This is very inelegant, but I have debugging fatigue, and this seems to work.
 

diff --git a/build/gegen.sthlp b/build/gegen.sthlp
@@ -1,5 +1,5 @@
 {smcl}
-{* *! version 0.6.16 13Sep2017}{...}
+{* *! version 0.6.17 17Sep2017}{...}
 {viewerdialog gegen "dialog gegen"}{...}
 {vieweralsosee "[R] gegen" "mansection R gegen"}{...}
 {viewerjumpto "Syntax" "gegen##syntax"}{...}

diff --git a/build/gtools.ado b/build/gtools.ado
@@ -1,4 +1,4 @@
-*! version 0.6.16 13Sep2017 Mauricio Caceres Bravo, mauricio.caceres.bravo@gmail.com
+*! version 0.6.17 17Sep2017 Mauricio Caceres Bravo, mauricio.caceres.bravo@gmail.com
 *! Program for managing the gtools package installation
 
 capture program drop gtools

diff --git a/build/gtools.pkg b/build/gtools.pkg
@@ -1,4 +1,4 @@
-v 0.6.16
+v 0.6.17
 d
 d 'GTOOLS': Faster -egen- and -collapse- using C plugins
 d
@@ -17,7 +17,7 @@ d
 d Author: Mauricio Caceres Bravo
 d Support: email mauricio.caceres.bravo@gmail.com
 d
-d Distribution-Date: 20170913
+d Distribution-Date: 20170917
 d
 f gcollapse.ado
 f gcollapse.sthlp

diff --git a/build/gtools_tests.do b/build/gtools_tests.do
@@ -5,7 +5,7 @@
 * Created: Tue May 16 07:23:02 EDT 2017
 * Updated: Thu Jul 27 10:49:56 EDT 2017
 * Purpose: Unit tests for gtools
-* Version: 0.6.16
+* Version: 0.6.17
 * Manual:  help gcollapse, help gegen
 
 * Stata start-up options
@@ -606,9 +606,9 @@ program checks_corners
         * Only fails in Stata/IC
         * gen x801 = 10
         * preserve
-        *     gcollapse zz, by(x*) `options'
+        *     collapse zz, by(x*) `options'
         * restore, preserve
-        *     gcollapse x*, by(zz) `options'
+        *     collapse x*, by(zz) `options'
         * restore
     }
 

diff --git a/build/gtools_tests.log b/build/gtools_tests.log
@@ -665,9 +665,9 @@ Note:  Your site can add messages to the introduction by editing the file
 .         * Only fails in Stata/IC
 .         * gen x801 = 10
 .         * preserve
-.         *     gcollapse zz, by(x*) `options'
+.         *     collapse zz, by(x*) `options'
 .         * restore, preserve
-.         *     gcollapse x*, by(zz) `options'
+.         *     collapse x*, by(zz) `options'
 .         * restore
 .     }
  49. 
@@ -1544,10 +1544,10 @@ Note:  Your site can add messages to the introduction by editing the file
 . * Run the things
 . 
 . main, checks test
-Start: 11:15:48 13 Sep 2017
+Start: 21:31:14 17 Sep 2017
 
 -------------------------------------
-Basic unit-tests 11:15:48 13 Sep 2017
+Basic unit-tests 21:31:14 17 Sep 2017
 -------------------------------------
     test(passed): checks_corners, oncollision(error)
     test(passed): checks_byvars_gcollapse, oncollision(error) debug_force_single
@@ -1560,7 +1560,7 @@ Basic unit-tests 11:15:48 13 Sep 2017
     test(passed): checks_options_gcollapse, oncollision(error) debug_force_multi
 
 -----------------------------------------------------------
-Consistency checks (vs collapse, egen) 11:16:37 13 Sep 2017
+Consistency checks (vs collapse, egen) 21:32:18 17 Sep 2017
 -----------------------------------------------------------
 
 --------------------------------------------------------------------------------
@@ -1991,111 +1991,111 @@ Short (quick) versions of the benchmarks
 Benchmarking N for J = 100; by(x3)
     vars  = y1 y2 y3 y4 y5 y6 y7 y8 y9 y10 y11 y12 y13 y14 y15
     stats = sum
-                    2,000 gcollapse  (.023)  collapse  (.016)  fcollapse  (.027)
-                   20,000 gcollapse  (.031)  collapse  (.094)  fcollapse  (.028)
+                    2,000 gcollapse  (.08)  collapse  (.015)  fcollapse  (.043)
+                   20,000 gcollapse  (.128)  collapse  (.127)  fcollapse  (.029)
 Results varying N for J = 100; by(x3)
 |              N | gcollapse |  collapse | fcollapse | ratio (f/g) | ratio (c/g) |
 | -------------- | --------- | --------- | --------- | ----------- | ----------- |
-|          2,000 |      0.02 |      0.02 |      0.03 |        1.17 |        0.70 |
-|         20,000 |      0.03 |      0.09 |      0.03 |        0.90 |        3.03 |
+|          2,000 |      0.08 |      0.01 |      0.04 |        0.54 |        0.19 |
+|         20,000 |      0.13 |      0.13 |      0.03 |        0.23 |        0.99 |
 Benchmarking N for J = 100; by(x3)
     vars  = y1 y2 y3
     stats = mean median
-                    2,000 gcollapse  (.01)  collapse  (.015)  fcollapse  (.007)
-                   20,000 gcollapse  (.017)  collapse  (.138)  fcollapse  (.03)
+                    2,000 gcollapse  (.033)  collapse  (.02)  fcollapse  (.012)
+                   20,000 gcollapse  (.101)  collapse  (.159)  fcollapse  (.038)
 Results varying N for J = 100; by(x3)
 |              N | gcollapse |  collapse | fcollapse | ratio (f/g) | ratio (c/g) |
 | -------------- | --------- | --------- | --------- | ----------- | ----------- |
-|          2,000 |      0.01 |      0.01 |      0.01 |        0.70 |        1.50 |
-|         20,000 |      0.02 |      0.14 |      0.03 |        1.76 |        8.12 |
+|          2,000 |      0.03 |      0.02 |      0.01 |        0.36 |        0.61 |
+|         20,000 |      0.10 |      0.16 |      0.04 |        0.38 |        1.57 |
 Benchmarking N for J = 100; by(x3)
     vars  = y1 y2 y3 y4 y5 y6
     stats = sum mean count min max
-                    2,000 gcollapse  (.033)  collapse  (.037)  fcollapse  (.014)
-                   20,000 gcollapse  (.04)  collapse  (.383)  fcollapse  (.034)
+                    2,000 gcollapse  (.046)  collapse  (.037)  fcollapse  (.013)
+                   20,000 gcollapse  (.119)  collapse  (.391)  fcollapse  (.035)
 Results varying N for J = 100; by(x3)
 |              N | gcollapse |  collapse | fcollapse | ratio (f/g) | ratio (c/g) |
 | -------------- | --------- | --------- | --------- | ----------- | ----------- |
-|          2,000 |      0.03 |      0.04 |      0.01 |        0.42 |        1.12 |
-|         20,000 |      0.04 |      0.38 |      0.03 |        0.85 |        9.57 |
+|          2,000 |      0.05 |      0.04 |      0.01 |        0.28 |        0.80 |
+|         20,000 |      0.12 |      0.39 |      0.04 |        0.29 |        3.29 |
 Benchmarking N for J = 10; by(group)
     vars  = x1 x2
     stats = sum mean max min count percent first last firstnm lastnm median iqr p23 p77
-                    2,000 gcollapse  (.031)  collapse  (.078)  fcollapse  (.015)
-                   20,000 gcollapse  (.038)  collapse  (1.238)  fcollapse  (.064)
+                    2,000 gcollapse  (.101)  collapse  (.083)  fcollapse  (.015)
+                   20,000 gcollapse  (.076)  collapse  (1.696)  fcollapse  (.074)
 Results varying N for J = 10; by(group)
 |              N | gcollapse |  collapse | fcollapse | ratio (f/g) | ratio (c/g) |
 | -------------- | --------- | --------- | --------- | ----------- | ----------- |
-|          2,000 |      0.03 |      0.08 |      0.01 |        0.48 |        2.52 |
-|         20,000 |      0.04 |      1.24 |      0.06 |        1.68 |       32.58 |
+|          2,000 |      0.10 |      0.08 |      0.01 |        0.15 |        0.82 |
+|         20,000 |      0.08 |      1.70 |      0.07 |        0.97 |       22.32 |
 Benchmarking J for N = 5,000; by(group)
     vars  = x1 x2
     stats = sum mean max min count percent first last firstnm lastnm median iqr p23 p77
-                      100 gcollapse  (.031)  collapse  (.206)  fcollapse  (.023)
-                    1,000 gcollapse  (.032)  collapse  (.282)  fcollapse  (.054)
+                      100 gcollapse  (.074)  collapse  (.262)  fcollapse  (.025)
+                    1,000 gcollapse  (.099)  collapse  (.385)  fcollapse  (.055)
 Results varying J for N = 5,000; by(group)
 |              J | gcollapse |  collapse | fcollapse | ratio (f/g) | ratio (c/g) |
 | -------------- | --------- | --------- | --------- | ----------- | ----------- |
-|            100 |      0.03 |      0.21 |      0.02 |        0.74 |        6.65 |
-|          1,000 |      0.03 |      0.28 |      0.05 |        1.69 |        8.81 |
+|            100 |      0.07 |      0.26 |      0.03 |        0.34 |        3.54 |
+|          1,000 |      0.10 |      0.39 |      0.06 |        0.56 |        3.89 |
 Benchmarking N for J = 100; by(x3)
     vars  = y1 y2 y3 y4 y5 y6 y7 y8 y9 y10 y11 y12 y13 y14 y15
     stats = sum
-                    2,000 gcollapse-default  (.027)  fcollapse  (.01) 
-                   20,000 gcollapse-default  (.029)  fcollapse  (.03) 
+                    2,000 gcollapse-default  (.102)  fcollapse  (.009) 
+                   20,000 gcollapse-default  (.052)  fcollapse  (.03) 
 Results varying N for J = 100; by(x3)
 |              N | gcollapse | fcollapse | ratio (f/g) |
 | -------------- | --------- | --------- | ----------- |
-|          2,000 |      0.03 |      0.01 |        0.37 |
-|         20,000 |      0.03 |      0.03 |        1.03 |
+|          2,000 |      0.10 |      0.01 |        0.09 |
+|         20,000 |      0.05 |      0.03 |        0.58 |
 Benchmarking N for J = 100; by(x3)
     vars  = y1 y2 y3
     stats = mean median
-                    2,000 gcollapse-default  (.01)  fcollapse  (.008) 
-                   20,000 gcollapse-default  (.014)  fcollapse  (.028) 
+                    2,000 gcollapse-default  (.099)  fcollapse  (.016) 
+                   20,000 gcollapse-default  (.084)  fcollapse  (.034) 
 Results varying N for J = 100; by(x3)
 |              N | gcollapse | fcollapse | ratio (f/g) |
 | -------------- | --------- | --------- | ----------- |
-|          2,000 |      0.01 |      0.01 |        0.80 |
-|         20,000 |      0.01 |      0.03 |        2.00 |
+|          2,000 |      0.10 |      0.02 |        0.16 |
+|         20,000 |      0.08 |      0.03 |        0.40 |
 Benchmarking N for J = 100; by(x3)
     vars  = y1 y2 y3 y4 y5 y6
     stats = sum mean count min max
-                    2,000 gcollapse-default  (.034)  fcollapse  (.015) 
-                   20,000 gcollapse-default  (.039)  fcollapse  (.033) 
+                    2,000 gcollapse-default  (.048)  fcollapse  (.023) 
+                   20,000 gcollapse-default  (.112)  fcollapse  (.044) 
 Results varying N for J = 100; by(x3)
 |              N | gcollapse | fcollapse | ratio (f/g) |
 | -------------- | --------- | --------- | ----------- |
-|          2,000 |      0.03 |      0.01 |        0.44 |
-|         20,000 |      0.04 |      0.03 |        0.85 |
+|          2,000 |      0.05 |      0.02 |        0.48 |
+|         20,000 |      0.11 |      0.04 |        0.39 |
 Benchmarking N for J = 10; by(group)
     vars  = x1 x2
     stats = sum mean max min count percent first last firstnm lastnm median iqr p23 p77
-                    2,000 gcollapse-default  (.029)  fcollapse  (.017) 
-                   20,000 gcollapse-default  (.037)  fcollapse  (.062) 
+                    2,000 gcollapse-default  (.105)  fcollapse  (.024) 
+                   20,000 gcollapse-default  (.112)  fcollapse  (.074) 
 Results varying N for J = 10; by(group)
 |              N | gcollapse | fcollapse | ratio (f/g) |
 | -------------- | --------- | --------- | ----------- |
-|          2,000 |      0.03 |      0.02 |        0.59 |
-|         20,000 |      0.04 |      0.06 |        1.68 |
+|          2,000 |      0.10 |      0.02 |        0.23 |
+|         20,000 |      0.11 |      0.07 |        0.66 |
 Benchmarking J for N = 5,000; by(group)
     vars  = x1 x2
     stats = sum mean max min count percent first last firstnm lastnm median iqr p23 p77
-                      100 gcollapse-default  (.031)  fcollapse  (.027) 
-                    1,000 gcollapse-default  (.094)  fcollapse  (.056) 
+                      100 gcollapse-default  (.13)  fcollapse  (.03) 
+                    1,000 gcollapse-default  (.086)  fcollapse  (.0610000000000001) 
 Results varying J for N = 5,000; by(group)
 |              J | gcollapse | fcollapse | ratio (f/g) |
 | -------------- | --------- | --------- | ----------- |
-|            100 |      0.03 |      0.03 |        0.87 |
-|          1,000 |      0.09 |      0.06 |        0.60 |
+|            100 |      0.13 |      0.03 |        0.23 |
+|          1,000 |      0.09 |      0.06 |        0.71 |
 
-End: 11:26:11 13 Sep 2017
+End: 21:44:55 17 Sep 2017
 tests ran
 
 tests finished running
 
-Start: 11:15:48 13 Sep 2017
-End: 11:26:11 13 Sep 2017
+Start: 21:31:14 17 Sep 2017
+End: 21:44:55 17 Sep 2017
 
 . 
 end of do-file
diff --git a/build/stata.toc b/build/stata.toc
@@ -1,3 +1,3 @@
-v 0.6.16
+v 0.6.17
 d Mauricio Caceres Bravo, mauricio.caceres.bravo@gmail.com
 p 'GTOOLS': Faster -egen- and -collapse- using C plugins