Using gegen and reg within program returns error 198

[ppaspp]

To parallelize bootstraps with the ado parallel I have included gegen and regress in a small program which returns the invalid syntax error 198.
When replacing gegen with egen the programm runs fine.

clear all
sysuse auto , clear

cap prog drop pargegen
prog define pargegen, rclass
	version 13
	syntax varlist [if]
	marksample touse
	gegen test = sum(price)
	reg `varlist' if `touse'
	drop test
end

pargegen price weight foreign rep78

Setting trace on I get the following error log:
gtools_log.txt

My suspicion is that the local level, which is defined by regress, is somehow altered by gegen. According to the stata manual level(95) is the default option but in line 3883 of the log file 95,0 shows up. Even when I define the regress option , level(95) explicitly, the same error is shown.

[mcaceresb]

I don't get this issue. Can you run

clear all
sysuse auto , clear

cap prog drop pargegen
prog define pargegen, rclass
	version 13
	syntax varlist [if]
	marksample touse
        disp "`level', $_level"
	gegen test = sum(price)
        disp "`level', $_level"
	reg `varlist' if `touse'
	drop test
end

pargegen price weight foreign rep78

I don't think gtools ever sets a local called "level" but I could be wrong.

[ppaspp]

disp gives no output, just the usual error message.

I don't know if this is helpful, but I noted the strange behavior that after a run with gegen, running the test program with egen again returned the same error message. Only after restarting Stata the egen version runs fine again.
Even more strange, on Win 10 the program runs fine ?!

[mcaceresb]

As I noted, I couldn't reproduce the issue, which is why I was trying to figure out if level really was defined by gegen. Ah, ok, I think this, perhaps:

clear all
log using test, text
sysuse auto , clear

cap prog drop pargegen
prog define pargegen, rclass
	version 13
	syntax varlist [if]
	marksample touse
        disp "`level', $_level"
	cap noi gegen test = sum(price)
        disp "`level', $_level"
        exit `=_rc'
	reg `varlist' if `touse'
	drop test
end

pargegen price weight foreign rep78
log close _all

And send the log file? You say you are using parallel, right? Is the first snippet you posted the exact code you were running?

[mcaceresb]

What is the OS where it gave the error? OSX? Win7?

[ppaspp]

running on arch linux. This is the log file:
test.log

[mcaceresb]

I'm on arch as well, so I'm not sure what's failing. gegen definitely doesn't do anything to level, which is what I thought, but that was your original error...

Can you run

version
which gegen
which gtools

?

[ppaspp]

. version
version 15.1

. which gegen
/home/pp/ado/plus/g/gegen.ado
*! version 0.13.1 02May2018 Mauricio Caceres Bravo, mauricio.caceres.bravo@gmail.com
*! implementation -egen- using C for faster processing

. which gtools
/home/pp/ado/plus/g/gtools.ado
*! version 0.13.1 02May2018 Mauricio Caceres Bravo, mauricio.caceres.bravo@gmail.com
*! Program for managing the gtools package installation

The starting point of this issue was gvegayon/parallel#66

[mcaceresb]

Thanks! I'll look over that issue thread and see if there's any clues there. I upgraded to parallel 0.19 and still no issue. Will keep you posted.

[mcaceresb]

I've tried it on 2 other Linux systems w/o error. It's really strange to me because we are using the same OS and gtools version anyway... Can you run:

clear all
log using test, text
which reg
which regress
which _get_diopts
sysuse auto , clear

cap prog drop pargegen
prog define pargegen, rclass
	version 13
	syntax varlist [if]
	marksample touse
	egen test = sum(price)
        reg `varlist' if `touse'
	gegen test = sum(price), replace
        reg `varlist' if `touse'
	drop test
end

pargegen price weight foreign rep78
log close _all

Sorry I'm basically going blind here, since I don't get the same issue you do. Hopefully we can find the root cause of the problem, though.

[ppaspp]

Thanks for all your efforts. I noticed that I can reproduce the error even with this code:

clear all
log using test, text
which reg
which regress
which _get_diopts
sysuse auto , clear

egen test = mean(price)
reg price weight
gegen test2 = mean(price)
reg price weight

log close _all

log: test.log

Removing all ados in PLUS and reinstalling only gtools didn't help either. Might want to try reinstalling Stata itself?

[mcaceresb]

Does this happen with other gtools commands or is it just gegen? So if you run gunique price or gtop price does it also give an error with reg? (Note: Maybe try regress and _regress instead of reg?)

[ppaspp]

gtop and gunique produce the same error, also with regress or _regress.

[mcaceresb]

Can I have the log for

findfile gtools_unix.plugin
!ldd `r(fn)'
clear all
sysuse auto
global GTOOLS_CALLER gegen
_gtools_internal price, gfunction(hash) gen(test) debug(9) v bench(3)
_regress price weight

? I expect you will get the same error. Hopefully the log has a clue...

[ppaspp]

here is the log: test.log

[mcaceresb]

log for

clear all
sysuse auto
global GTOOLS_CALLER gegen
_gtools_internal price, gfunction(hash) gen(test)
set trace on
_get_diopts diopts options,

?

[ppaspp]

test.log

[mcaceresb]

This is very strange; there is nothing in the source file that suggests where level gets assigned to its default value of 95. It should be blank throughout... Could you edit _get_diopts.ado?

findfile _get_diopts.ado
!cp `r(fn)' `r(fn)'.bak

Could you add

disp "`level'"

to GetDiopts, before and after the first syntax statement (for me about line 81). Also after the last opts_exclusive (for me around line 96) as well as before and after the second syntax statement (for me around line 157, right before the level error.

Then re-run the latest snippet. Thank you for all your help debugging.

[ppaspp]

log: test.log

[mcaceresb]

Ok, I think this at least gives a clue, since level is getting assigned to its default value and that turns out to be "95,0" for some bizarre reason... Although you said you you get the same error when you pass level. Ok, could you run:

clear all
sysuse auto
regress price weight, level(90)
gegen test = mean(price)
creturn list
disp "`c(level)', `c(clevel)'"
set trace on
_get_diopts diopts options, level(89)

(keep the modifications to _get_diopts.ado).

[ppaspp]

log: test.log

[mcaceresb]

Oh, wow, so it's not setting the default. It's actually appending ",0" to whatever you pass. That is exceedingly bizarre. Can you try:

capture program drop test
program test
    syntax, [level(cilevel)]
    disp "`level'"
end

clear all
set obs 10
gen x = 1
test
gegen y = sum(x)
test

[ppaspp]

Yeah, bizarre is probably the right description. Here the output:

. set obs 10
number of observations (_N) was 0, now 10

. gen x = 1

. test
95

. gegen y = sum(x)

. test
95,0

[mcaceresb]

Can you try that with the develop branch? net uninstall gtools; then from a fresh Stata session can you run:

local github "https://raw.githubusercontent.com"
net install gtools, from(`github'/mcaceresb/stata-gtools/develop/build/)

clear all
capture program drop test
program test
    disp "`0'"
    syntax, [foo(cilevel)]
    disp "`foo'"
end

set obs 10
gen x = 1
test
gegen y = sum(x)
test

Can you also try

clear all
set obs 10
gen x = 1
syntax, [foo(cilevel)]
disp "`foo'"
gegen y = sum(x)
syntax, [foo(cilevel)]
disp "`foo'"

[ppaspp]

same result after installing dev branch:

. gen x = 1
. test
95
. gegen y = sum(x)
. test
95,0

and for the second program

. gen x = 1
. syntax, [foo(cilevel)]
. disp "`foo'"
95
. gegen y = sum(x)
. syntax, [foo(cilevel)]
. disp "`foo'"
95,0

[ppaspp]

By the way, I was unsure if some local configurations caused the error. Therefore, I installed Stata MP 15.1 + gtools in a fresh Ubuntu 18.04 virtualbox environment. Too bad, I was able to reproduce the same error.

[mcaceresb]

Tried fresh 18.04 in virtualbox as well and no error for me. I have no clue what's up! I'm really sorry. Un_n

• Ubuntu 18.04
• Virtualbox version 5.2.12 r122571
• Stata 15.1 SE

. version
version 15.1

. !uname -a

Linux mauricio-vb 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

. clear all

. set obs 10
number of observations (_N) was 0, now 10

. gen x = 1

. syntax, [foo(cilevel)]

. disp "`foo'"
95

. gegen y = sum(x)

. syntax, [foo(cilevel)]

. disp "`foo'"
95

[mcaceresb]

Can you try again? Gtools 1.0 just hit the develop branch. While it doesn't address this specifically, the plugin has changed enough that I'm hoping whatever internals were messing it up are not there anymore (on Stata 15 it should use a newer version of the plugin interface, for example; make sure you have version 1.0.0 by running which gtools).

[ppaspp]

same same ...

. which gegen
/home/pp/ado/plus/g/gegen.ado
*! version 1.0.0 21Jul2018 Mauricio Caceres Bravo, mauricio.caceres.bravo@gmail.com
*! implementation -egen- using C for faster processing

. clear all
. set obs 10
number of observations (_N) was 0, now 10

. gen x = 1
. syntax, [foo(cilevel)]
. disp "`foo'"
95
. gegen y = sum(x)
. syntax, [foo(cilevel)]
. disp "`foo'"
95,0

[mcaceresb]

Ok, that one is on me; really sorry. I had compiled that for Windows and I forgot you're also on Arch. I re-posted the files as a ZIP here. Can you try again? You can just do do test.do for the test script on that folder.

[ppaspp]

this is the output:

. do test.do
. clear all
. program test1, plugin using(test1.plugin)
. plugin call test1
Hello World
. syntax, [foo(cilevel)]
. disp "`foo'"
95
. 
. program test2, plugin using(test2.plugin)
. plugin call test2
foo:      0.00
hash: 12485269540145512313, 5018404763475985066
. syntax, [foo(cilevel)]
. disp "`foo'"
95
. 
. set obs 1
number of observations (_N) was 0, now 1
. global GTOOLS_CALLER ghash
. _gtools_internal
. syntax, [foo(cilevel)]
. disp "`foo'"
95,0

[mcaceresb]

I was really hoping it was the plugin interface in general, but it seems to be something in gtools.

Can you add

syntax, [foo(cilevel)]
disp "`foo'"

right before and right after line 2157 of _gtools_internal.ado (it would be in your adopath, for me ~/ado/plus/_/_gtools_internal.ado) and re-run only the third chunk of the snippet with set trace on then post the log?

The idea is to know whether it's something in the Sata code or the C code.

[ppaspp]

here is the log file: log.log

[lmusolff]

I've come across a similar (the same?) issue while using ppmlhdfe and gegen in the same do-file: using gegen before ppmlhdfe causes ppmlhdfe to fail with "invalid syntax" while using egen causes no such problem. This issue only occurs when using Stata on our linux cluster; the code with gegen worked without issues on my MacOS X laptop. (This causes no problems for me as I can simply use egen, but I thought it might be helpful to know that the issue might be Linux-specific. Although of course many other variables also differ between the cluster and my laptop...)

[sergiocorreia]

If you run ppmlhdfe with the verbose(1) option, or if you run "set trace" beforehand, where in the program is it reporting an error?

Ppmlhdfe doesn't use C plugins, so on principle it should be like any other programs

[lmusolff]

Using verbose(1) does not change the output (it still just says "invalid syntax"). The output from set trace on is attached; I believe it fails in _get_diopts. (I don't think this is an issue with ppmlhdfe; rather I think ppmlhdfe in my example fails in the same way as reg fails above. The error is presumably in gegen, or even more low-level)

output.log

[mcaceresb]

For whatever reason it seems that there are corner cases where gegen causes the default cilevel to be set to 95,0 instead of just 95, leading to syntax errors.

It is very puzzling to me because I never set ci level in any gtools programs and I cannot reproduce the error (be it in my Linux laptop or the Linux servers I use).

[mcaceresb]

@ppoppitz @lmusolff If you still get this bug, can you try the develop branch? (gtools version 1.3.2; gtools, upgrade branch(develop))

I just got a report that I suspect is related (#53). It occurs to me that maybe if I save the state of level before running the plugin and restore it after then maybe I can bypass the problem.

[ppaspp]

sry, I still get the same errors with version 1.3.2 14Feb2019

[mcaceresb]

@ppoppitz Can you try gtools version 1.3.3? gtools, upgrade branch(develop); I removed a dependency that might have been causing the problem. In #53 it looks like gegen no longer sets level to 95,0; hope it's the same for you.

[lmusolff]

I verified the code from #53 gives me the same error as there with gtools 1.1.2, but when I try to upgrade I get the following error:

. gtools, upgrade branch(develop)
checking gtools consistency and verifying not already installed...
could not copy https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop

/build/fasterxtile.sthlp
(no action taken)
--Break--
r(1);

[mcaceresb]

Try net uninstall gtools and then

local github "https://raw.githubusercontent.com"
net install gtools, from(`github'/mcaceresb/stata-gtools/develop/build/)

If you get the same error, what happens if you run

do https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthl

You should see

. .h gquantiles
invalid syntax
r(198);

Let me know if you don't.

[lmusolff]

Here's the output from those commands:

. net uninstall gtools
package not found
r(111);

. local github "https://raw.githubusercontent.com"

. net install gtools, from(`github'/mcaceresb/stata-gtools/develop/build/)
checking gtools consistency and verifying not already installed...
could not copy https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthlp
(no action taken)
--Break--
r(1);

. do https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthl
file https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthl not found
r(601);

[mcaceresb]

That was a typo on my part, I meant do https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthlp

In any case, maybe try this:

copy https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/src/github-issues/40/gtools-1.3.3.zip gtools.zip
unzipfile gtools.zip
net install gtools, from(`c(pwd)'/gtools/) replace

[lmusolff]

The former gives me r(1), but the second successfully upgraded to "version 1.3.3 15Feb2019". And now I can run the code from #53 without an error :) (I'll try my own at some point but I doubt it would differ, and it is part of a big job that I can't really isolate so need to schedule a run for)

Thanks for looking into this!

[mcaceresb]

I think I'll ask Stata why that file can't be copied on some systems. You're not the first person to have this issue. Can you run

copy https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthlp fasterxtile.sthlp

and let me know the error code, if any?

[lmusolff]

Yes, I vaguely remember having trouble installing it the normal way back when I did because I think of the same error. From that command I get:

. copy https://raw.githubusercontent.com/mcaceresb/stata-gtools/develop/build/fasterxtile.sthlp fasterxtile.sthlp
--Break--
r(1);

EDIT: I just ran the same command on my MacOsX Laptop and it gives me the same r(1). But maybe this should be a different issue; let me know if you'd like me to open one.

[mcaceresb]

Can you re-try? I want to know if this had anything to do with the fact it's a tiny file (14 bytes). I just modified the file to be a copy of gquantiles.sthlp (what's your Stata version, by the way?)

[lmusolff]

Oh wow, apparently that did it: the copy command executes now without problem (both on my laptop and the server). My Stata version is 13.1.

[mcaceresb]

Ok, great. I'm just going to have my build file copy gquantiles.sthlp into fasterxtile.sthlp and bypass this altogether. Thanks for testing it out!

[lmusolff]

Of course, no worries! Thanks for writing & maintaining this awesome package!

[ppaspp]

Awesome, it's working here too. Thanks again for your efforts!