Weird error with $variable and grep in command line magic (!command)

[ghost]

With the latest stable version of IPython the following works fine in the IPython notebook (I had to unalias ls for my particular case, not sure if that's part of the error):

In[1]: dir = "/path/to/some/dir/"
  ...: !echo $dir
  ...: !\ls "/path/to/some/dir/" | grep "^ab\{2\}"

/path/to/some/dir/
abb-1
abb-2
abb-3

However, the following should provide the same output but instead provides the output shown:

In[2]: dir = "/path/to/some/dir/"
  ...: !echo $dir
  ...: !\ls $dir | grep "^ab\{2\}"

/path/to/some/dir/

In this second case the matching directories aren't listed. It seems to involve the use of \{#\} in grep as things work if I simply try to match "^a" even with the use of $dir instead of the literal directory string.

[takluyver]

That regex looks like you'll end up matching names that start with the exact string ab{2}. If you're trying to escape the braces, try doubling them: {{a}}.

[ghost]

I'm not trying to escape the braces. I'm trying to match all names with a given prefix. This was the use case in which I stumbled across the apparent error mentioned.

[takluyver]

To put it another way: if you're trying to pass grep the regex ab{2}, try doubling the braces rather than sticking slashes in front of them.

[minrk]

IPython applies Python string formatting to your subcommands, so {expr} is replaced with the result of eval(expr). Now, you might expect that the backslash functions as an escape, but it doesn't (for whatever reason). If you want to escape the braces so they arrive at the subcommand, you will want to double up ({{2}}). You can see the string that IPython passes to the shell by testing ip.var_expand("command"). The one additional piece of relevant information is that if variable expansion fails, none of it is done. For example:

In [2]: ip = get_ipython()

In [3]: ip.var_expand('!ls $dir')
Out[3]: u'!ls /Users/minrk/temp/somedir'

In [4]: ip.var_expand('!ls $dir | grep "^ab{2}"')
Out[5]: u'!ls /Users/minrk/temp/somedir | grep "^ab2"'

In [5]: ip.var_expand('!ls $dir | grep "^ab\{2\}"')
Out[5]: '!ls $dir | grep "^ab\\{2\\}"'

In [6]: ip.var_expand('!ls $dir | grep "^ab\{{2\}}"')
Out[6]: u'!ls /Users/minrk/temp/somedir | grep "^ab\\{2\\}"'

In [7]: !ls $dir | grep "^ab\{{2\}}"
abb-1
abb-2
abb-3

• Case 3: works fine
• Case 4: {2} is replaced with eval('2'), which is just 2
• Case 5: {2\} tries to replace with eval('2\') which fails, resulting in untransformed input (note that even $dir isn't replaced)
• Case 6: {{2\}} is the regular Python formatting way to escape braces that should not do anything, and is the actual command that you want, as confirmed in input 7.

[ghost]

Ah, I think I get it now.

In[1]: !\ls "/path/to/some/dir/" | grep "^ab\{2\}"

works fine because variable expansion isn't triggered as there's no $var to interpret, but

In[2]: adir = "/path/to/some/dir/" 
  ...: !\ls $adir | grep "^ab\{2\}"

breaks because the $adir triggers expansion which breaks due to the {2\} and thus I don't get my expected results.

I can fix my notebook now.

[minrk]

That's close, but a little bit backwards - it's the attempted expansion of {2\} that fails, which failure prevents the substitution of $adir:

In [1]: a = 5

In [2]: !echo a=$a {2}
a=5 2

In [3]: !echo a=$a {2\\}
a= {2\}

In [4]: !echo a=$a {{2}}
a=5 {2}

In [5]: !echo a=$a {{2\\}}
a=5 {2\}

[ghost]

Ah, yes. So expansion/interpretation is always happening and it fails on the {2\} in both cases. It's only noticeable in case 2 due to the $adir needing to be expanded. Thanks for the insight.

[minrk]

So expansion/interpretation is always happening

Yes, this is a key point, I should have mentioned that.