Propagating values back from Python script to SmPL rule with other metavariable type than “identifier” #86

elfring · 2016-10-21T17:00:54Z

I tried the following small script out for the semantic patch language of the software combination “Coccinelle 1.0.6-00006-gc8025ac (OCaml 4.03.0)” on another tiny source code example.

enumerate_single_string_parameters-20161021.cocci:

@initialize:python@
@@
count = 0

@find_update_candidate@
constant string;
position pos;
@@
 puts@pos(string);

@script:python generation@
text;
@@
count = count + 1
coccinelle.text = "Test " + str(count)

@replacement@
constant find_update_candidate.string,
         generation.text;
position find_update_candidate.pos;
@@
 puts@pos(
-         string
+         text
         )

puts1.c:

#include <stdio.h>

int main(void)
{
 puts("first");
 puts("second");
 return 0;
}

Test result:

elfring@Sonne:~/Projekte/Coccinelle/janitor> spatch.opt enumerate_single_string_parameters-20161021.cocci ../Probe/puts1.c
…
Fatal error: exception Common.Impossible(145)

Is my understanding of the used SmPL Python programming interface incomplete here?
How does this error message fit to the functionality which is demonstrated by the script “pythontococci.cocci”?

The text was updated successfully, but these errors were encountered:

elfring · 2016-10-23T06:57:02Z

I would appreciate another constructive comment for this issue.

JuliaLawall · 2016-10-25T18:23:31Z

On Fri, 21 Oct 2016, Markus Elfring wrote:

I tried the following small script out for the semantic patch language of
the software combination “Coccinelle 1.0.6-00006-gc8025ac (OCaml 4.03.0)” on
another tiny source code example.

enumerate_single_string_parameters-20161021.cocci:

@initialize:python@
@@
count = 0

@find_update_candidate@
constant string;
position pos;
@@
puts@pos(string);

@script:python generation@
text;
@@
count = count + 1
coccinelle.text = "Test " + str(count)

@replacement@
constant find_update_candidate.string,
generation.text;

The problem is here. When you construct the value of a metavariable in
python code, it is an identifier. Here you have inherited it in a
metavariable of type constant, which is an expression not an identifier.

It's not clear at all why you want to put a term of the form Test 1 in a
metavariable of type constant. It doesn't look like a constant, and,
because of the space, it is not even an expression.

If you change the metavariable type of generation.text to identifier,
Coccinelle generates some code. It is not correct code, however, because
of the space in the middle of the argument of puts.

If you want to leave the type of generation.text as constant, then in the
python code you need to make an expression. For this, the code could be:

coccinelle.text = cocci.make_expr("Test " + str(count))

But that will not work because Test 1, with the space in the middle, is
not a valid expression. Everything is fine if you remove the space.

julia

position find_update_candidate.pos;
@@
puts@pos(
    string
    text
 )
puts1.c:

#include <stdio.h>

int main(void)
{
puts("first");
puts("second");
return 0;
}

Test result:

elfring@Sonne:~/Projekte/Coccinelle/janitor> spatch.opt enumerate_single_str
ing_parameters-20161021.cocci ../Probe/puts1.c
…
Fatal error: exception Common.Impossible(145)

Is my understanding of the used SmPL Python programming interface
incomplete here?

How does this error message fit to the functionality which is
demonstrated by the script “pythontococci.cocci”?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the
thread.[AAesmltrf42dhSrDIx6fsNwoeIgcMI18ks5q2O_GgaJpZM4Kda9T.gif]

elfring · 2016-10-26T07:34:59Z

Thanks for your constructive explanation (which happened not together with the closing action for this clarification request).

When you construct the value of a metavariable in python code, it is an identifier.

Unfortunately, I was not really aware of this software detail before.

Here you have inherited it in a metavariable of type constant, which is an expression

I can not completely understand this aspect at the moment.

not an identifier.

I tried to achieve something different then.

It's not clear at all why you want to put a term of the form Test 1 in a metavariable of type constant.

My knowledge was incomplete for your SmPL programming interface. I tried to construct a string literal in the Python script.

It doesn't look like a constant, and, because of the space, it is not even an expression.

May expressions contain extra white-space characters?

But that will not work because Test 1, with the space in the middle, is not a valid expression.

How can a literal be created including a desired space character?

JuliaLawall · 2016-10-26T15:41:54Z

On Wed, 26 Oct 2016, Markus Elfring wrote:

Thanks for your constructive explanation (which happened not together with
the closing action for this clarification request).

It happened before. I responded to the email and then went to github and
closed the issue.

  When you construct the value of a metavariable in python code,
  it is an identifier.
Unfortunately, I was not really aware of this software detail before.
  Here you have inherited it in a metavariable of type constant,
  which is an expression
I can not completely understand this aspect at the moment.

Which part don't you understand?

  not an identifier.
I tried to achieve something different then.
  It's not clear at all why you want to put a term of the form
  Test 1 in a metavariable of type constant.
My knowledge was incomplete for your SmPL programming interface. I tried to
construct a string literal in the Python script.

A string literal would be a value that has quotes around it. You would
make it like ""a string"". Terms that are created in python always have
to have extra quotes around the outside. Otherwise, python would try to
evaluate the code.

  It doesn't look like a constant, and, because of the space, it
  is not even an expression.
May expressions contain extra white-space characters?

It depends on where they occur. foo bar is not a valid subterm of a C
program. foo + bar and foo + bar are completely fine.

  But that will not work because Test 1, with the space in the
  middle, is not a valid expression.
How can a literal be created including a desired space character?

See above. WIth "

julia

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the
thread.[AAesmtRPIB5jxmVl0T29Su4gUPLXs5bsks5q3wKjgaJpZM4Kda9T.gif]

elfring · 2016-10-26T17:34:36Z

Which part don't you understand?

How do string literals fit into the discussed taxonomy of “expressions”?
Which source code (and SmPL rule) triggered the message “Fatal error: exception Common.Impossible(145)”?

A string literal would be a value that has quotes around it.

I have tried another SmPL script variant out.

enumerate_single_string_parameters-20161026-b.cocci:

@initialize:python@
@@
count = 0
mark = ['"Test ', '', '"']

@find_update_candidate@
constant string;
position pos;
@@
 puts@pos(string);

@script:python generation@
text;
@@
count = count + 1
mark[1] = str(count)
coccinelle.text = cocci.make_expr(''.join(mark))

@replacement@
constant find_update_candidate.string,
         generation.text;
position find_update_candidate.pos;
@@
 puts@pos(
-         string
+         text
         )

Test result:

diff = 
--- ../Probe/puts1.c
+++ /tmp/cocci-output-5978-0d97b7-puts1.c
@@ -2,7 +2,7 @@

 int main(void)
 {
- puts("first");
- puts("second");
+ puts("Test 1");
+ puts("Test 1");
  return 0;
 }

This display looks only partly in the way that I expected. Should each matched function call get a string with a higher number in such an use case?

JuliaLawall · 2016-10-26T18:48:57Z

On Wed, 26 Oct 2016, Markus Elfring wrote:

  Which part don't you understand?
How do string literals fit into the discussed taxonomy of “expressions”?

Does a string have a value? Yes. Therefore it is an expression.

Which source code (and SmPL rule) triggered the message “Fatal error:
exception Common.Impossible(145)”?

For the SmPL rule, you could have figured this out yourself using the
--debug option. The message came when trying to bind an identifier value
to an expression metavariable. Actually, you could also have figured that
out by grepping for the number 145.

  A string literal would be a value that has quotes around it.
I have tried another SmPL script variant out.

enumerate_single_string_parameters-20161026-b.cocci:

@initialize:python@
@@
count = 0
mark = ['"Test ', '', '"']

@find_update_candidate@
constant string;
position pos;
@@
puts@pos(string);

@script:python generation@
text;
@@
count = count + 1
mark[1] = str(count)
coccinelle.text = cocci.make_expr(''.join(mark))

@replacement@
constant find_update_candidate.string,
generation.text;
position find_update_candidate.pos;
@@
puts@pos(
    string
    text
 )
Test result:

diff =
--- ../Probe/puts1.c
+++ /tmp/cocci-output-5978-0d97b7-puts1.c
@@ -2,7 +2,7 @@

int main(void)
{

puts("first");

puts("second");

puts("Test 1");

puts("Test 1");
return 0;
}

This display looks only partly in the way that I expected. Should each
matched function call get a string with a higher number in such an use case?

No. The script is run exactly once per file, because it doesn't depend on
any metavariables.

julia

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the
thread.[AAesmuGgysdGJ-DSnR70CiBsyeQytftjks5q348tgaJpZM4Kda9T.gif]

elfring · 2016-10-26T19:31:55Z

The script is run exactly once per file, because it doesn't depend on any metavariables.

Thanks for another useful information.

The following SmPL approach shows the desired data display finally.

enumerate_single_string_parameters-20161026-c.cocci:

@initialize:python@
@@
count = 0
mark = ['"Test ', '', '"']

@find_update_candidate@
constant string;
position pos;
@@
 puts@pos(string);

@script:python generation@
pos << find_update_candidate.pos;
text;
@@
count = count + 1
mark[1] = str(count)
coccinelle.text = cocci.make_expr(''.join(mark))

@replacement@
constant find_update_candidate.string,
         generation.text;
position find_update_candidate.pos;
@@
 puts@pos(
-         string
+         text
         )

Test result:

diff = 
--- ../Probe/puts1.c
+++ /tmp/cocci-output-9286-21b87a-puts1.c
@@ -2,7 +2,7 @@

 int main(void)
 {
- puts("first");
- puts("second");
+ puts("Test 1");
+ puts("Test 2");
  return 0;
 }

elfring · 2016-10-26T20:44:39Z

The script is run exactly once per file, because it doesn't depend on any metavariables.

I find that this information is also worth for further considerations after a desired transformation result was achieved by a test script.

The semantic patch language provides an interface for the specification of dependencies in each rule. I got the impression that these specifications refer to SmPL rule names.
It seems that one place where specific metavariables would be usually referenced is the mapping of inherited metavariables to embedded programming script variables like in the shown example “pos << find_update_candidate.pos;”. The variable “pos” is not used further in the corresponding Python code.
- Can this detail trigger doubts about its usefullness?
- Does this extra mapping specification create another kind of dependency as a “side effect”?
- It it possible to specify such a dependency also without an assignment to a local variable?

JuliaLawall · 2016-10-27T15:47:11Z

On Wed, 26 Oct 2016, Markus Elfring wrote:

  The script is run exactly once per file, because it doesn't
  depend on any metavariables.
I find that this information is also worth for further considerations after
a desired transformation result was achieved by a test script.

The semantic patch language provides an interface for the specification
of dependencies in each rule. I got the impression that these
specifications refer to SmPL rule names.

It seems that one place where specific metavariables would be usually
referenced is the mapping of inherited metavariables to embedded
programming script variables like in the shown example “pos <<
find_update_candidate.pos;”. The variable “pos” is not used further in
the corresponding Python code.

Can this detail trigger doubts about its usefullness?

No idea what "its" refers to.

 +  Does this extra mapping specification create another kind of
    dependency as a “side effect”?

Yes.

 +  It it possible to specify such a dependency also without an
    assignment to a local variable?

No. If you want one call per possible value of pos, then you have to
express that in some way. This allows you to rather precisely control hoe
often the script code is executed.

julia

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the
thread.[AAesmn6pPxOxtvAh8BYOcgqExFDoR6Keks5q37u3gaJpZM4Kda9T.gif]

elfring · 2016-10-27T18:14:35Z

No idea what "its" refers to.

The variable “pos” which was specified in the definition area of the SmPL rule “generation”.

If you want one call per possible value of pos, then you have to express that in some way.

Yes, of course.

How do you think about to use a special syntax for this purpose so that such a dependency variant can be expressed independent from an assignment to a variable in a programming language script?

JuliaLawall · 2016-10-27T18:36:15Z

On Thu, 27 Oct 2016, Markus Elfring wrote:

  No idea what "its" refers to.
The variable “pos” which was specified in the definition area of the SmPL
rule “generation”.
  If you want one call per possible value of pos, then you have to
  express that in some way.
Yes, of course.

How do you think about to use a special syntax for this purpose so that such
a dependency variant can be expressed independent from an assignment to a
variable in a programming language script?

Why? To make Coccinelle hundreds of times more efficient?

julia

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the
thread.[AAesmlDr8pH_KtflMvAyluyafT_GACuiks5q4OoLgaJpZM4Kda9T.gif]

elfring · 2016-10-27T18:52:13Z

Why?

I would appreciate if the suggested fine-tuning of the dependency specification can help to improve the software execution characteristics a bit.
Have you selected any special name for this kind of dependency already?

The following thread was helpful while writing this script: coccinelle/coccinelle#86 Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20170718045540.16322-3-f4bug@amsat.org> Signed-off-by: Richard Henderson <rth@twiddle.net>

JuliaLawall closed this as completed Oct 22, 2016

elfring mentioned this issue Oct 27, 2016

Development challenges around using the function “make_expr” for expression construction by SmPL #88

Open

Propagating values back from Python script to SmPL rule with other metavariable type than “identifier” #86

Propagating values back from Python script to SmPL rule with other metavariable type than “identifier” #86

elfring commented Oct 21, 2016

elfring commented Oct 23, 2016

JuliaLawall commented Oct 25, 2016

elfring commented Oct 26, 2016

JuliaLawall commented Oct 26, 2016

elfring commented Oct 26, 2016

JuliaLawall commented Oct 26, 2016

elfring commented Oct 26, 2016

elfring commented Oct 26, 2016

JuliaLawall commented Oct 27, 2016

elfring commented Oct 27, 2016

JuliaLawall commented Oct 27, 2016

elfring commented Oct 27, 2016

Propagating values back from Python script to SmPL rule with other metavariable type than “identifier” #86

Propagating values back from Python script to SmPL rule with other metavariable type than “identifier” #86

Comments

elfring commented Oct 21, 2016

elfring commented Oct 23, 2016

JuliaLawall commented Oct 25, 2016

elfring commented Oct 26, 2016

JuliaLawall commented Oct 26, 2016

elfring commented Oct 26, 2016

JuliaLawall commented Oct 26, 2016

elfring commented Oct 26, 2016

elfring commented Oct 26, 2016

JuliaLawall commented Oct 27, 2016

elfring commented Oct 27, 2016

JuliaLawall commented Oct 27, 2016

elfring commented Oct 27, 2016