Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100755 426 lines (387 sloc) 16.963 kb
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
1 #!/usr/bin/env python
2 # -*- coding: utf-8 -*-
3 # Copyright © 2010 Progiciels Bourbeau-Pinard inc.
4 # François Pinard <pinard@iro.umontreal.ca>, 2010.
5
6 """\
165fba0 @pinard Rename p4 to pppp, avoiding a clash with Perforce
authored
7 Poor's Python Pre-Processor (pppp).
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
8
165fba0 @pinard Rename p4 to pppp, avoiding a clash with Perforce
authored
9 Usage: pppp -m [OPTION]... FILE1 FILE2
10 or: pppp [OPTION]... [FILE]...
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
11
12 General options:
13 -h Print this help and do nothing else
14 -m Produce, on standard output, a merged version
e0f5ed1 @pinard In p4, exchange the meaning of -c and -C
authored
15 -c Clean files which would normally have been produced
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
16 -v Be verbose about written (or deleted) files
6399f2f @pinard Implement modified file detection in p4, and option -f
authored
17 -f Force deletion of rewriting, even if files were modified
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
18
19 Context setting options:
e0f5ed1 @pinard In p4, exchange the meaning of -c and -C
authored
20 -C FILE Evaluate Python FILE for preparing context
21 -D name Define "name" as True
22 -D name=expr Define "name" as the value of Python "expr"
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
23
24 Transformation options:
25 -o OUTPUT Collect output files into the OUTPUT directory
26 -i INDENT Indentation step, default value is 4
27 -s SUFFIX Suffix to mark transformable paths, if not '.in'
28 -p Force all files to be interpreted as Python
6c9fe0c @pinard Implement line synchronisation in p4, and option -n
authored
29 -n Avoid trying to keep line numbers synchronized
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
30
31 With -m, a single -Dname option is also required. FILE1 and FILE2 are
32 merged and the result written to standard output, augmented as needed
33 with "if name:", "if not name:" and "else:" directives, such that FILE1
34 is meant when "name" is False, FILE2 is meant when "name" is True.
35
e0f5ed1 @pinard In p4, exchange the meaning of -c and -C
authored
36 Without -cm, files go through an elementary pre-processing. If no FILE
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
37 is given, standard input is transformed and written to standard output.
e0f5ed1 @pinard In p4, exchange the meaning of -c and -C
authored
38 Otherwise, if FILE is a directory, it is recursively traversed for the
39 files it contains. A file is eligible for transformation only when
40 it's full path name (starting at FILE and down) has a component for
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
41 which there is a '.in' suffix. The file receiving a transformed file
42 is derived removing all such '.in' suffixes, and prepending OUTPUT as a
43 directory if specified. Output directories are created as needed.
44
45 Within each file to transform, each occurrence of @name@ gets replaced
46 by the string of the context value for "name", when such is defined.
47 Moreover, a Python source (either when -p, or a file which name ends
48 with .py or .py.in, or for which the first line starts with "!#" and has
49 some "ython" string in it) is further handled as described below
50
51 A Python source has all its "if EXPR:", "elif EXPR:" and corresponding
52 "else:" lines checked (each on a single line, and without comments).
53 If "EXPR" is a valid Python expression for which primitives are either
54 constants or names introduced through -D options, the "if" or "elif"
55 line is removed and succeeding lines are either shifted or removed.
56 """
57
58 __metaclass__ = type
1cc4c78 @pinard pppp, Pymacs.py.in: Some PEP8.
authored
59 import os
60 import re
61 import sys
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
62
165fba0 @pinard Rename p4 to pppp, avoiding a clash with Perforce
authored
63 endif_pppp = '#endif (pppp)'
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
64
1cc4c78 @pinard pppp, Pymacs.py.in: Some PEP8.
authored
65
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
66 class Main:
67 output = None
68 context = {}
69 merge = False
70 indent = 4
71 suffix = '.in'
72 python = False
73 verbose = False
74 clean = False
6c9fe0c @pinard Implement line synchronisation in p4, and option -n
authored
75 synclines = True
6399f2f @pinard Implement modified file detection in p4, and option -f
authored
76 force = False
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
77
78 def main(self, *arguments):
79 import getopt
e0f5ed1 @pinard In p4, exchange the meaning of -c and -C
authored
80 options, arguments = getopt.getopt(arguments, 'C:D:cfhi:mno:ps:v')
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
81 for option, value in options:
82 if option == '-C':
e0f5ed1 @pinard In p4, exchange the meaning of -c and -C
authored
83 exec(compile(open(value).read(), value, 'exec'), self.context)
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
84 elif option == '-D':
85 if '=' in value:
86 name, value = value.split('=', 1)
e0f5ed1 @pinard In p4, exchange the meaning of -c and -C
authored
87 self.context[name] = eval(value, {})
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
88 else:
892d1fb @bravegnu Fix incorrect access of context for -Dvar definitions.
bravegnu authored
89 self.context[value] = True
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
90 elif option == '-c':
e0f5ed1 @pinard In p4, exchange the meaning of -c and -C
authored
91 self.clean = True
6399f2f @pinard Implement modified file detection in p4, and option -f
authored
92 elif option == '-f':
93 self.force = True
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
94 elif option == '-h':
95 sys.stdout.write(__doc__)
96 return
97 elif option == '-i':
98 self.indent = int(value)
99 elif option == '-m':
100 self.merge = True
6c9fe0c @pinard Implement line synchronisation in p4, and option -n
authored
101 elif option == '-n':
102 self.synclines = False
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
103 elif option == '-o':
104 self.output = value
105 elif option == '-p':
106 self.python = True
107 elif option == '-s':
108 self.suffix = value
109 elif option == '-v':
110 self.verbose = True
d3892b6 @pinard Added documentation for p4.
authored
111 if not self.suffix and not self.output:
112 sys.exit("Option -o is needed with an empty suffix.")
113
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
114 if self.merge:
115 if len(arguments) != 2 or len(self.context) != 1:
116 sys.exit("Try `%s -h' for help" % sys.argv[0])
117 self.merge_files(arguments[0], arguments[1], sys.stdout.write)
118 elif not arguments:
119 if not self.clean:
120 self.transform_file(
1cc4c78 @pinard pppp, Pymacs.py.in: Some PEP8.
authored
121 sys.stdin.name, sys.stdin, sys.stdout.write)
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
122 else:
123 for argument in arguments:
124 if os.path.isdir(argument):
125 self.transform_all_files(argument)
126 elif argument.endswith(self.suffix):
127 self.transform_all_files(argument)
128 else:
129 sys.stderr.write(
1cc4c78 @pinard pppp, Pymacs.py.in: Some PEP8.
authored
130 "* %s does not end with %s, ignored.\n"
131 % (argument, self.suffix))
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
132
133 def merge_files(self, file1, file2, write):
134 left = list(open(file1))
135 right = list(open(file2))
136 block_margin = None
137
138 def protect_block(margin, danger):
139 if block_margin is None:
140 return
141 if margin is None:
142 return
143 if margin < block_margin:
144 return
145 if margin == block_margin and not danger:
146 return
165fba0 @pinard Rename p4 to pppp, avoiding a clash with Perforce
authored
147 write(' ' * block_margin + endif_pppp + '\n')
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
148
149 def check_next(lines, lo, hi):
150 for index in range(lo, hi):
151 line = lines[index].rstrip()
152 short = line.lstrip()
153 if short:
154 return (len(line) - len(short),
155 short.startswith('elif ')
156 or short.startswith('else:'))
157 return None, False
158
159 def lines_margin(lines, lo, hi):
160 margin = None
161 for index in range(lo, hi):
162 line = lines[index].rstrip()
163 short = line.lstrip()
164 if short:
165 width = len(line) - len(short)
166 if margin is None:
167 margin = width
168 else:
169 margin = min(margin, width)
170 return margin or 0
171
172 def copy_lines(lines, lo, hi, prefix):
173 for index in range(lo, hi):
174 line = lines[index]
175 if line.lstrip(' \t') == '\n':
176 write(line)
177 else:
178 write(prefix + line)
179
180 name = list(self.context.keys()).pop()
181 import difflib
182 matcher = difflib.SequenceMatcher(None, left, right)
1cc4c78 @pinard pppp, Pymacs.py.in: Some PEP8.
authored
183 for (tag, low_left, high_left,
184 low_right, high_right) in matcher.get_opcodes():
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
185 if tag == 'equal':
186 margin, danger = check_next(left, low_left, high_left)
187 protect_block(margin, danger)
188 copy_lines(left, low_left, high_left, '')
189 block_margin = None
190 elif tag == 'delete':
191 margin = lines_margin(left, low_left, high_left)
192 protect_block(margin, False)
193 write(' ' * margin + 'if not ' + name + ':\n')
194 copy_lines(left, low_left, high_left, ' ' * self.indent)
195 block_margin = margin
196 elif tag == 'insert':
197 margin = lines_margin(right, low_right, high_right)
198 protect_block(margin, False)
199 write(' ' * margin + 'if ' + name + ':\n')
200 copy_lines(right, low_right, high_right, ' ' * self.indent)
201 block_margin = margin
202 elif tag == 'replace':
203 margin = min(lines_margin(left, low_left, high_left),
204 lines_margin(right, low_right, high_right))
205 protect_block(margin, False)
206 write(' ' * margin + 'if ' + name + ':\n')
207 copy_lines(right, low_right, high_right, ' ' * self.indent)
208 write(' ' * margin + 'else:\n')
209 copy_lines(left, low_left, high_left, ' ' * self.indent)
210 block_margin = margin
211 else:
212 assert False, tag
213
214 def transform_all_files(self, input):
215
216 def ensure_directory(name):
217 base = os.path.dirname(name)
218 if base and not os.path.isdir(base):
219 ensure_directory(base)
220 if self.verbose:
221 sys.stderr.write("creating %s\n" % base)
222 os.mkdir(base)
223
224 if self.output:
225 output = os.path.join(self.output, input)
226 else:
227 output = input
228 for input, output in self.each_pair(input, output):
6399f2f @pinard Implement modified file detection in p4, and option -f
authored
229 if not self.force:
1cc4c78 @pinard pppp, Pymacs.py.in: Some PEP8.
authored
230 if ((os.path.exists(output)
231 and os.path.getmtime(output) > os.path.getmtime(input))):
6399f2f @pinard Implement modified file detection in p4, and option -f
authored
232 sys.exit("ERROR: %s has been modified, keeping it!\n"
233 % output)
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
234 if self.clean:
235 if os.path.exists(output):
236 if self.verbose:
237 sys.stderr.write("deleting %s\n" % output)
238 os.remove(output)
239 else:
240 ensure_directory(output)
241 if self.verbose:
242 sys.stderr.write("writing %s\n" % output)
243 self.transform_file(
1cc4c78 @pinard pppp, Pymacs.py.in: Some PEP8.
authored
244 input, open(input), open(output, 'w').write)
434b6f6 @pinard In p4, remove any .pyc file next to a resulting .py file
authored
245 if output.endswith('.py'):
246 pyc_file = output[:-2] + '.pyc'
247 if os.path.exists(pyc_file):
248 if self.verbose:
249 sys.stderr.write("deleting %s\n" % pyc_file)
250 os.remove(pyc_file)
6399f2f @pinard Implement modified file detection in p4, and option -f
authored
251 os.utime(output, (os.path.getatime(input),
252 os.path.getmtime(input)))
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
253
254 def each_pair(self, input, output):
255 stack = []
256 output = output.replace(self.suffix + '/', '/')
257 if output.endswith(self.suffix):
258 output = output[:-len(self.suffix)]
259 stack.append((input, output))
260 while stack:
261 input_path, output_path = stack.pop()
262 if os.path.isdir(input_path):
263 for base in os.listdir(input_path):
264 input = os.path.join(input_path, base)
265 if base.endswith(self.suffix):
266 output = os.path.join(output_path,
267 base[:-len(self.suffix)])
268 else:
269 output = os.path.join(output_path, base)
270 stack.append((input, output))
271 elif ((self.suffix + '/') in input_path
272 or input_path.endswith(self.suffix)):
273 yield input_path, output_path
274
275 def transform_file(self, name, lines, write):
ad8aa77 @pinard Let p4 handle more complicated situations
authored
276
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
277 # MARGIN is the number of spaces preceding the previous "if" line.
278 # A virtual "if True:" is assumed above and left of the whole module.
279 margin = -1
280
ad8aa77 @pinard Let p4 handle more complicated situations
authored
281 # REMOVE is the number of leading spaces to delete on copied lines.
282 remove = 0
283
284 # STATE drives the copying or skipping of code. When the test
285 # expression of an "if" statement is known to be True, known to be
286 # False, or cannot be evaluated, STATE becomes TRUE, FALSE or UNKNOWN
287 # respectively (an "else" clause exchanges TRUE and FALSE, but leaves
288 # UNKNOWN undisturbed). FALSE2 is a special case of FALSE, for when
289 # some "if UNKNOWN:" is followed by "elif FALSE:", an "else:" clause
290 # might then be needed before resuming copy. STATE is SKIP when some
291 # "if" or "elif" clause has been known to be True, meaning that all
292 # following clauses may be removed. STATE becomes SKIP as well for a
293 # whole embedded "if" within some code which was already being skipped.
294 TRUE = 'TRUE'
295 FALSE = 'FALSE'
296 FALSE2 = 'FALSE2'
297 UNKNOWN = 'UNKNOWN'
298 SKIP = 'SKIP'
299 state = TRUE
300
301 # STACK saves the previous (MARGIN, REMOVE, STATE) whenever a nested
302 # "if" is met, and restores it when the actual margin goes left enough.
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
303 # A nested "if" is ignored unless its expression can be evaluated.
304 stack = []
305
ad8aa77 @pinard Let p4 handle more complicated situations
authored
306 def expression_value(text):
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
307 try:
ad8aa77 @pinard Let p4 handle more complicated situations
authored
308 value = eval(text, {'__builtins__': {}}, self.context)
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
309 except:
ad8aa77 @pinard Let p4 handle more complicated situations
authored
310 return UNKNOWN
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
311 else:
ad8aa77 @pinard Let p4 handle more complicated situations
authored
312 if value:
313 return TRUE
314 return FALSE
315
316 def write_shifted(line):
317 assert remove >= 0, (remove, line)
318 assert line[:remove] == ' ' * remove, (remove, line)
6c9fe0c @pinard Implement line synchronisation in p4, and option -n
authored
319 write_verbatim(line[remove:])
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
320
6c9fe0c @pinard Implement line synchronisation in p4, and option -n
authored
321 def write_verbatim(line):
322 write(line)
323 self.output_counter += line.count('\n')
324
325 self.output_counter = 0
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
326 python = (self.python
1cc4c78 @pinard pppp, Pymacs.py.in: Some PEP8.
authored
327 or name.endswith('.py')
328 or name.endswith('.py' + self.suffix))
6c9fe0c @pinard Implement line synchronisation in p4, and option -n
authored
329 for input_counter, line in enumerate(
330 self.each_substituded_line(lines)):
331 if (input_counter == 0
332 and line.startswith('#!') and 'ython' in line):
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
333 python = True
6c9fe0c @pinard Implement line synchronisation in p4, and option -n
authored
334 if self.synclines:
335 while self.output_counter < input_counter:
336 write_verbatim('\n')
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
337 if not python:
6c9fe0c @pinard Implement line synchronisation in p4, and option -n
authored
338 write_verbatim(line)
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
339 continue
340 short = line.lstrip()
341 if not short:
ad8aa77 @pinard Let p4 handle more complicated situations
authored
342 if state in (TRUE, UNKNOWN):
6c9fe0c @pinard Implement line synchronisation in p4, and option -n
authored
343 write_verbatim(line)
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
344 continue
345 width = len(line) - len(short)
346 while width < margin:
ad8aa77 @pinard Let p4 handle more complicated situations
authored
347 margin, remove, state = stack.pop()
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
348 if width == margin:
349 match = re.match('else: *$', short)
350 if match:
ad8aa77 @pinard Let p4 handle more complicated situations
authored
351 if state is TRUE:
352 state = FALSE
353 elif state is FALSE:
354 state = TRUE
355 elif state is FALSE2:
356 write_shifted(line)
357 state = TRUE
358 elif state in UNKNOWN:
359 write_shifted(line)
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
360 continue
361 match = re.match('elif (.*): *$', short)
362 if match:
ad8aa77 @pinard Let p4 handle more complicated situations
authored
363 if state is TRUE:
364 state = SKIP
365 elif state is FALSE:
366 value = expression_value(match.group(1))
367 if value is UNKNOWN:
368 remove -= self.indent
369 write_shifted(' ' * width + 'if' + short[4:])
370 state = value
371 elif state is FALSE2:
372 value = expression_value(match.group(1))
373 if value is UNKNOWN:
374 write_shifted(line)
375 else:
6c9fe0c @pinard Implement line synchronisation in p4, and option -n
authored
376 write_verbatim(' ' * width + 'else:\n')
ad8aa77 @pinard Let p4 handle more complicated situations
authored
377 state = value
378 elif state is UNKNOWN:
379 value = expression_value(match.group(1))
380 if value is TRUE:
381 write_shifted(' ' * width + 'else:\n')
382 state = TRUE
383 elif value is FALSE:
384 state = FALSE2
385 elif value is UNKNOWN:
386 write_shifted(line)
387 continue
388 margin, remove, state = stack.pop()
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
389 match = re.match('if (.*): *$', short)
390 if match:
ad8aa77 @pinard Let p4 handle more complicated situations
authored
391 stack.append((margin, remove, state))
392 margin = width
393 if state in (TRUE, UNKNOWN):
394 value = expression_value(match.group(1))
395 if value is UNKNOWN:
396 write_shifted(line)
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
397 else:
ad8aa77 @pinard Let p4 handle more complicated situations
authored
398 remove += self.indent
399 state = value
400 else:
401 state = SKIP
402 continue
165fba0 @pinard Rename p4 to pppp, avoiding a clash with Perforce
authored
403 if state in (UNKNOWN, TRUE) and not short.startswith(endif_pppp):
ad8aa77 @pinard Let p4 handle more complicated situations
authored
404 write_shifted(line)
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
405
406 def each_substituded_line(self, lines):
407 if self.context:
408 pattern = re.compile(
1cc4c78 @pinard pppp, Pymacs.py.in: Some PEP8.
authored
409 '@('
410 + '|'.join([re.escape(key) for key in self.context])
411 + ')@')
3fa2a32 @pinard New p4 script (Poor Python Pre Processor). Use it!
authored
412 for line in lines:
413 yield pattern.sub(self.substitute, line)
414 else:
415 for line in lines:
416 yield line
417
418 def substitute(self, match):
419 return str(self.context[match.group(1)])
420
421 run = Main()
422 main = run.main
423
424 if __name__ == '__main__':
425 main(*sys.argv[1:])
Something went wrong with that request. Please try again.