Skip to content

Commit

Permalink
parallel_textgen: run textgen in parallel
Browse files Browse the repository at this point in the history
textgen: instead of running ImageMagick yourself, output a Makefile rule
for each graphic. Dependencies are passed on the command line.

Makefile: Add rule for 'textgen.mk' which is built by running textgen.
graphics.stamp depends on textgen.mk and all the built graphics files.
Then, each built graphic depends on textgen.mk, ensuring it is made
first. textgen.mk itself depends on textgen config, font, and dehacked.
(Makefile itself also depends on textgen.mk implicitly via include)

This duplicates previous behaviour where changing textgen config remakes
all text graphics, but one Make rule for each, allowing parallel make
with -j instead of the script doing one at a time.

The built IWADs are byte-for-byte identical before and after this patch
series is applied (provided VERSION is set to the same thing of course)

One problem is 'make clean' rebuilds textgen.mk because Make thinks the
Makefile needs it, but then immediately deletes it. Not sure how to fix.
Use git clean -fdx instead, or just don't clean twice in succession.

----

This gives the following improvement in build time (-j1 as control,
built on 4 cores and a large tmpfs so disk speed isn't a factor)

(master)

    make -j1  4.83s user 6.77s system 93% cpu 12.444 total
    make -j1  4.74s user 6.72s system 93% cpu 12.267 total
    make -j1  4.72s user 6.68s system 92% cpu 12.292 total

    make -j4  5.72s user 6.77s system 109% cpu 11.414 total
    make -j4  5.39s user 6.85s system 107% cpu 11.419 total
    make -j4  5.66s user 6.79s system 109% cpu 11.383 total

parallel_textgen

    make -j1  4.57s user 6.66s system 92% cpu 12.185 total
    make -j1  4.73s user 6.57s system 93% cpu 12.152 total
    make -j1  4.60s user 6.72s system 93% cpu 12.152 total

    make -j4  5.62s user 7.72s system 262% cpu 5.084 total
    make -j4  5.82s user 7.76s system 262% cpu 5.165 total
    make -j4  5.79s user 7.73s system 261% cpu 5.161 total
  • Loading branch information
rjy committed Mar 14, 2017
1 parent 946db56 commit 3143339
Show file tree
Hide file tree
Showing 3 changed files with 22 additions and 15 deletions.
1 change: 1 addition & 0 deletions graphics/text/.gitignore
Expand Up @@ -10,6 +10,7 @@ helpoverlay.png
helpttl.gif
prboom.gif
graphics.stamp
textgen.mk
credit.gif
credtext*.png
freettl.gif
Expand Down
27 changes: 15 additions & 12 deletions graphics/text/Makefile
Expand Up @@ -52,24 +52,27 @@ TEXTGEN_GRAPHIC_LUMPS = \
TEXTGEN_GRAPHICS = $(TEXTGEN_GRAPHIC_LUMPS) \
helpttl.gif freettl.gif

all: $(TEXTGEN_GRAPHICS) help.gif credit.gif wikilrs.gif wivctms.gif
all: graphics.stamp help.gif credit.gif wikilrs.gif wivctms.gif

# textgen creates multiple outputs, which is awkward to express in
# make. Use a witness file (graphics.stamp) as suggested in the
# automake manual: "Handling Tools that Produce Many Outputs"

$(TEXTGEN_GRAPHICS): graphics.stamp
@if test -f $@; then :; else \
rm -f graphics.stamp; \
$(MAKE) data.stamp; \
fi
graphics.stamp: textgen.mk $(TEXTGEN_GRAPHICS)
cp $(TEXTGEN_GRAPHIC_LUMPS) ../
@touch $@

graphics.stamp: config.py fontchars ../../lumps/dehacked.lmp
@rm -f graphics.tmp
@touch graphics.tmp
./textgen
@mv graphics.tmp $@
# Construct a file of Make directives for each text graphic.
# Each graphic depends on the directives file, so we tell textgen
# what its name is, as a command line parameter. The directives
# file depends on textgen's input (config, font, dehacked).

textgen.mk: config.py fontchars ../../lumps/dehacked.lmp
./textgen $@ > $@

# Then, include the file of Make directives constructed above.

include textgen.mk

# Generate transparent image containing text for the HELP screen:
helptext.png: helpttl.gif
Expand Down Expand Up @@ -172,7 +175,7 @@ wivctms.gif:

clean:
rm -f $(TEXTGEN_GRAPHICS) helpbg.png help.gif helptext.png \
helptext2.png graphics.stamp *.pyc credtext.png \
helptext2.png graphics.stamp textgen.mk *.pyc credtext.png \
credtext2.png credit.gif dmwilv*.gif wikilrs.gif \
wivctms.gif wikilrs_horiz.png ../credit.gif ../help.gif \
../wikilrs.gif ../wivctms.gif
Expand Down
9 changes: 6 additions & 3 deletions graphics/text/textgen
Expand Up @@ -140,7 +140,7 @@ class Font(object):

def get_command(self, text, output_filename,
color=COLOR_WHITE, bgcolor=BACKGROUND_COLOR):
"""Get command line to render text to a file
"""Get command to render text to a file
with the given background color.
"""

Expand All @@ -159,9 +159,12 @@ class Font(object):
def generate_graphics(graphics, color=COLOR_WHITE, bgcolor=BACKGROUND_COLOR):
for name, text in sorted(graphics.items()):
print("# %s.gif: '%s'" % (name, text))
cmd = font.get_command(text, '%s.gif' % name,
# write a makefile fragment
target = '%s.gif' % name
cmd = font.get_command(text, target,
color=color, bgcolor=bgcolor)
invoke_command(cmd)
print("%s: %s" % (target, " ".join(sys.argv[1:])))
print("\t" + " ".join("'%s'" % i for i in cmd))

def generate_kerning_test():
pairs = []
Expand Down

1 comment on commit 3143339

@fragglet
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Late reply, but kudos for this - it's an elegant solution / improvement.

Please sign in to comment.