Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use line buffering by awk in make import-sql #958

Closed
zstadler opened this issue Jul 30, 2020 · 0 comments · Fixed by #962
Closed

Use line buffering by awk in make import-sql #958

zstadler opened this issue Jul 30, 2020 · 0 comments · Fixed by #962

Comments

@zstadler
Copy link
Contributor

When running quickstart.sh, the output is redirected to a pipe.
When the output is not a terminal, awk uses block buffering.
As a result, the output from make import-sql is block-buffered and the progress is not clear, especially when dealing with a large data set or a less powerful machine.

For example, here is a snapshot of the output:

CREATE FUNCTION
Time: 1.935 ms
CREATE TRIGGER
Time: 2.219 ms
psql:/sql/parallel/water__wate

The last line is completed only when the next buffer is flushed:

CREATE FUNCTION
Time: 1.935 ms
CREATE TRIGGER
Time: 2.219 ms
psql:/sql/parallel/water__waterway.sql:525: NOTICE:  trigger "trigger_store" for relation "osm_waterway_linestring" does not exist, skipping

The issue can be resolved by replacing

import-sql: all start-db-nowait
	$(DOCKER_COMPOSE) run $(DC_OPTS) openmaptiles-tools sh -c 'pgwait && import-sql' | \
	  awk -v s=": WARNING:" '$$0~s{print; print "\n*** WARNING detected, aborting"; exit(1)} 1'

with

import-sql: all start-db-nowait
	$(DOCKER_COMPOSE) run $(DC_OPTS) openmaptiles-tools sh -c 'pgwait && import-sql' | \
	  awk -v s=": WARNING:" '1{print; fflush()} $$0~s{print "\n*** WARNING detected, aborting"; exit(1)}'
zstadler added a commit to zstadler/openmaptiles that referenced this issue Aug 8, 2020
TomPohys pushed a commit that referenced this issue Aug 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant