Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ref ticket #794 awk overflow #796

Merged
merged 7 commits into from Jul 16, 2017
Merged

Ref ticket #794 awk overflow #796

merged 7 commits into from Jul 16, 2017

Conversation

KD4Z
Copy link
Collaborator

@KD4Z KD4Z commented Jul 15, 2017

refactored brute for file concatenation to avoid 64K overflow in awk when executing combine.awk in stripped.csv target of the DB make.

Copy link
Collaborator

@sijskes sijskes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please review awk logic before replacing with cat.

possible fix for original problem would be to change $1 into string. in combine.awk

Add duplicate filtering to stripped.csv processing
@KD4Z
Copy link
Collaborator Author

KD4Z commented Jul 15, 2017

@sijskes Simon, I saw your first comment regarding the de-duping, and added pipe with !seen[] before I saw your second comment regarding combine.awk. I've updated the PR with !seen[] approach which works well in in my own post-build processing of the contact data. For your review.

@sijskes
Copy link
Collaborator

sijskes commented Jul 15, 2017

What you have done, is basically added a cat step. Better fix the combine.awk instead. The '!seen' fix u used suffers from the same problem as combine.awk (same functionality, same logic). The fix is converting the implicit integer into a string. This could be applied to combine.awk. Something like "x"$1 or ""$1 will convert to string. The $^ you have removed, is also a proper Makefile construct.

KD4Z added 2 commits July 16, 2017 11:23
However this line works just as well, and avoids free() overflow.   cat
$^ | awk -F',' '!seen[$$1]++' | $(ICONV) | sort -t ',' -k1g,1 | awk -F,
-f strip.awk >stripped.csv
…'t important

Avoids pointer overflow of free() in awk
@sijskes sijskes merged commit 1688e7e into travisgoodspeed:master Jul 16, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants