Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Work around slow path searches on MacOS #571

Merged
merged 1 commit into from
Jul 24, 2021
Merged

Conversation

fhunleth
Copy link
Member

If a directory under /home was in $PATH on MacOS, the path search for
sed was surprisingly slow. One an M1 MBP, adding /home/missing to
$PATH caused mix firmware to regularly take >40 seconds. Without that
in the $PATH, mix firmware almost always takes 10 or 11 seconds.

See the conversation at
https://elixirforum.com/t/mix-firmware-takes-very-long-time-to-run-how-to-debug/41201
for much, much worse experience on a different MBP. A huge thanks to
jpalley for figuring out that the slowdown was correlated with the
/home directory in the path.

This PR caches the absolute path to sed to avoid searching $PATH each
time. On an M1 MBP, this reduces the mix firmware time from 40 seconds
to about 11 seconds.

If a directory under /home was in $PATH on MacOS, the path search for
`sed` was surprisingly slow. One an M1 MBP, adding `/home/missing` to
$PATH caused `mix firmware` to regularly take >40 seconds. Without that
in the $PATH, `mix firmware` almost always takes 10 or 11 seconds.

See the conversation at
https://elixirforum.com/t/mix-firmware-takes-very-long-time-to-run-how-to-debug/41201
for much, much worse experience on a different MBP. A huge thanks to
jpalley for figuring out that the slowdown was correlated with the
`/home` directory in the path.

This PR caches the absolute path to sed to avoid searching `$PATH` each
time. On an M1 MBP, this reduces the `mix firmware` time from 40 seconds
to about 11 seconds.
@jpalley
Copy link

jpalley commented Jul 24, 2021

Nice job on the fast turnaround! (I'm the guy from the forum). FWIW:

  • I was doing a bit of experimenting, and I think you can substantially speed up mix firmware, even when "/home" is not an issue by, by not forking sed 3+ times for every file. From my testing, these forks are a significant amount of the overall run time. I don't feel I know enough to pursue this (given it's day 1 for me with nerves!) but I was thinking an approach where unsquash_to_psuedofile and find_to_psuedofile basically sort the files objects into different lists for files/dirs/symlinks/etc. and then call a sed or awk program once for each list would see meaningful speedups.

  • This is definitely a MacOS issue and nothing related to other stuff in this project. Stick something like this, "time echo "hi" | sed G" into a .sh file and run it with and without /home/ in the path. I see a 4x difference in "real" time to execute.

BTW - Loving nerves so far! Really slick.

@fhunleth
Copy link
Member Author

@jpalley Thanks. I totally agree. The amount of calls to sed is completely over the top. There are so many things with this script that this is now really bugging me that there may be a significant speedup to mix firmware out there.

@fhunleth fhunleth merged commit 1ee5816 into main Jul 24, 2021
@fhunleth fhunleth deleted the merge-squashfs-fix branch July 24, 2021 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants