Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update-check: download each url once #6395

Merged
merged 1 commit into from
Jun 12, 2019

Conversation

Chocimier
Copy link
Member

Speed up for packages with multiple downloads from same location or having homepage and distfiles at gitlab or github.

fi
curl -H 'Accept: text/html,application/xhtml+xml,application/xml,text/plain,application/rss+xml' -A "xbps-src-update-check/$XBPS_SRC_VERSION" --max-time 10 -Lsk "$url" |
grep -Po -i "$rx"
fetchedurls="$fetchedurls\n$url"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure that works:

$ a="Hello"
$ a="$a\nWorld"
$ echo "$a"
Hello\nWorld

There's a bashism for that:

$ a="Hello"
$ a="$a"$'\n'"World"
$ echo "$a"
Hello
World

But not sure if we want that. Opinions?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happens to work because variable is used with printf, not echo, but it will break on % in url.

What about literal newline and echo?

        fetchedurls="$fetchedurls
$url"
echo "$fetchedurls" | …

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

printf is actually a good idea:

$ a=$(print "%s\n%s" "$a" "$b")

@maxice8
Copy link
Contributor

maxice8 commented Mar 24, 2019

ping

@Chocimier
Copy link
Member Author

pong

@Chocimier Chocimier force-pushed the update-check-fetch-once branch 2 times, most recently from 732b66d to ea8112b Compare May 9, 2019 08:11
@pullmoll
Copy link
Member

pullmoll commented Jun 1, 2019

I just now stumbled across this PR. Since xbps-src expects bash I think you could simplify the code by using an associative array of URLs. Something along the lines:

local -A fetchedurls
…
        if [ "${fechedurls[$url]}" = "ok" ]; then
            if [ -n "$XBPS_UPDATE_CHECK_VERBOSE" ]; then
                echo "already fetched $url" 1>&2
            fi
            continue
        fi
        …
        fetchedurls[$url]="ok"

You could this way even store an error state (e.g. bad domain name, non-transient resolver error) and report that on a second try for the same $url.

@Chocimier
Copy link
Member Author

Agreed, associative array is good idea here. However as errors aren't printed on first try, printing on second try would be strange.

@Hoshpak Hoshpak merged commit b8039ef into void-linux:master Jun 12, 2019
@Chocimier Chocimier deleted the update-check-fetch-once branch July 26, 2019 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants