Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add shell script version #9

Closed
wants to merge 3 commits into from
+417 −0
Closed

add shell script version #9

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ef635b2
add shell script version
JayBrown Oct 3, 2017
1d249ee
cli v1.1
JayBrown Oct 7, 2017
d2229b2
swap for // fix
JayBrown Oct 7, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Next
add shell script version 
IPFS public gateway checker: shell script version
  • Loading branch information
@JayBrown
JayBrown committed Oct 3, 2017
commit ef635b24ffc7970b98f00f18d182c2f3fc62c3cb
36 shell-script/public-gateway-checker.sh
View file
Open in desktop
@@ -0,0 +1,36 @@
#!/bin/bash
Copy link
Member

@victorb victorb Oct 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be /usr/bin/env bash here instead to ensure portability.

Copy link
Contributor Author

@JayBrown JayBrown Oct 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK


# IPFS public gateway checker v1.0 (shell script version)
# original repository: https://github.com/ipfs/public-gateway-checker
# this script added by Joss Brown: https://github.com/JayBrown

JSON_URL="https://raw.githubusercontent.com/ipfs/public-gateway-checker/master/gateways.json"
Copy link
Member

@koalalorenzo koalalorenzo Oct 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it better to hardcode the list?

Copy link
Member

@victorb victorb Oct 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to have the list in one place instead of multiple ones though.

Copy link
Contributor Author

@JayBrown JayBrown Oct 3, 2017 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could (in a later version!) add some extra functionality. The script would then create an invisible file in $HOME, e.g. .ipfg (for "interplanetary file gateways"), which it will read at every run, and diff it against the canonical list on GitHub. If it's empty, nothing happens, i.e. no diff, just default run. If there is content, it will diff: ignore doubles, don't ignore additional URLs, then run. There could also be an extra --local option, and then the script would only read gateway URLs from the local file. For now, though, grabbing it from a canonical list on github.com/ipfs is preferable.

TESTHASH="Qmaisz6NMhDB51cCvNWa1GMS7LU1pAxdF4Ld6Ft9kZEP2a"

echo "Public IPFS Gateways"

GATEWAYS=$(curl --silent "$JSON_URL" | sed -e '$ d' -e '1,1d' -e 's/\"//g')
if [[ $GATEWAYS == "" ]] ; then
echo "Error: no gateways found."
exit
fi

GWNUM=$(echo "$GATEWAYS" | wc -l | xargs)

while read -r GATEWAY
do

GATEWAY=$(echo "$GATEWAY" | xargs)
TESTURL=$(echo "$GATEWAY" | sed -e "s-:hash,-$TESTHASH-" -e "s-:hash-$TESTHASH-")
DOMAIN=$(echo "$GATEWAY" | awk -F/ '{print $1"//"$3}')

if [[ $(curl --silent "$TESTURL") == "Hello from IPFS Gateway Checker" ]] ; then
Copy link
Member

@koalalorenzo koalalorenzo Oct 3, 2017 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about using the option -L to follow redirects?
That fixes #4

Copy link
Member

@victorb victorb Oct 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, would better to fix the URLs to be correct from the beginning instead, but be more future-proof to follow redirect so I agree.

Copy link
Contributor Author

@JayBrown JayBrown Oct 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use case: you run the script and awk -F"Online:" '{print $2}' to get the domains, then you won't have the redirect domain, but only the domain of the original request. If you want to do something with the gateway domain, e.g. automate a file upload if it's writable or simply cache an IPFS object via cURL, then running the script without an -L option would just tell you that something isn't right with the gateway URL. This would in turn benefit the canonical list.

But in this case, too, we could add a redirect option -r. Then the script would just need to print the final URL, not the initial one.

Copy link
Contributor Author

@JayBrown JayBrown Oct 7, 2017 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@victorbjelkholm @koalalorenzo

You can ignore the original pull request. I have a new cli here. EDIT: seems to have been auto-added to the pull request. Neat. :)

I haven't yet had the time to test it thoroughly, but preliminary runs seem to go well.

Changes:

renamed to ipfg
shebang with env bash
on 301-redirects etc. there's now a secondary curl with follow-redirect if the first curl returns an offline
invisible directory: $HOME/.ipfg
local backup of remote gateway list incl. update option -s
local gateway list with user-specified gateway urls
various options to add and remove gateway urls to/from the local list
options to check only remote gateways, only local gateways, or all gateways
option to manually check gateways with specified urls as args
options to only list (without checking) gateway urls (all, remote, local), either as domain or as raw url
miscellaneous options for: help page, print version, open GitHub repo

echo -e "✅ Online:\t$DOMAIN"
else
echo -e "❌ Offline:\t$DOMAIN"
fi

done < <(echo "$GATEWAYS")

echo "Done: $GWNUM gateways checked."
exit