Keep website hidden service mirror up-to-date

[andrewdavidwong]

We have a Tor hidden service mirror, thanks to @Nukama. However, it has fallen woefully behind the updates to the canonical website. (As of this writing, most pages on it are inaccessible.) Is there any way to automate the task of keeping it up-to-date? If not, perhaps we shouldn't list it in the footer of the site (at least not without some sort of disclaimer).

For reference, here's the URI: http://qubesosmamapaxpa.onion/

[marmarek]

@Nukama, I can setup some hook to call when the repository is updated.
On your side this would require exposing some address (either clearnet
or tor service), which will call _utils/update-submodules (or similar). The
script would take care of git tag verification. That exposed hook would
not need to parse any parameters, just fetch updates from github repo.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

@Nukama, I can setup some hook to call when the repository is updated.
On your side this would require exposing some address (either clearnet
or tor service), which will call _utils/update-submodules (or similar). The
script would take care of git tag verification. That exposed hook would
not need to parse any parameters, just fetch updates from github repo.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[andrewdavidwong]

We probably shouldn't wait for a reply from @Nukama on this.

[andrewdavidwong]

We probably shouldn't wait for a reply from @Nukama on this.

[qubenix]

Maybe host the hidden service on the same server as the clearnet site? This will dramatically simplify this process and make an easier workflow for the dev team (if @Nukama will give the private key of course).

[qubenix]

Maybe host the hidden service on the same server as the clearnet site? This will dramatically simplify this process and make an easier workflow for the dev team (if @Nukama will give the private key of course).

[marmarek]

Currently clearnet site is hosted on github pages, so not exactly possible to host hidden service from there. But cloning github pages (jekyll) is very simple.

[marmarek]

Currently clearnet site is hosted on github pages, so not exactly possible to host hidden service from there. But cloning github pages (jekyll) is very simple.

[fortasse]

I can probably host this along side Whonix's hidden services. However, if you want to keep the same .onion domain, you'll have to get me the private key somehow. Otherwise, we can have another subdomain on kkkkkkkkkk63ava6.onion.

If it's just a matter of pulling a git repo every so often, that should be pretty easy to do.

[fortasse]

I can probably host this along side Whonix's hidden services. However, if you want to keep the same .onion domain, you'll have to get me the private key somehow. Otherwise, we can have another subdomain on kkkkkkkkkk63ava6.onion.

If it's just a matter of pulling a git repo every so often, that should be pretty easy to do.

[marmarek]

I can probably host this along side Whonix's hidden services. However, if you want to keep the same .onion domain, you'll have to get me the private key somehow. Otherwise, we can have another subdomain on kkkkkkkkkk63ava6.onion.

Unfortunately, we don't have private key for the old address :/
@andrewdavidwong do you have any preference as for the address - from the list I've sent you?

If it's just a matter of pulling a git repo every so often, that should be pretty easy to do.

I can also setup github hook for the repository - it will sent a simple POST request to configured URL. Possible problem here: it must be clearnet URL, onion addresses are not supported. If really needed, I could proxy it myself, but not sure if worth the effort.

[marmarek]

I can probably host this along side Whonix's hidden services. However, if you want to keep the same .onion domain, you'll have to get me the private key somehow. Otherwise, we can have another subdomain on kkkkkkkkkk63ava6.onion.

Unfortunately, we don't have private key for the old address :/
@andrewdavidwong do you have any preference as for the address - from the list I've sent you?

If it's just a matter of pulling a git repo every so often, that should be pretty easy to do.

I can also setup github hook for the repository - it will sent a simple POST request to configured URL. Possible problem here: it must be clearnet URL, onion addresses are not supported. If really needed, I could proxy it myself, but not sure if worth the effort.

[fortasse]

Our hidden service and clearnet sites are hosted on the same machine, so having the hook tie back to a clearnet address should be perfectly fine. I don't have too much experience with webhooks, however.

[fortasse]

Our hidden service and clearnet sites are hosted on the same machine, so having the hook tie back to a clearnet address should be perfectly fine. I don't have too much experience with webhooks, however.

[marmarek]

It's easy: https://github.com/marmarek/signature-checker/blob/master/github-webhook - expose as CGI.
If unique script could be used for this particular purpose - probably you don't need to parse POST data at all. Some documentation: https://developer.github.com/v3/repos/hooks/#receiving-webhooks
And actual example event for git push: https://developer.github.com/v3/activity/events/types/#pushevent

[marmarek]

It's easy: https://github.com/marmarek/signature-checker/blob/master/github-webhook - expose as CGI.
If unique script could be used for this particular purpose - probably you don't need to parse POST data at all. Some documentation: https://developer.github.com/v3/repos/hooks/#receiving-webhooks
And actual example event for git push: https://developer.github.com/v3/activity/events/types/#pushevent

[andrewdavidwong]

Unfortunately, we don't have private key for the old address :/
@andrewdavidwong do you have any preference as for the address - from the list I've sent you?

No preference. There weren't any outstanding ones. Here were the slightly recognizable addresses:

qubesoseggc4nxja.onion
qubesos4rrrrz6n4.onion
qubesoseaaeexn7o.onion
qubesosxuuxxnkzb.onion
qubesosangieqolx.onion
qubesosnojpylaqe.onion
qubesosxydie7non.onion
qubesosgtgxgxdot.onion
qubesosprrwr4z3h.onion
qubesoscirect6nz.onion

IMO, using a subdomain on kkkkkkkkkk63ava6.onion would also be fine.

[andrewdavidwong]

Unfortunately, we don't have private key for the old address :/
@andrewdavidwong do you have any preference as for the address - from the list I've sent you?

No preference. There weren't any outstanding ones. Here were the slightly recognizable addresses:

qubesoseggc4nxja.onion
qubesos4rrrrz6n4.onion
qubesoseaaeexn7o.onion
qubesosxuuxxnkzb.onion
qubesosangieqolx.onion
qubesosnojpylaqe.onion
qubesosxydie7non.onion
qubesosgtgxgxdot.onion
qubesosprrwr4z3h.onion
qubesoscirect6nz.onion

IMO, using a subdomain on kkkkkkkkkk63ava6.onion would also be fine.

[marmarek]

I think I like qubesos4rrrrz6n4.onion from this list (for me the easiest one to remember). @mfc?

[marmarek]

I think I like qubesos4rrrrz6n4.onion from this list (for me the easiest one to remember). @mfc?

[mfc]

yep qubesos4rrrrz6n4.onion sounds good.

[mfc]

yep qubesos4rrrrz6n4.onion sounds good.

[fortasse]

Cool. If you want to email me the private key, my contact info is here: https://www.whonix.org/wiki/contact#For_Website_Issues

[fortasse]

Cool. If you want to email me the private key, my contact info is here: https://www.whonix.org/wiki/contact#For_Website_Issues

[fortasse]

I have qubesos4rrrrz6n4.onion up and running. @marmarek: I sent you an email to get the webhook / automation side part set up as well.

[fortasse]

I have qubesos4rrrrz6n4.onion up and running. @marmarek: I sent you an email to get the webhook / automation side part set up as well.

[marmarek]

Ok, webhook enabled. @andrewdavidwong do you have some pending edits to test if automation works?

[marmarek]

Ok, webhook enabled. @andrewdavidwong do you have some pending edits to test if automation works?

[fortasse]

For full disclosure, the reason I emailed the webhook link directly is to prevent it from getting crawled by search engines and regenerating the site unnecessarily. If you email me directly and provide a half-decent reason for needing the endpoint, I'll happily provide it.

[fortasse]

For full disclosure, the reason I emailed the webhook link directly is to prevent it from getting crawled by search engines and regenerating the site unnecessarily. If you email me directly and provide a half-decent reason for needing the endpoint, I'll happily provide it.

[fortasse]

If there are no editorial changes to make, adding a comment to robots.txt would likely work.

[fortasse]

If there are no editorial changes to make, adding a comment to robots.txt would likely work.

[marmarek]

If you want to prevent spurious calls, github provide a mechanism for verifying if request really come from github - X-Hub-Signature header: https://developer.github.com/v3/repos/hooks/#webhook-headers
But just for anti-crawlers, you can just check some simple property of received data - like if that is a POST request (very rare for crawlers), if X-GitHub-Event header is present etc.

Anyway, just keeping URL non-public is also some solution.

[marmarek]

If you want to prevent spurious calls, github provide a mechanism for verifying if request really come from github - X-Hub-Signature header: https://developer.github.com/v3/repos/hooks/#webhook-headers
But just for anti-crawlers, you can just check some simple property of received data - like if that is a POST request (very rare for crawlers), if X-GitHub-Event header is present etc.

Anyway, just keeping URL non-public is also some solution.

[fortasse]

Aah, good things to point out. Once we make sure everything's working I may go back and improve the actual webhook receiver, it's a bit inelegant at the moment.

[fortasse]

Aah, good things to point out. Once we make sure everything's working I may go back and improve the actual webhook receiver, it's a bit inelegant at the moment.

[marmarek]

If there are no editorial changes to make, adding a comment to robots.txt would likely work.

I'd like to check also submodules update. Just pushed some minor edit.

[marmarek]

If there are no editorial changes to make, adding a comment to robots.txt would likely work.

I'd like to check also submodules update. Just pushed some minor edit.

[fortasse]

Doesn't look like it worked. @marmarek, would you mind hopping in #whonix on OFTC to help me get this debugged?

[fortasse]

Doesn't look like it worked. @marmarek, would you mind hopping in #whonix on OFTC to help me get this debugged?

[fortasse]

I've redone the webhook to essentially use a modified version of the post-receive hook in the readme here. That should deal with the submodules properly.

[fortasse]

I've redone the webhook to essentially use a modified version of the post-receive hook in the readme here. That should deal with the submodules properly.

[fortasse]

@marmarek joined me in IRC, and I believe everything is working now. I will make a note to clean up my implementation and potentially add it to _utils in the main repo.

[fortasse]

@marmarek joined me in IRC, and I believe everything is working now. I will make a note to clean up my implementation and potentially add it to _utils in the main repo.

[andrewdavidwong]

Added a footer link to the new hidden service. Thanks, @fortasse!

[andrewdavidwong]

Added a footer link to the new hidden service. Thanks, @fortasse!

[anonmos1]

Can we have https://www.qubes-os.org/qubes-issues/ working on the onion too?

[anonmos1]

Can we have https://www.qubes-os.org/qubes-issues/ working on the onion too?

[andrewdavidwong]

Can we have https://www.qubes-os.org/qubes-issues/ working on the onion too?

That one's hosted from a different repo. The original intention was for it not to share the qubes-os.org domain at all, but because of the way the website is set up, there's an automatic redirect from https://qubesos.github.io/qubes-issues/ to https://www.qubes-os.org/qubes-issues/. Not sure how much trouble it'd be.

CC @fortasse @marmarek

[andrewdavidwong]

Can we have https://www.qubes-os.org/qubes-issues/ working on the onion too?

That one's hosted from a different repo. The original intention was for it not to share the qubes-os.org domain at all, but because of the way the website is set up, there's an automatic redirect from https://qubesos.github.io/qubes-issues/ to https://www.qubes-os.org/qubes-issues/. Not sure how much trouble it'd be.

CC @fortasse @marmarek

[fortasse]

If we're cool with it just living at qubesos4rrrrz6n4.onion/qubes-issues/ it shouldn't be too much trouble. We can use the existing webhook, I can just update the script to update both repos / folders.

[fortasse]

If we're cool with it just living at qubesos4rrrrz6n4.onion/qubes-issues/ it shouldn't be too much trouble. We can use the existing webhook, I can just update the script to update both repos / folders.

[andrewdavidwong]

@fortasse: Sounds good to me. Thanks!

[andrewdavidwong]

@fortasse: Sounds good to me. Thanks!

[fortasse]

@andrewdavidwong: http://qubesos4rrrrz6n4.onion/qubes-issues/ is up. Do you want to confirm the webhook works? It's exactly the same as the old webhook, just with a few more steps to pull and build https://github.com/QubesOS/qubes-issues/tree/gh-pages. I assume it will work fine, but more testing is never a bad thing. :)

[fortasse]

@andrewdavidwong: http://qubesos4rrrrz6n4.onion/qubes-issues/ is up. Do you want to confirm the webhook works? It's exactly the same as the old webhook, just with a few more steps to pull and build https://github.com/QubesOS/qubes-issues/tree/gh-pages. I assume it will work fine, but more testing is never a bad thing. :)

[andrewdavidwong]

@fortasse Ok, I've pushed several commits to the gh-pages branch. They're not reflected on the onion version yet, but it might still be too soon.

[andrewdavidwong]

@fortasse Ok, I've pushed several commits to the gh-pages branch. They're not reflected on the onion version yet, but it might still be too soon.

[fortasse]

This might be a stupid question, but did you configure this repo to hit the same webhook as the main website repo? The last hit to the webhook was at 27/Apr/2017:17:49:44 +0200 according to logs.

[fortasse]

This might be a stupid question, but did you configure this repo to hit the same webhook as the main website repo? The last hit to the webhook was at 27/Apr/2017:17:49:44 +0200 according to logs.

[andrewdavidwong]

No, I've never done that before. (I think @marmarek did it for the main website repo.) How would I do that? Feel free to point me to documentation.

[andrewdavidwong]

No, I've never done that before. (I think @marmarek did it for the main website repo.) How would I do that? Feel free to point me to documentation.

[fortasse]

https://developer.github.com/webhooks/creating/

TLDR: To set up a repository webhook on GitHub, head over to the Settings page of your repository, and click on Webhooks & services.

You should clone whatever settings are used for the QubesOS/qubes-doc repo, I'm not sure what the exact settings are on the github-side.

[fortasse]

https://developer.github.com/webhooks/creating/

TLDR: To set up a repository webhook on GitHub, head over to the Settings page of your repository, and click on Webhooks & services.

You should clone whatever settings are used for the QubesOS/qubes-doc repo, I'm not sure what the exact settings are on the github-side.

[andrewdavidwong]

Oh, I don't have admin access to either repo, so I can't even view the Settings page. I think @marmarek will have to do it.

[andrewdavidwong]

Oh, I don't have admin access to either repo, so I can't even view the Settings page. I think @marmarek will have to do it.

[fortasse]

Alright. I just hit the webhook manually, can you confirm the changes are reflected on qubesos4rrrrz6n4.onion/qubes-issues/ as expected? If hitting it manually works, it should have no issues once the webhook config gets updated on Github.

[fortasse]

Alright. I just hit the webhook manually, can you confirm the changes are reflected on qubesos4rrrrz6n4.onion/qubes-issues/ as expected? If hitting it manually works, it should have no issues once the webhook config gets updated on Github.

[fortasse]

I'm 95% sure it should work when the repo config is updated. Let me know if it still doesn't work once the webhook is configured on the github-side.

[fortasse]

I'm 95% sure it should work when the repo config is updated. Let me know if it still doesn't work once the webhook is configured on the github-side.

[andrewdavidwong]

Yep, looks good. Thanks, @fortasse!

[andrewdavidwong]

Yep, looks good. Thanks, @fortasse!

[marmarek]

Webhook added. But there is some problem - it failed with "Service timeout", and also the last one for main repo failed at 2017-04-27 17:47:51 (the last one).

[marmarek]

Webhook added. But there is some problem - it failed with "Service timeout", and also the last one for main repo failed at 2017-04-27 17:47:51 (the last one).

[fortasse]

Marek Marczykowski-Górecki:

Webhook added. But there is some problem - it failed with "Service timeout", and also the last one for main repo failed at 2017-04-27 17:47:51 (the last one).

That might be an issue on the github side, and it happens occasionally? I see 200s for the webhook from the github hookshot in the logs, so I think it works? I do see the HTTP 499 at 28/Apr/2017:09:59:50 +0200, but everything after that is 200s. I believe github's behavior is to retry webhooks that fail a given number of times.

[fortasse]

Marek Marczykowski-Górecki:

Webhook added. But there is some problem - it failed with "Service timeout", and also the last one for main repo failed at 2017-04-27 17:47:51 (the last one).

That might be an issue on the github side, and it happens occasionally? I see 200s for the webhook from the github hookshot in the logs, so I think it works? I do see the HTTP 499 at 28/Apr/2017:09:59:50 +0200, but everything after that is 200s. I believe github's behavior is to retry webhooks that fail a given number of times.

[anonmos1]

onion doesn't have the latest announcement

[anonmos1]

onion doesn't have the latest announcement

[fortasse]

anonmos1:

onion doesn't have the latest announcement

I don't see any POSTs to the webhook from Github. Did someone change the config? The last log line is from 03/May/2017:15:10:01 +0200. I'll update it manually in the meantime.

[fortasse]

anonmos1:

onion doesn't have the latest announcement

I don't see any POSTs to the webhook from Github. Did someone change the config? The last log line is from 03/May/2017:15:10:01 +0200. I'll update it manually in the meantime.

[marmarek]

According to github - this is also when the last POST was sent (successfully). And it is after adding an announcement. So, something is wrong with the update script...

[marmarek]

According to github - this is also when the last POST was sent (successfully). And it is after adding an announcement. So, something is wrong with the update script...

[fortasse]

Shoot, you're right - I had a typo in one of the directories. It should be fixed now. My bad, thanks for pointing it out @anonmos1!

[fortasse]

Shoot, you're right - I had a typo in one of the directories. It should be fixed now. My bad, thanks for pointing it out @anonmos1!

[anonmos1]

Unfortunately, the onion is not up to date

[anonmos1]

Unfortunately, the onion is not up to date

[fortasse]

Just updated it manually, and made a couple tweaks to the update
process. Hopefully that will prevent this from happening again.

Sorry for the trouble, and thanks for letting me know.

(Also, it looks like reply-by-email is broken on Github. If this double posts, sorry.)

[fortasse]

Just updated it manually, and made a couple tweaks to the update
process. Hopefully that will prevent this from happening again.

Sorry for the trouble, and thanks for letting me know.

(Also, it looks like reply-by-email is broken on Github. If this double posts, sorry.)

[andrewdavidwong]

Thanks for taking care of it, @fortasse!

[andrewdavidwong]

Thanks for taking care of it, @fortasse!

[anonmos1]

I still cannot see the latest announcements

[anonmos1]

I still cannot see the latest announcements

[fortasse]

Hm, apparently the --incremental flag for Jekyll doesn't work like I thought. I've removed it, and confirmed the updates are now there. It seems like I was pulling the files from Git properly, but not generating the site completely...

Can you confirm @anonmos1?

[fortasse]

Hm, apparently the --incremental flag for Jekyll doesn't work like I thought. I've removed it, and confirmed the updates are now there. It seems like I was pulling the files from Git properly, but not generating the site completely...

Can you confirm @anonmos1?

[anonmos1]

Yes, all seems to be good

[anonmos1]

Yes, all seems to be good

[anonmos1]

Last 4 announcements are missing

[anonmos1]

Last 4 announcements are missing

[anonmos1]

@fortasse

[anonmos1]

@fortasse

[fortasse]

Jekyll decided to be picky. I've got it sorted now, thanks for bringing it to my attention @anonmos1, and sorry for the delay.

[fortasse]

Jekyll decided to be picky. I've got it sorted now, thanks for bringing it to my attention @anonmos1, and sorry for the delay.

[andrewdavidwong]

FYI @fortasse:

On 2018-02-11 18:57, Gaijin wrote:

The Qubes OS v3 and v2 onion services sites don't appear to have been
updated since January when this announcement was made. Shouldn't they
show the same content as the normal Qubes site?

For example I don't see the news of the 4.0-rc4 release, or recent
changes to the Docs.

[andrewdavidwong]

FYI @fortasse:

On 2018-02-11 18:57, Gaijin wrote:

The Qubes OS v3 and v2 onion services sites don't appear to have been
updated since January when this announcement was made. Shouldn't they
show the same content as the normal Qubes site?

For example I don't see the news of the 4.0-rc4 release, or recent
changes to the Docs.

[andrewdavidwong]

Bump/poke @fortasse:

On 2018-03-12 02:26, Gaijin wrote:

On 2018-02-12 01:47, Andrew David Wong wrote:

On 2018-02-11 18:57, Gaijin wrote:

The Qubes OS v3 and v2 onion services sites don't appear to have been
updated since January when this announcement was made. Shouldn't they
show the same content as the normal Qubes site?

For example I don't see the news of the 4.0-rc4 release, or recent
changes to the Docs.

Thanks for the report! Tracking:

#1352 (comment)

FYI: The Onion sites are still not syncing with the Qubes site. They're
still stuck in January 2018...

CC: @adrelanos

[andrewdavidwong]

Bump/poke @fortasse:

On 2018-03-12 02:26, Gaijin wrote:

On 2018-02-12 01:47, Andrew David Wong wrote:

On 2018-02-11 18:57, Gaijin wrote:

The Qubes OS v3 and v2 onion services sites don't appear to have been
updated since January when this announcement was made. Shouldn't they
show the same content as the normal Qubes site?

For example I don't see the news of the 4.0-rc4 release, or recent
changes to the Docs.

Thanks for the report! Tracking:

#1352 (comment)

FYI: The Onion sites are still not syncing with the Qubes site. They're
still stuck in January 2018...

CC: @adrelanos

[fortasse]

Shoot, thanks again for the heads up - I've adjusted the script to address cases where the gems need upgrading, and to email me when it doesn't exit cleanly. Thanks for your patience!

[fortasse]

Shoot, thanks again for the heads up - I've adjusted the script to address cases where the gems need upgrading, and to email me when it doesn't exit cleanly. Thanks for your patience!

[andrewdavidwong]

Thanks!

[andrewdavidwong]

Thanks!

[marmarek]

Looks like tor onion service is down.

[marmarek]

Looks like tor onion service is down.

[andrewdavidwong]

Yes. Not sure if @marmarek is referring to just the website, but the yum and deb onion repos also appear to be down, @fortasse.

[andrewdavidwong]

Yes. Not sure if @marmarek is referring to just the website, but the yum and deb onion repos also appear to be down, @fortasse.

[adrelanos]

This is unlikely being fixed anytime soon.

https://forums.whonix.org/t/whonix-org-webmaster-fortasse-stepped-down

[adrelanos]

This is unlikely being fixed anytime soon.

https://forums.whonix.org/t/whonix-org-webmaster-fortasse-stepped-down

[andrewdavidwong]

That's very sad news. Thank you for letting us know.

In the meantime, I suggest that users who want to update over Tor change their repo definitions to use the non-onion domains (i.e., yum.qubes-os.org and deb.qubes-os.org).

[andrewdavidwong]

That's very sad news. Thank you for letting us know.

In the meantime, I suggest that users who want to update over Tor change their repo definitions to use the non-onion domains (i.e., yum.qubes-os.org and deb.qubes-os.org).