More URL updates #1694
More URL updates #1694
Conversation
|
That's all for the moment. |
|
gmsl.html is synced from upstream cvs so changes of this file from this pull request will be lost when it is synced next time. Can you send the patch to the upstream, please? https://sourceforge.net/p/gmsl/_list/tickets |
|
Reverted those. Here's the diff for reference, it's only minor changes, so I think it's fine as it is. --- gmsl.html 2017-03-07 15:51:48.000000000 +0000
+++ gmsl-new.html 2017-03-05 11:29:35.000000000 +0000
@@ -10,8 +10,8 @@
string manipulation, integer arithmetic, associative arrays, stacks,
and debugging facilities. The GMSL is released under the BSD License.<br>
<br>
-<a href="http://sourceforge.net/projects/gmsl/">[Project Page]</a> <a href="http://sourceforge.net/project/showfiles.php?group_id=129887">[Download]</a>
-<a href="http://sourceforge.net/forum/forum.php?forum_id=443916">[Discussion
+<a href="https://sourceforge.net/projects/gmsl/">[Project Page]</a> <a href="https://sourceforge.net/projects/gmsl/files/">[Download]</a>
+<a href="https://sourceforge.net/p/gmsl/discussion/443916/">[Discussion
Forum]</a><br>
<h2>Using GMSL</h2>
The two files needed are <span style="font-family: monospace;">gmsl</span>
@@ -696,18 +696,18 @@
</table>
<span style="font-family: monospace;"></span><br>
<hr>
-Copyright (c) 2005-2014 <a href="http://www.jgc.org/">John Graham-Cumming</a>.<br>
+Copyright (c) 2005-2014 <a href="https://www.jgc.org/">John Graham-Cumming</a>.<br>
<hr style="width: 100%; height: 2px;">
<table style="width: 100%; text-align: left;" border="0" cellpadding="2" cellspacing="2">
<tbody>
<tr>
<td style="width: 50%;">John Graham-Cumming's work on this
-project was sponsored by <a href="http://www.electric-cloud.com/">Electric
+project was sponsored by <a href="https://electric-cloud.com/">Electric
Cloud, Inc</a>.<br>
- <a href="http://www.electric-cloud.com/"><img alt="" src="http://gmsl.sf.net/ec_logo.gif" style="border: 0px solid ; width: 223px; height: 47px;"></a><br>
+ <a href="https://electric-cloud.com/"><img alt="" src="http://gmsl.sf.net/ec_logo.gif" style="border: 0px solid ; width: 223px; height: 47px;"></a><br>
</td>
<td align="right">
- <p><a href="http://sourceforge.net/"><img src="http://sourceforge.net/sflogo.php?group_id=129887&type=1" alt="SourceForge.net Logo" border="0" height="31" width="88"></a></p>
+ <p><a href="https://sourceforge.net/"><img src="https://sourceforge.net/sflogo.php?group_id=129887&type=1" alt="SourceForge.net Logo" border="0" height="31" width="88"></a></p>
</td>
</tr>
</tbody> |
|
https://pdfgrep.sourceforge.io redirects to https://pdfgrep.org/ |
|
Updated to follow the redirects. |
Checking control sum is needed for security. Are you sure that the file armadillo-6.400.3.tar.gz was not compromised? This PR has too much "garbage" commits. You need squash commits: |
|
@pavelvat Restored the original checksum and updated the URL to point to the correct file. (I have zero information about this project or why they have multiple packages under the same version number.) |
|
I plan to squash commits once any potential problems are found and fixed. (GitHub now also has a "Squash and Merge" option at merge time, so it can be done automatically.) |
|
Squashed now. |
|
config.guess is also external as well as gmsl |
|
For the record: |
|
https://harfbuzz.sourceforge.io/ returns 404 |
|
Please rebase the PR to resolve conflicts. |
|
https://hci.iwr.uni-heidelberg.de/vigra also returns 404. |
|
Updated these. http://www.genivia.com/dev.html is redirected to https://www.genivia.com/dev.html, which works OK from here. |
|
I just noticed that all UPDATE: Tests finished without any problem. |
Package files are checked against checksums after downloading, so |
|
I think, we should re-add diff --git a/Makefile b/Makefile
index 6fb880e..a9aec5f 100644
--- a/Makefile
+++ b/Makefile
@@ -248,19 +248,19 @@ ESCAPE_PKG = \
BACKUP_DOWNLOAD = \
(echo "MXE Warning! Downloading $(1) from backup." >&2 && \
- ($(WGET) -O '$(PKG_DIR)/.tmp-$($(1)_FILE)' $(PKG_MIRROR)/`$(call ESCAPE_PKG,$(1))` || \
- $(WGET) -O '$(PKG_DIR)/.tmp-$($(1)_FILE)' $(PKG_CDN)/`$(call ESCAPE_PKG,$(1))` || \
- $(WGET) -O '$(PKG_DIR)/.tmp-$($(1)_FILE)' $(GITLAB_BACKUP)/`$(call ESCAPE_PKG,$(1))`_$($(1)_CHECKSUM)))
+ ($(WGET) --no-check-certificate -O '$(PKG_DIR)/.tmp-$($(1)_FILE)' $(PKG_MIRROR)/`$(call ESCAPE_PKG,$(1))` || \
+ $(WGET) --no-check-certificate -O '$(PKG_DIR)/.tmp-$($(1)_FILE)' $(PKG_CDN)/`$(call ESCAPE_PKG,$(1))` || \
+ $(WGET) --no-check-certificate -O '$(PKG_DIR)/.tmp-$($(1)_FILE)' $(GITLAB_BACKUP)/`$(call ESCAPE_PKG,$(1))`_$($(1)_CHECKSUM)))
DOWNLOAD_PKG_ARCHIVE = \
$(if $($(1)_SOURCE_TREE),\
true\
$(else),\
mkdir -p '$(PKG_DIR)' && ( \
- $(WGET) -T 30 -t 3 -O '$(PKG_DIR)/.tmp-$($(1)_FILE)' '$($(1)_URL)' \
+ $(WGET) --no-check-certificate -T 30 -t 3 -O '$(PKG_DIR)/.tmp-$($(1)_FILE)' '$($(1)_URL)' \
$(if $($(1)_URL_2), \
|| (echo "MXE Warning! Downloading $(1) from second URL." >&2 && \
- $(WGET) -T 30 -t 3 -O '$(PKG_DIR)/.tmp-$($(1)_FILE)' '$($(1)_URL_2)')) \
+ $(WGET) --no-check-certificate -T 30 -t 3 -O '$(PKG_DIR)/.tmp-$($(1)_FILE)' '$($(1)_URL_2)')) \
$(if $(MXE_NO_BACKUP_DL),, \
|| $(BACKUP_DOWNLOAD)) \
) && cat '$(PKG_DIR)/.tmp-$($(1)_FILE)' \This doesn't add security risk, because downloaded files are verified against checksums. The advantage of having this option is more reliability on old systems lacking a root cert or some new crypto algorithm used by a webserver. Adding it to |
|
If there are systems that don't have correct CA bundles, I'd argue this option should be enabled explicitly there and only there, but it should operate securely by default. This option won't help with missing crypto algo, as it serves to disable server certificate verification (one pillar of TLS), but leaves the transfer protocol encrypted. The checksum is good, but it's only good as long as you verify it against a value you received from a verified source. If you download a package via HTTP or |
|
Notice that by using |
The checksums are in the MXE source in all Try to change |
|
I understand that. The question is how these checksums are formed/calculated? If they are calculated after getting the package via an insecure link, it can be spoofed/compromised. But, it's not only the checksums that are a concern here. When using |
|
|
That is a good point! In majority of cases they are calculated with Ideally we should download securely in context of |
|
This sounds better, though I'd personally only enable I've committed a patch here to add support for envvar named |
|
I moved fix of armadillo URL to #1701 not to mix it with hundreds of URL updates to http. |
|
Would you mind changing the protocols to Ops, a bit more than that, here's the full armadillo patch: diff --git a/src/armadillo.mk b/src/armadillo.mk
index 359055fe..33ea58d3 100644
--- a/src/armadillo.mk
+++ b/src/armadillo.mk
@@ -1,14 +1,14 @@
# This file is part of MXE. See LICENSE.md for licensing information.
PKG := armadillo
-$(PKG)_WEBSITE := http://arma.sourceforge.net/
+$(PKG)_WEBSITE := https://arma.sourceforge.io/
$(PKG)_DESCR := Armadillo C++ linear algebra library
$(PKG)_IGNORE :=
$(PKG)_VERSION := 6.400.3
$(PKG)_CHECKSUM := 019ce442a1bcad4c5da0bc01ee35333c9a0783ec6a58237ae1e774da68cd4f2f
$(PKG)_SUBDIR := $(PKG)-$($(PKG)_VERSION)
$(PKG)_FILE := $(PKG)-$($(PKG)_VERSION).tar.gz
-$(PKG)_URL := http://$(SOURCEFORGE_MIRROR)/project/arma/$($(PKG)_FILE)
+$(PKG)_URL := https://$(SOURCEFORGE_MIRROR)/project/arma/$($(PKG)_FILE).orig
$(PKG)_DEPS := gcc blas lapack
define $(PKG)_UPDATE |
Can outdated systems fall back to downloading from mirrors (or we can add a native build of wget)? |
|
There is no security risk of even using plain HTTP downloads as long as results are verified with checksums. Debian APT works this way.
This is not a backwards-compatible change. Consider an automated system with outdated software. Adding such an option will require a human to figure out about it and to putting
I really like the idea! If we remove
IMHO, we can avoid this. |
|
I merged #1701 |
|
Debian APT works differently. There, there is digitally signed/verified file ( |
|
As for adding insecurity for Probably it'd be good idea to refresh certificates when starting the Travis builds via It may also help to migrate to Ubuntu 14.04 LTS (Trusty) (from current 12.02), which is offered since last autumn as a beta. This needs |
I wanted to rethink this proposal, but then I realized that it would introduce chicken-and-egg problem. |
+1
Some upstream sites are down sometimes. E.g. sourceforge returns 404 from time to time. |
|
Backup downloads are a good thing. I only argue that they should be downloaded via a secure channel. |
Regular downloads of packages are verified by checksums, so --no-check-certificate doesn't compromise the build system, but the checksums themselves are often updated with update-checksum-% which in turn calls the regular package download mechanism, so there is a possibility of downloading and sealing a poisoned file. On the one hand, old systems may still rely on --no-check-certificate, so it is not nice to completely disable it for regular downloads. However keeping this option enabled for backup servers only is enough to support such systems because of the fallback mechanism. On the other hand, download from a backup doesn't make sense while updating a package, because the package is definetely not in the backup yet. So --no-check-certificate is now enabled only for backup servers and backup servers are disabled while updating packages. See mxe#1694 (comment)
|
Backup downloads seem useless while updating packages because packages are added to backups after updating them in the MXE source. I moved disabling of --no-check-certificate while updating packages to #1702
Given #1702 we can add an option to disable --no-check-certificate entirely to support this use-case. However this seems to be different from securing the build process. Even without |
|
I understand that using HTTPS won't protect from all imaginable threats but it's the current best we can do with ubiquitous and standard technology. Reflecting to your MITM example, WoSign has been since suspended/put on probation for actions like these. I'd argue that we should use available best practices as far as the HTTPS protocol and current CA system allows. It will still protect from a lot of obvious attacks and give better privacy than without. |
|
Can you define the threat model, please? In #1702 the attack on privacy is possible only if an attacker blocks access to all sites used for downloading packages except those three backup servers. To do that, she already needs knowledge that you use MXE. Given she knows you use MXE, what new information can she receive from MiTM attack on connection between you and backup servers? |
|
A MiTM attacker can see and alter all my traffic cleartext. Including the exact packages and package versions downloaded. It can be revealing information. It can likely be used with something else combined. I'd reverse the question though: If there are some known outdated systems that have problems handling current live certificates, is it optimal to disable default security for everyone just to address that, or wouldn't it be better to fix said outdated systems (if possible), or enable such workaround on those systems only? |
|
In a test done using Trusty, the number of fallbacks to backup (or 2nd URL in a few cases) went down from 77 to 9. PR: #1703 |
|
Most of those 9 were valid problems, where the URL is broken even when testing it using different method. Committed a few related homepage fixes and bumped a version in this PR: #1704 |
Given the attacker knows about MXE (which she is likely to know about because she blocked all non-backup servers and kept backup servers unblocked) this information corresponds to MXE commit hash on one-to-one basis. I.e. the attacker knows your MXE commit.
Again, if she already know about MXE, she can deploy her attack against all recent MXE commits. A user is likely to use latest head, so if there is an attack benefiting from knowledge of exact MXE commit, it can be successfully used against majority of users. Can you provide an example of such attack, btw?
If #1702 hurts security of build for everyone, then it should not be used (and |
|
#1702 does two different things: It restores security for non-backup downloads, which is perfect and very much welcome. And it removes security from backup downloads, which is however not following good security practices. |
Currently that PR provides plain HTTP downloads (with unprotected HTTPS redirects in case of GitLab) for backup downloads. Downloaded files are verified against checksums. I don't see how it can be insecure in terms of content of downloaded files. If this does not follow security practices, then APT used in Debian also do not follow good security practices.
I thought on this argument a lot and I think that hiding a list of software used by the system can be classified as security through obscurity which is known to be far from good security practice. Moreover using of strong HTTPS doesn't hide this list from a smart attacker, so it is similar to other well known security-through-obscurity approaches that seem to work but in practice provide very small security if any. I think, we should have clear goals and non-goals in terms of security. Is hiding a list of used packages from ISP one of our goals? |
|
This pull request looks good to me. |
|
Many thanks! |
Regular downloads of packages are verified by checksums, so --no-check-certificate doesn't compromise the build system, but the checksums themselves are often updated with update-checksum-% which in turn calls the regular package download mechanism, so there is a possibility of downloading and sealing a poisoned file. On the one hand, old systems may still rely on --no-check-certificate, so it is not nice to completely disable it for regular downloads. However keeping this option enabled for backup servers only is enough to support such systems because of the fallback mechanism. On the other hand, download from a backup doesn't make sense while updating a package, because the package is definetely not in the backup yet. So --no-check-certificate is now enabled only for backup servers and backup servers are disabled while updating packages. See mxe#1694 (comment)
|
Thank you very much for the merge! On hiding traffic: It's primarily a privacy matter, not a security one. Not revealing more information than necessary is usually good practice. As for 'security through obscurity' it's also not as black and white as it may seem; see some valid points in this article: https://danielmiessler.com/study/security-by-obscurity/. |
Regular downloads of packages are verified by checksums, so --no-check-certificate doesn't compromise the build system, but the checksums themselves are often updated with update-checksum-% which in turn calls the regular package download mechanism, so there is a possibility of downloading and sealing a poisoned file. On the one hand, old systems may still rely on --no-check-certificate, so it is not nice to completely disable it for regular downloads. However keeping this option enabled for backup servers only is enough to support such systems because of the fallback mechanism. On the other hand, download from a backup doesn't make sense while updating a package, because the package is definetely not in the backup yet. So --no-check-certificate is now enabled only for backup servers and backup servers are disabled while updating packages. See #1694 (comment)
No description provided.