Allow publication of documents with `HTTPS` current/previous version links. #282

mikewest · 2016-04-25T09:01:55Z

As a tiny, tiny step towards #145, it would be lovely to allow publications of documents whose "This version:" and "Previous version:" links are HTTPS as opposed to HTTP.

mikewest · 2016-04-25T09:02:09Z

/cc @plehegar @deniak @tripu

mikewest · 2016-04-27T06:11:21Z

Ping? ~2 weeks ago, @plehegar suggested the following in #pub:

"""
the way I think things at the moment is two possibilities:
- either we require https
- or we simply forbid http and let authors decide whether to use absolute or relative
"""

Implementing either of those suggestions would preclude the current behavior of outright rejecting links to https://www.w3.org/. Would it be possible to at least remove that restriction while we discuss whether or not to mandate some other behavior? :)

plehegar · 2016-04-27T16:15:38Z

We're making progress but still have some issues: we're looking at implications in tr.rdf and the w3c api. I think we can do 1 or 2 as I proposed on irc but before we can do it, we have to figure how to represent that in the data, not just the document, otherwise we'll break tools like specref.

plehegar · 2016-04-27T17:04:23Z

Let's imagine that HTML 5.1 gets published in June, July and August 2016
and that we do the switch for https-only on July 1, 2016. The current idea
is to obtain the following:

HTML document published in /TR

June 2016 document:

 This version:
   http://www.w3.org/TR/2016/WD-html51-20160602/
 Previous Version:
   http://www.w3.org/TR/2016/WD-html51-20160502/
 Latest version:
   http://www.w3.org/TR/html51/

July 2016 document:

 This version:
   https://www.w3.org/TR/2016/WD-html51-20160717/
 Previous version:
   https://www.w3.org/TR/2016/WD-html51-20160602/ (or http://...)
 Latest version:
   https://www.w3.org/TR/html51/

August 2016 document:

 This version:
   https://www.w3.org/TR/2016/WD-html51-20160817/
 Previous Version:
   https://www.w3.org/TR/2016/WD-html51-20160717/
 Latest version:
   https://www.w3.org/TR/html51/

tr.rdf representation of those documents:

   <!-- W3C server redirects from http to https, so let's represent this
     equivalence here -->
   <rdf:Description rdf:about="http://www.w3.org/TR/html51/">
     <sameWorkAs rdf:resource="https://www.w3.org/TR/html51/"/>
   </rdf:Description>

   <WD rdf:about="http://www.w3.org/TR/2016/WD-html51-20160602/">
     <dc:date>2017-06-02</dc:date>
     <dc:title>HTML 5.1</dc:title>
     <doc:obsoletes 
rdf:resource="http://www.w3.org/TR/2016/WD-html51-20160502/"/>
     <doc:versionOf rdf:resource="http://www.w3.org/TR/html51/"/>
     <patentRules 
rdf:resource="https://www.w3.org/Consortium/Patent-Policy-20040205/"/>
   </WD>

   <WD rdf:about="https://www.w3.org/TR/2016/WD-html51-20160717/">
     <dc:date>2017-07-17</dc:date>
     <dc:title>HTML 5.1</dc:title>
     <!-- we force http:// for all *dated* URI published before the 
switch -->
     <doc:obsoletes 
rdf:resource="http://www.w3.org/TR/2016/WD-html51-20160602/"/>
     <doc:versionOf rdf:resource="https://www.w3.org/TR/html51/"/>
     <patentRules 
rdf:resource="https://www.w3.org/Consortium/Patent-Policy-20040205/"/>
   </WD>

   <WD rdf:about="https://www.w3.org/TR/2016/WD-html51-20160817/">
     <dc:date>2017-08-17</dc:date>
     <dc:title>HTML 5.1</dc:title>
     <doc:obsoletes 
rdf:resource="https://www.w3.org/TR/2016/WD-html51-20160717/"/>
     <doc:versionOf rdf:resource="https://www.w3.org/TR/html51/"/>
     <patentRules 
rdf:resource="http://www.w3.org/Consortium/Patent-Policy-20040205/"/>
   </WD>

w3c api:

Before the switch:

     "_embedded": {
         "version-history": [
             {
                 "status": "Working Draft",
                 "uri": 
"http:\/\/www.w3.org\/TR\/2016\/WD-html51-20160602\/",
                 "date": "2016-06-02",
                 "informative": false,
                 "title": "HTML 5.1",
                 "shortlink": "http:\/\/www.w3.org\/TR\/html51\/",
                 "editor-draft": "https:\/\/w3c.github.io\/html\/",
                 "process-rules": 
"http:\/\/www.w3.org\/2015\/Process-20150901\/",
             }, ... ]

After the switch: we can't represent the equivalence between
http://www.w3.org/TR/html51/ and https://www.w3.org/TR/html51/ so we
make the choice of using https. This could break things but the API
isn't largely used yet so we might get away with it. Most critical would
be that we might break pheme (and consequently ash-nazg...).

     "_embedded": {
         "version-history": [
             {
                 "status": "Working Draft",
                 "uri": 
"http:\/\/www.w3.org\/TR\/2016\/WD-html51-20160602\/",
                 "date": "2016-06-02",
                 "title": "HTML 5.1",
                 "shortlink": "https:\/\/www.w3.org\/TR\/html51\/",
                 "process-rules": 
"https:\/\/www.w3.org\/2015\/Process-20150901\/",
             },
             {
                 "status": "Working Draft",
                 "uri": 
"https:\/\/www.w3.org\/TR\/2016\/WD-html51-20160717\/",
                 "date": "2016-07-17",
                 "title": "HTML 5.1",
                 "shortlink": "https:\/\/www.w3.org\/TR\/html51\/",
                 "process-rules": 
"https:\/\/www.w3.org\/2015\/Process-20150901\/",
             },
             {
                 "status": "Working Draft",
                 "uri": 
"https:\/\/www.w3.org\/TR\/2016\/WD-html51-20160817\/",
                 "date": "2016-08-17",
                 "title": "HTML 5.1",
                 "shortlink": "https:\/\/www.w3.org\/TR\/html51\/",
                 "process-rules": 
"https:\/\/www.w3.org\/2015\/Process-20150901\/",
             }, ... ]

plehegar · 2016-05-04T14:48:06Z

I circulated the proposal internally and didn't get pushback. So, my next step is to send the proposal to spec-prod and chairs. I ought to get to this this week. If you don't see anything from him, feel free to poke me in the eye until I do it.

plehegar · 2016-05-09T16:19:57Z

Proposal was sent to spec-prod:
https://lists.w3.org/Archives/Public/spec-prod/2016AprJun/0024.html

marcoscaceres · 2016-05-24T03:18:32Z

ReSpec has a PR in waiting to support this.

mikewest · 2016-06-02T04:35:19Z

@plehegar: Can you update the status here? I see that there's been some discussion on that thread, but the conclusion isn't at all clear to me.

This blocked @estark37 from publishing a draft earlier in the week, and I can only imagine that it's biting other folks as well, given that HTTPS is fairly accepted as best practice.

marcoscaceres · 2016-06-02T05:08:20Z

I'm also waiting on an Ok to merge in ReSpec from @plehegar: I'm worried that if I merge Echinda and PubRules will cry because of the new URLs.

plehegar · 2016-06-02T16:53:56Z

I need to catch up with Tobie. As far as we know, he is the long pole at the moment in making the switch. If we can get specref.org to digest the https switch, then we're good to go. If we can't, they doing the switch means that we would break specref, and breaks respec and bikeshed subsequently. See also
https://lists.w3.org/Archives/Public/spec-prod/2016AprJun/0028.html

mikewest · 2016-06-02T17:02:00Z

@tobie: What can we do to help out? Has anything happened since your comments on that thread a month ago?

plehegar · 2016-06-02T17:06:33Z

To be fair with @tobie, I didn't follow up well enough with him. Part is because I got confused what he was asking for. The only way to create data before the switch is to create fake data as far as I know and he was pushing back on this idea.

tobie · 2016-06-02T17:16:42Z

My initial request was for a grace period during which both https and http would coexist so as to give me enough time to figure out what impact the change had.

If that's impossible to provide, the other option is to run a frozen version of Specref until I figure out the consequences of these changes.

That later solution does imply, however, that Specref will be out of date for a while. And given I work on this on my free time, and will be OoO for part of the summer, I can't really commit to a date at which I can do the transition.

Frankly, specref code isn't the highest quality software available, and while the transition could literally take less than 2 hours, it might also be much more painful than that; I had to backpedal out of a PR recently because it broke things in weird and unexpected ways. I wouldn't be surprised finding lots of similar issues here.

marcoscaceres · 2016-06-02T17:22:33Z

On 3 Jun 2016, at 3:16 AM, Tobie Langel notifications@github.com wrote:

My initial request was for a grace period during which both https and http would coexist so as to give me enough time to figure out what impact the change had.

If that's impossible to provide, the other option is to run a frozen version of Specref until I figure out the consequences of these changes.

That later solution does imply, however, that Specref will be out of date for a while. And given I work on this on my free time, and will be OoO for part of the summer, I can't really commit to a date at which I can do the transition.

Frankly, specref code isn't the highest quality software available, and while the transition could literally take less than 2 hours, it might also be much more painful than that; I had to backpedal out of a PR recently because it broke things in weird and unexpected ways. I wouldn't be surprised finding lots of similar issues here.

If it would be of help, I'm available next week to assist in whatever way I can. I don't fully understand the problem at this point, but happy to set time aside to see if we can come up with a strategy or try some things.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

tobie · 2016-06-02T17:26:40Z

I don't fully understand the problem at this point,

Me neither. That's precisely the problem.

plehegar · 2016-06-02T17:59:58Z

which part of the problem is obscure? I tried to be as precise as possible in how the change would be implemented. I'm happy to clarify things if it helps.

tobie · 2016-06-02T18:30:00Z

which part of the problem is obscure?

How Specref will react to these changes is what's obscure.

plehegar · 2016-06-14T19:37:03Z

so, do we know what to do here? As far as I can tell, W3C can either switch to https and break specref.org (and bikeshed and respec as a consequence), or we should delay the switch until a solution for specref.org is found...

tobie · 2016-06-14T19:43:27Z

Well, as I said, if the preferred solution of publishing the new (https-aware) rdf file in parallel with the old one isn't possible, the next best solution is for me to freeze specref updates until I figure out how to handle the new rdf format.

tobie · 2016-06-14T19:46:06Z

What I need for this 2nd solution is a schedule + ideally an on-call person on your side so we can fix possible rdf bugs quickly.

plehegar · 2016-06-14T20:03:00Z

[Tobie and I caught up on irc]

Here is the solution to do the switch sooner rather than later:
1- provide a separate tr rdf file, like tr-http.rdf or something. It won't be authoritative but will allow specref.org to keep working until Tobie finds time to upgrade it. The sooner W3C can provide this, the better.
2- on July 1st, do the https switch in tr.rdf as documented previously.

if we can't do the solution above, we're probably looking at moving the switch to August 1st instead.

tobie · 2016-06-17T14:43:18Z

Any news? /cc @deniak

deniak · 2016-06-17T17:14:22Z

@tobie I will provide the new rdf beginning of next week. I talked to @plehegar and we are targetting Aug 1st for the switch.

tobie · 2016-06-17T18:42:11Z

Sounds good. By new RDF, do you mean the one containing https links or the current one on a new URL?

deniak · 2016-06-20T12:42:44Z

@tobie you can find the experimental rdf at https://www.w3.org/2002/01/tr-automation/tr-https.rdf. For testing purpose, I only moved the specs published after May 31 to https.

deniak · 2016-06-20T13:36:13Z

Moving earlier discussions on GH:

On 06/20/2016 04:25 PM, Tobie Langel wrote:

Thanks a bunch. See inline comments.

On Mon, 20 Jun 2016, at 12:54, Denis Ah-Kang wrote:

Tobie, Philippe,

As agreed [1], here's the experimental rdf:
https://www.w3.org/2002/01/tr-automation/tr-https.rdf

For testing purposes, I tweaked the code so all the documents published
after May 31 are under https (eg.
https://www.w3.org/TR/2016/WD-audio-output-20160601/).
A few things you need to pay attention to:

the previous versions are under http as they were previously

Is this going to change? I.e., are all links going to be https when you
flip the bit?

No, after the switch, the WG will first have to republish their specs
with:

the "This Version" under https

the "Previous Version" under http (because it was already published
this way)

Then, the next versions of the spec will have both links under https.

This comes from a request from the semweb guys who want to be able to
dereference the URIs and they see http://www.w3.org and
https://www.w3.org as 2 different URIs.

published with http. However, the next versions will have the
doc:Obsoletes under https.

the latest versions are under https

the WGs homepage links are the ones from
http://www.w3.org/Member/Mail. Some WGs still have their homepage under
http. If the WG updates
its homepage to https, it'll replicated onto the rdf.

patentRules, errata, translations should be under https

starting from line 21043, you will see the sameWorkAs between
the http and https versions of each spec.

That rdf is here only for testing purposes, that's why I'm not
sharing that link publicly (on GitHub) as I will drop it later

This is such a weird way of doing things. Why not do this in the open
and state this link will change or just 301 it to the main file post
transition period.

You have a point. I was just being too lazy and wanted to avoid adding
one more redirect. OK to share publicly.

but
feel free to share with the people who could be interested in it.
It's also not complete because we don't have any specs having a previous
version with https and unless we have fake data, it's not possible to
represent that use case yet.
Let me know if there's any problem.

I'll probably have more questions once I start working on this.

Thanks again,

--tobie

tobie · 2016-06-20T15:00:13Z

Thanks for moving the conversation on a public forum. Much appreciated.

the previous versions are under http as they were previously
Is this going to change? I.e., are all links going to be https when you
flip the bit?

No, after the switch, the WG will first have to republish their specs
with:

the "This Version" under https

the "Previous Version" under http (because it was already published this way)
Then, the next versions of the spec will have both links under https.

This comes from a request from the semweb guys who want to be able to
dereference the URIs and they see http://www.w3.org and
https://www.w3.org as 2 different URIs.

OK. I'm not sure I'm following.

Couple of questions:

Will https://xxx -301-> http://xxx?
Will previous specs be available both over http and https?
Will newer specs be available only over https and not over http?

tobie · 2016-06-20T15:13:50Z

For context, here, I take a pragmatic approach to this issue, not a (technically pure) semweb one.

As far as I'm concerned, the content on both http and https versions of the spec should be the same (with the possible exception of internal links which might have matching protocols for absolute URLs). So Specref internals will consider it as such (that's what I have to fix, basically, to make sure that URL-based comparisons are protocol-agnostic).

Ultimately, I'd like Specref to expose https URLs for all W3C specs regardless of when said specs will have been published.

I feel like this publish-date-based role-out of https is strange and potentially prone to creating issues over a substantial amount of time. So I have two follow-up questions:

Are http://www.w3.org/2002/01/tr-automation/tr.rdf and https://www.w3.org/2002/01/tr-automation/tr.rdf going to serve the same content or is the latter going to have all links be https?
Is the JSON API moving towards all https or going through this same, publish-date-based system?

deniak · 2016-06-20T15:30:34Z

To answer your questions, all the specs under /TR are and will remain available on http and https. This is the case since our recent HSTS support. You will be redirected only if your browser supports HSTS/CSP.

Now, the problem is how to represent the data and make everyone happy. We cannot simply update all the URIs to https even if that's what the user actually sees if he's using a recent browser. Take a look at that blog post @philarcher1 wrote to explain why we can't just update the scheme.

tobie · 2016-06-20T15:39:40Z

I'm kind of confused.

@philarcher1's post you link above explicitly says W3C is treating resources on www.w3.org as identical regardless of their protocol:

Firstly, is the community agreed that if two URIs differ only in the scheme (http://, https:// and perhaps whatever comes in future) then they identify the same resource? We believe that this can only be asserted by the domain owner. In the specific case of http://www.w3.org/* we do make that assertion.

So I guess I don't understand why you're not switching all of the URLs within the APIs to https. (Note I'm not suggesting you actually modify the specs themselves.)

tobie · 2016-06-20T15:47:51Z

To answer your questions, all the specs under /TR are and will remain available on http and https. This is the case since our recent HSTS support.

Note this link requires a member account.

deniak · 2016-06-20T16:18:11Z

The blog post suggests to keep using http:

In short, keep writing “http:” and trust that the infrastructure will quietly switch over to TLS (https) whenever both client and server can handle it. Meanwhile, let’s try to get SemWeb software to be doing TLS+UIR+HSTS and be as secure as modern browsers.

On the other hand, some editors are pushing towards https (topic of that issue) so we need to find a way to make both worlds happy.

deniak · 2016-06-20T16:21:04Z

To answer your questions, all the specs under /TR are and will remain available on http and https. This is the case since our recent HSTS support.

Note this link requires a member account.

Oups, sorry. You can find the public post there: https://www.w3.org/blog/news/archives/5263

tobie · 2016-06-20T16:31:28Z

On the other hand, some editors are pushing towards https (topic of that issue) so we need to find a way to make both worlds happy.

So one one hand, you have people arguing to use http and have the server/browser combo upgrade to https if they can, and on the other, you have people pushing for https everywhere. I get how that's an unfortunate position to be in. But I'm not sure how your solution to let https trickle down is going to make both worlds happy (I'm pretty sure it's going to piss off everyone instead 😃 ).

tobie · 2016-06-20T17:20:19Z

Just rolled-out a brute force solution to Specref (tobie/specref#286). Let's see if it works or if I have to roll everything back.

If it does, you can consider the Specref blocker resolved.

tobie · 2016-06-21T09:55:25Z

OK, Specref seems to be working properly so far serving only https links for W3C specs, so we should pretty much be good on the Specref side at this point.

marcoscaceres · 2016-06-21T09:56:03Z

Woot woot!

philarcher · 2016-06-21T10:05:44Z

From my POV, the key thing is that the original http URIs still work.
As long as that's true, we should be OK. It doesn't prevent making new
links to https://...

The underlying issue is persistence of identifiers. No doubt this
conversation will be repeated in future when https is replaced by
something else, so today's solution shouldn't apply specifically to
today's technology.

Thanks for taking this seriously,

Phil.

On 21/06/2016 10:56, Marcos Cáceres wrote:

Woot woot!

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#282 (comment)

Phil Archer
W3C Data Activity Lead
http://www.w3.org/2013/data/

http://philarcher.org
+44 (0)7887 767755
@philarcher1

mikewest · 2016-06-21T10:19:40Z

Thank you @tobie!

@deniak, @plehegar, et al: When you remove the requirement that "This version" and "Previous Version" links be insecure, could you also allow HTTPS for the following:

http://www.w3.org/Consortium/Legal/ipr-notice#Copyright, http://www.w3.org/, http://www.w3.org/Consortium/Legal/ipr-notice#Legal_Disclaimer, http://www.w3.org/Consortium/Legal/ipr-notice#W3C_Trademarks, http://www.w3.org/Consortium/Legal/copyright-documents in the copyright boilerplate (rule at https://www.w3.org/pubrules/doc/rules?profile=WD#copyright)
http://www.w3.org/Consortium/Patent-Policy-20040205/#def-essential, http://www.w3.org/Consortium/Patent-Policy-20040205/, and http://www.w3.org/Consortium/Patent-Policy-20040205/#sec-Disclosure in the patent policy boilerplate (rule at https://www.w3.org/pubrules/doc/rules?profile=WD#patPolReq)
http://www.w3.org/2015/Process-20150901/ (rule at https://www.w3.org/pubrules/doc/rulesprofile=WD#whichProcess)
The Working Group link (rule at https://www.w3.org/pubrules/doc/rules?profile=WD#WGLink)

I can also file separate bugs for those, if that would be helpful. :)

deniak · 2016-06-21T10:53:49Z

http://www.w3.org/Consortium/Legal/ipr-notice#Copyright, http://www.w3.org/, http://www.w3.org/Consortium/Legal/ipr-notice#Legal_Disclaimer, http://www.w3.org/Consortium/Legal/ipr-notice#W3C_Trademarks, http://www.w3.org/Consortium/Legal/copyright-documents in the copyright boilerplate (rule at https://www.w3.org/pubrules/doc/rules?profile=WD#copyright)

http://www.w3.org/Consortium/Patent-Policy-20040205/#def-essential, http://www.w3.org/Consortium/Patent-Policy-20040205/, and http://www.w3.org/Consortium/Patent-Policy-20040205/#sec-Disclosure in the patent policy boilerplate (rule at https://www.w3.org/pubrules/doc/rules?profile=WD#patPolReq)

http://www.w3.org/2015/Process-20150901/ (rule at https://www.w3.org/pubrules/doc/rulesprofile=WD#whichProcess)

For the 3 points above, this is inline with the proposal. We will even make https a requirement.

The Working Group link (rule at https://www.w3.org/pubrules/doc/rules?profile=WD#WGLink)

Regarding the WG homepage link, it's a bit tricky. Specberus checks that link against what's listed on https://www.w3.org/Member/Mail. As you can see, some WGs still have their homepage under http.
Short term solution is to update the rule and look for the homepage under both http and https.
In the future, it would be better to update all the WG homepages to https.

deniak · 2016-08-01T09:19:26Z

Starting from Aug 1, 2016, specberus requires https links.

mikewest · 2016-08-01T09:21:28Z

Thank you. :)

mikewest added a commit to w3c/webappsec-csp that referenced this issue Apr 25, 2016

Ugh. w3c/echidna#282

cb28a6c

plehegar self-assigned this Apr 27, 2016

travisleithead mentioned this issue May 19, 2016

Update all TR links to https w3c/html#399

Closed

deniak closed this as completed Aug 1, 2016

ryandel8834 added a commit to ryandel8834/WebAppSec-CSP that referenced this issue Aug 13, 2022

Ugh. w3c/echidna#282

46bbed9

Allow publication of documents with HTTPS current/previous version links. #282

Allow publication of documents with HTTPS current/previous version links. #282

Comments

mikewest commented Apr 25, 2016

mikewest commented Apr 25, 2016

mikewest commented Apr 27, 2016

plehegar commented Apr 27, 2016

plehegar commented Apr 27, 2016 • edited by deniak Loading

plehegar commented May 4, 2016 • edited Loading

plehegar commented May 9, 2016

marcoscaceres commented May 24, 2016

mikewest commented Jun 2, 2016

marcoscaceres commented Jun 2, 2016

plehegar commented Jun 2, 2016

mikewest commented Jun 2, 2016

plehegar commented Jun 2, 2016

tobie commented Jun 2, 2016

marcoscaceres commented Jun 2, 2016

tobie commented Jun 2, 2016

plehegar commented Jun 2, 2016

tobie commented Jun 2, 2016

plehegar commented Jun 14, 2016

tobie commented Jun 14, 2016

tobie commented Jun 14, 2016

plehegar commented Jun 14, 2016 • edited Loading

tobie commented Jun 17, 2016

deniak commented Jun 17, 2016

tobie commented Jun 17, 2016

deniak commented Jun 20, 2016

deniak commented Jun 20, 2016

tobie commented Jun 20, 2016 • edited Loading

tobie commented Jun 20, 2016

deniak commented Jun 20, 2016

tobie commented Jun 20, 2016 • edited Loading

tobie commented Jun 20, 2016

deniak commented Jun 20, 2016

deniak commented Jun 20, 2016

tobie commented Jun 20, 2016

tobie commented Jun 20, 2016 • edited Loading

tobie commented Jun 21, 2016

marcoscaceres commented Jun 21, 2016

philarcher commented Jun 21, 2016

mikewest commented Jun 21, 2016

deniak commented Jun 21, 2016

deniak commented Aug 1, 2016

mikewest commented Aug 1, 2016

Allow publication of documents with `HTTPS` current/previous version links. #282

Allow publication of documents with `HTTPS` current/previous version links. #282

plehegar commented Apr 27, 2016 •

edited by deniak

Loading

plehegar commented May 4, 2016 •

edited

Loading

plehegar commented Jun 14, 2016 •

edited

Loading

tobie commented Jun 20, 2016 •

edited

Loading

tobie commented Jun 20, 2016 •

edited

Loading

tobie commented Jun 20, 2016 •

edited

Loading