move "D Reference" pages to /spec/ #896

aG0aep6G · 2015-02-09T13:27:20Z

Adding redirects so that the old URLs stay functional.

This is mostly moving pages, creating redirects, and updating links.
Non-bulk changes are:

dlang.org.ddoc: Changed GLINK2 to $(ROOT)/language/*. I'm assuming that
GLINK2 is for links to /language/ pages only.
doc.ddoc: A bulk change on its own, but distinct from the others.
language/language.ddoc: New file used with /language/ pages.
posix.mak

I used HTML meta refresh for the redirects, because we use it already (e.g. appendices.html).

That technique has at least one major flaw: it loses the fragment/anchor/hash-thingy. E.g., http://dlang.org/appendices.html#copyright redirects to http://dlang.org/dstyle.html, losing the #copyright part.

I'd rather use HTTP 301 redirects, but I don't know how to go about it.
Do we have .htaccess and mod_rewrite?
Do we have PHP and would MultiViews load foo.html.php when foo.html is requested?

As a next step, I'm going to add a search box option for the language reference, when Google has indexed the new locations.

CyberShadow · 2015-02-09T14:46:11Z

I'd rather use HTTP 301 redirects, but I don't know how to go about it.

I agree.

Do we have .htaccess and mod_rewrite?

No, but we should. I'll ask the system administrator to enable it.

Do we have PHP and would MultiViews load foo.html.php when foo.html is requested?

Yes: http://dlang.org/test/

But I don't like this solution. It requires two .php files to maintain compatibility with URLs with and without the .html extension (which are used in some places).

CyberShadow · 2015-02-09T15:03:26Z

We now have .htaccess redirects!

The existing redirects should be migrated to those. We should also see if we have links to redirect pages in existing pages, and either fix those to point to the redirect targets, or leave the HTML meta redirects as is, so the redirects keep working without Apache.

CyberShadow · 2015-02-09T15:12:46Z

Note: RewriteBase seems be required by the current server configuration for .htaccess redirects.

andralex · 2015-02-09T18:47:50Z

What is the motivation behind this? It looks like a lot of churn for a small benefit. Also, assuming this goes through, the redirect pages should be all generated with a macro.

CyberShadow · 2015-02-09T18:59:48Z

Ideally, the redirect pages should be all replaced with a few lines in .htaccess.

aG0aep6G · 2015-02-09T21:12:00Z

We now have .htaccess redirects!

Sweet. Added a commit to use them. Not 100% sure about RewriteBase.

What is the motivation behind this?

The goal is an indepently searchable language reference. As discussed (a little) here: http://forum.dlang.org/post/mailman.5287.1422186138.9932.digitalmars-d@puremagic.com

andralex · 2015-02-10T06:24:03Z

Related: http://forum.dlang.org/post/mbc83q$6dn$1@digitalmars.com

ghost · 2015-02-12T20:40:06Z

What is the motivation behind this?

I think this will enable us to search the language spec via google (currently you can't limit searching only to the spec because all pages are in root apparently).

CyberShadow · 2015-02-12T20:43:49Z

It looks like a lot of churn for a small benefit.

The benefit is small, but you might be overestimating the amount of "churn" as well. The redirects will make the move transparent to users.

ghost · 2015-02-13T15:11:40Z

So, what's the status on this? And is my statement true about making things searchable? It would really be a welcome change.

CyberShadow · 2015-02-13T15:16:42Z

Yes, it is true. I have attempted to construct a Google search that only searched the reference, and could not. Even if it is possible, it would require many search operators and filters.

CyberShadow · 2015-02-13T15:18:09Z

@andralex If you don't find this pull request acceptable as it is now, we can only change the location of the generated .html files, without moving the .dd files. This should avoid most of the "churn" you speak of.

ghost · 2015-02-13T19:21:21Z

It makes it even easier to edit these files if you know the language reference is in its own folder.

MartinNowak · 2015-03-14T19:24:49Z

Sounds like we could achieve the searching using annotation files.
https://developers.google.com/custom-search/docs/annotations

CyberShadow · 2015-03-14T19:34:04Z

Hmm, looks like those files still need to be uploaded manually every time they change, as opposed to auto-discovery like robots.txt or sitemap.xml (or even a parameter to the search widget)...

CyberShadow · 2015-03-14T19:35:38Z

I just realized that we can implement this with only .htaccess changes. Make /language/ URLs redirect to non-/language/ locations internally, and non-/language/ URLs redirect to /language/ URLs using a HTTP redirect.

CyberShadow · 2015-03-14T19:47:52Z

OK, not really, because of all the relative links to static resources, and because we can't rely on the .htaccess always being in effect (as it would prohibit testing offline).

MartinNowak · 2015-03-15T17:47:04Z

Hmm, looks like those files still need to be uploaded manually every time they change, as opposed to auto-discovery like robots.txt or sitemap.xml (or even a parameter to the search widget)...

Where is the problem, you need to upload the updated pages as well. It's just a matter of generating them automatically.

CyberShadow · 2015-03-15T17:50:17Z

You have to use a browser or write an API client, assuming one exists, and then there's the matter of Google account access. OTOH uploading dlang.org is one command.

MartinNowak · 2015-03-15T17:52:20Z

I had much less trouble to support dlang.org/spec/template.html than dlang.org/language/template.html, which avoids to repeat "language" in the URL and is shorter.

MartinNowak · 2015-03-15T17:53:16Z

as it would prohibit testing offline

We shouldn't break that.

aG0aep6G · 2015-03-16T15:54:26Z

Sounds like we could achieve the searching using annotation files.
https://developers.google.com/custom-search/docs/annotations

If I get this right, that would mean updating the annotation files when a spec page is added/moved. I'd prefer a more automatic solution.

I had much less trouble to support dlang.org/spec/template.html than dlang.org/language/template.html, which avoids to repeat "language" in the URL and is shorter.

/spec/ would be fine with me. If there's consensus, I'll change things over.

MartinNowak · 2015-04-25T16:14:03Z

We could reuse our .dpl_rewrite_map.txt file for the redirects. I think a separate search for the language specs is well worth the effort.

CyberShadow · 2015-10-08T04:34:29Z

So should we go forward with this?

andralex · 2015-11-21T19:35:02Z

I think we should, with the following ultimate purpose in mind: language reference should be accessible for specific versions of the language. For example, lexical is now http://dlang.org/lex.html. We should have http://dlang.org/reference/2.067/lex.html and so on for each version. http://dlang.org/lex.html redirects to the latest version.

That way people who work with a specific version of the language can see what the definition was at that release point.

CyberShadow · 2015-11-22T07:52:09Z

Not sure we want a redirect, in most cases the intention is to link to the latest version of a page rather than a permalink to whatever version is latest at the time, so I think having the latest version at one location would be better. There's also the issue with search engine indexing that I mentioned before in regards to versioned Phobos docs.

How would this be achieved? Forward-porting DDoc files to the current layout and macros, or archiving old HTML files as they were built? Either way, given that this task is non-trivial, and given that we distribute documentation in a variety of formats with the compiler, I doubt that this will add enough value for the effort.

CyberShadow · 2015-11-22T07:52:59Z

@aG0aep6G Anyway, if you want to update this we try an expedited merge. Last I looked I didn't see any technical issues with this PR.

aG0aep6G · 2015-11-22T16:21:47Z

Updated. Still going with the /language/ URLs. If you guys prefer /spec/ or /reference/, please agree on one.

Regarding versioned variants of the spec/reference: I think /lex.html should be a legacy redirect to language/lex.html (as with this PR), and /language/lex.html should be a copy/alias of the newest versioned URL (but not a redirect). That scheme could be done on top of this PR.

CyberShadow · 2015-11-23T07:50:50Z

make -f win32.mak chm gives:

Error: don't know how to make 'objc_interface.html'

Looks like the makefiles need to be updated due to #1129.

aG0aep6G · 2015-11-23T11:38:36Z

Fixed win32.mak. Apparently I had missed some more files than just the new objc_interface.html.

CyberShadow · 2015-11-23T12:06:43Z

Something is wrong.

The style, images etc. are broken on the moved pages in the doc autotester, however they look fine when I build them here on Windows.

Still, locally there are other issues with the generated docs, e.g. GLINK2 seems to be wrong from inside the language directory.

aG0aep6G · 2015-11-23T12:33:14Z

Fixed GLINK2. But I'm at a loss as to why the autotester doesn't pick up the root path. I tried a clean clone; no problems.

aG0aep6G · 2015-11-23T12:47:46Z

Figured it out. The order of rules in posix.mak matters. I think I had fallen in the same trap with the changelog subdir. Autotester displays properly now.

CyberShadow · 2015-11-23T13:34:44Z

Could you add a comment to the makefile describing the issue to warn future contributors?

CyberShadow · 2015-11-23T13:47:00Z

Even though there are the Apache redirects, we need to ensure this PR doesn't create broken links when not under Apache, for the CHM and offline HTML documentation. One example is the link to ../hash-map.html in arrays.dd. chmgen displays a list of broken links, but not their location, there is probably a better way to do this.

aG0aep6G · 2015-11-23T14:29:12Z

Could you add a comment to the makefile describing the issue to warn future contributors?

Done.

we need to ensure this PR doesn't create broken links when not under Apache, for the CHM and offline HTML documentation.

I used the linkchecker command line tool and found a couple links that still needed updating. Will have to update some phobos pages, too. linkchecker also found some unrelated broken links, but those are for another pull request.

CyberShadow · 2015-11-23T15:03:29Z

I see a few more broken links that look related to this PR:

Warning: Unknown page: attribute.html
Warning: Unknown page: contracts.html
Warning: Unknown page: cpp0x.html
Warning: Unknown page: expression.html
Warning: Unknown page: language/        contracts.html
Warning: Unknown page: language/phobos/core_attribute.html
Warning: Unknown page: language/phobos/object.html
Warning: Unknown page: property.html
Warning: Unknown page: type.html
Warning: Unknown page: version.html

aG0aep6G · 2015-11-23T19:53:07Z

Warning: Unknown page: cpp0x.html

cpp0x.html is not being built (not changed in this PR). So that's expected.

Warning: Unknown page: language/ contracts.html

Changed the only spot I could find that could end up with that.

Warning: Unknown page: language/phobos/core_attribute.html
Warning: Unknown page: language/phobos/object.html

Fixed.

Otherwise, I can't find these links, except from phobos. These are from the .chm file and that includes phobos, right?

Should I make phobos/druntime PRs so that we can test everything together?

CyberShadow · 2015-11-23T20:07:34Z

Otherwise, I can't find these links, except from phobos. These are from the .chm file and that includes phobos, right?

Yep.

Should I make phobos/druntime PRs so that we can test everything together?

Yes, and ideally they should be merged simultaneously too.

aG0aep6G · 2015-11-23T20:38:13Z

Yes, and ideally they should be merged simultaneously too.

Ok, that'll have to wait until tomorrow, though.

Maybe we can decide on a name for the directory until then. Ping @MartinNowak @andralex: Which one shall it be, /language/, /spec/, or /reference/?

andralex · 2015-11-23T20:41:38Z

I'm preapproving this as a step in a good direction. @CyberShadow please take over the merging process once your concerns are addressed.

FWIW I personally have a mild preference for /spec/ over /language/ but out of respect I'll leave it to the folks involved in the work.

Adding redirects so that the old URLs stay functional. Links to the moved pages are updated in a separate commit.

Assuming that dlang.org.ddoc's GLINK2 macro is for links to language reference pages only.

aG0aep6G · 2015-11-24T17:38:06Z

Changed to /spec/, squashed down to two commits, made companion PRs:

aG0aep6G · 2015-11-24T18:22:27Z

@CyberShadow: I just noticed your PR against my branch. After a little git adventure, I think I managed to include it properly.

CyberShadow · 2015-11-24T18:25:23Z

Thanks, will have a look again tomorrow :)

CyberShadow · 2015-11-25T15:41:03Z

Alright, Let's Do This.

move "D Reference" pages to /spec/

CyberShadow · 2015-12-01T15:28:00Z

I guess we can now add the search option in the dropdown?

aG0aep6G · 2015-12-01T15:36:57Z

I guess we can now add the search option in the dropdown?

I'm waiting for google to pick up the new locations. So far it seems to have only found some of the pages: https://www.google.de/search?q=site%3Adlang.org%2Fspec

aG0aep6G force-pushed the language-subdir branch 3 times, most recently from 9ee6aad to d940988 Compare November 22, 2015 15:46

aG0aep6G added 2 commits November 24, 2015 18:02

move "D Reference" pages to /spec/

3bbb0e8

Adding redirects so that the old URLs stay functional. Links to the moved pages are updated in a separate commit.

update links to "D Reference" pages

7aced8d

Assuming that dlang.org.ddoc's GLINK2 macro is for links to language reference pages only.

aG0aep6G force-pushed the language-subdir branch from 6920297 to 7aced8d Compare November 24, 2015 17:33

This was referenced Nov 24, 2015

update links to spec pages dlang/druntime#1440

Merged

update links to spec pages dlang/phobos#3830

Merged

chmgen: Update inclusion paths

a6a8015

aG0aep6G changed the title ~~move "D Reference" pages to /language/~~ move "D Reference" pages to /spec/ Nov 24, 2015

CyberShadow added a commit that referenced this pull request Nov 25, 2015

Merge pull request #896 from aG0aep6G/language-subdir

c1da655

move "D Reference" pages to /spec/

CyberShadow merged commit c1da655 into dlang:master Nov 25, 2015

CyberShadow mentioned this pull request Nov 25, 2015

function.dd: Table of Contents 2 #1151

Merged

Uh oh!

move "D Reference" pages to /spec/ #896

move "D Reference" pages to /spec/ #896

Uh oh!

Conversation

aG0aep6G commented Feb 9, 2015

Uh oh!

CyberShadow commented Feb 9, 2015

Uh oh!

CyberShadow commented Feb 9, 2015

Uh oh!

CyberShadow commented Feb 9, 2015

Uh oh!

andralex commented Feb 9, 2015

Uh oh!

CyberShadow commented Feb 9, 2015

Uh oh!

aG0aep6G commented Feb 9, 2015

Uh oh!

andralex commented Feb 10, 2015

Uh oh!

ghost commented Feb 12, 2015

Uh oh!

CyberShadow commented Feb 12, 2015

Uh oh!

ghost commented Feb 13, 2015

Uh oh!

CyberShadow commented Feb 13, 2015

Uh oh!

CyberShadow commented Feb 13, 2015

Uh oh!

ghost commented Feb 13, 2015

Uh oh!

MartinNowak commented Mar 14, 2015

Uh oh!

CyberShadow commented Mar 14, 2015

Uh oh!

CyberShadow commented Mar 14, 2015

Uh oh!

CyberShadow commented Mar 14, 2015

Uh oh!

MartinNowak commented Mar 15, 2015

Uh oh!

CyberShadow commented Mar 15, 2015

Uh oh!

MartinNowak commented Mar 15, 2015

Uh oh!

MartinNowak commented Mar 15, 2015

Uh oh!

aG0aep6G commented Mar 16, 2015

Uh oh!

MartinNowak commented Apr 25, 2015

Uh oh!

CyberShadow commented Oct 8, 2015

Uh oh!

andralex commented Nov 21, 2015

Uh oh!

CyberShadow commented Nov 22, 2015

Uh oh!

CyberShadow commented Nov 22, 2015

Uh oh!

aG0aep6G commented Nov 22, 2015

Uh oh!

CyberShadow commented Nov 23, 2015

Uh oh!

aG0aep6G commented Nov 23, 2015

Uh oh!

CyberShadow commented Nov 23, 2015

Uh oh!

aG0aep6G commented Nov 23, 2015

Uh oh!

aG0aep6G commented Nov 23, 2015

Uh oh!

CyberShadow commented Nov 23, 2015

Uh oh!

CyberShadow commented Nov 23, 2015

Uh oh!

aG0aep6G commented Nov 23, 2015

Uh oh!

CyberShadow commented Nov 23, 2015