add unicode support in slugs

[thecotne]

Unicode compatible slug generation code is taken from here https://github.com/bryanbraun/anchorjs/blob/master/anchor.js#L208
so if this accepted we need to add @bryanbraun in contributors list

for me this fixes #194
this will work for any language around the globe

[franzliedke]

Hmm, so this does not do any transliteration, but instead only removes those characters which should not appear in URLs? Non-ASCII characters would remain, right?

As per discussion in #194, that's not what we wanted to do, but I'm unsure whether this solution may be okay, too.

[thecotne]

there are unicode TLDs for example .გე and there is website http://რეგისტრაცია.გე and it does not make any sense to transliteration slugs in that website or any website with unicode domain name

and also i have not invented this method of creating slugs it is created by wikipedia

[thecotne]

so this does not do any transliteration

yes

but instead only removes those characters which should not appear in URLs?

yes

Non-ASCII characters would remain, right?

right

[tobyzerner]

Without commenting on the actual method used for slug generation... I was under the impression that recently @franzliedke made a change so that slugs are generated on the server-side. This PR however only touches the JS version of the function, which is no longer used.

[franzliedke]

Well, it is used somewhere; I believe the tags extension uses it.

[thecotne]

This PR however only touches the JS version of the function, which is no longer used.
-- @tobscure

updated php version (have not tested)

[thecotne]

@franzliedke @tobscure any progress on this?

[luceos]

If @franzliedke moved slug generation to the backend, wouldn't it make more sense to remove all logic from the frontend - SRP?

[tobyzerner]

@luceos the slug logic on the frontend is only used to live autocomplete the slug field when typing a tag name in the admin interface.

[luceos]

I see, so both frontend and backend need to mimic the same behavior closely. Isn't slug creation something we want enforced from one side, with the frontend being the most logical choice?

[franzliedke]

Well, if anything, we can only enforce it on the server. But since we want these live-preview features on the client, we'll have to go with a duplicate implementation here. (The only way around that is isomorphic JS and, well, that doesn't really fit here, hehe.)

[johannsa]

The URLs is something we definitely would like to control server side as we would not like somebody to control from his client side how a post URL is going to look like as the potencial for abuse/trolling is enormous.

@franzliedke Maybe for preview we can query the server?

On my opinion, this commit fixes #194. What is stopping this PR to be accepted and merged?

[franzliedke]

@johannsa We haven't tested it yet. And we might use a library instead.

[franzliedke]

I'll look at it in time for the next beta, though.

[dcsjapan]

On my opinion, this commit fixes #194. What is stopping this PR to be accepted and merged?

I like the idea of unicode slugs, but some admins may prefer transliterated slugs if that's the norm in their region. So it might be a good idea to add an admin preference selecting between this method and a slug transliteration library/whatever.

Going back to the comment by @luceos ...

I see, so both frontend and backend need to mimic the same behavior closely. Isn't slug creation something we want enforced from one side, with the frontend being the most logical choice?

There's an important difference between tag slugs and title slugs:

• Title slugs aren't crucial: they can change whenever a discussion title is changed, and so must be generated automatically based on the data entered by the user.
• Tag slugs, by contrast, are crucial ... but they can be set manually by the admin, and aren't likely to change all that often.

Given this difference, we could probably get away with leaving the tag slugs as they are now: no unicode support, no transliteration ... there's no need for transliteration, since the admin should be able to do that manually as well, so this could be an option if, for example, there's some reason that unicode can't be used in tag slugs.

But the best solution would probably be to have the admin panel suggest tag slugs using the same logic (transliteration or unicode support) as is used for the title slugs.

[tobyzerner]

Superseded by #1385.