Unicode mapping, codepoint fixes, and tests! #37

javan · 2013-11-06T02:20:33Z

Hot on the heels of the http://emojisaurus.com launch comes this hot pull request.

Inspired by #35, I added methods to convert name→unicode and unicode→name.

>> Emoji.unicode_for("+1")
=> "" [redacted to pass github's PR validation]
>> Emoji.name_for("\u{1F496}")
=> "sparkling_heart"

I added a suite of tests, one of which attempts to verify our unicode codepoints. Here's how it looked prior to this PR.

I merged in #26 to fix all but one of those. Turns out 🌠's codepoint was misidentified so I fixed that up too (8e09854).

That leaves us with one new failure because ⚠️we're missing an emoji⚠️

@juliamae, how did you generate the .pngs initially? We need a new one for "NIGHT WITH STARS" U+1F303

/cc @josh @trevorturk @asianmack @jonasd

Emoji digits and # are normal ASCII digits followed by COMBINING ENCLOSING KEYCAP (0x20e3). http://www.unicode.org/~scherer/emoji4unicode/20091221/emojidata.html

@mbillard

… h/t @mbillard

…entionally failing. Fix to follow.

…-mapping

trevorturk · 2013-11-06T03:05:27Z

Neato!

I generated the emoji initially -- lemme see if I can get NIGHT WITH STARS...

trevorturk · 2013-11-06T03:06:48Z

Hrm... I'm a bit confused. We already seem to have a 1F303 that's aliased as "stars" -- you know, I think they may have changed this between iOS versions... Any one have any ideas? I'm attaching what I believe is the right "night with stars" at the right size, but I'm not sure what to make of the code name conflict.

...pretty sure that's the right one (according to the OS X Font Book, anyway, which is where we got the original names...)

trevorturk · 2013-11-06T03:11:41Z

Oh, duh -- I skimmed over your comment about that, @javan. I'll add you to the repo and you can fix that up and merge this in, aight?

...aaand I see you're already a collaborator. Make it so! 👈

javan · 2013-11-06T03:14:42Z

I think it got mixed up in #16 because http://code.iamcal.com/php/emoji/ has it wrong. See "night with stars" and "shooting star" there.

trevorturk · 2013-11-06T03:16:08Z

Word -- this is ready to go, then. Drop in that image and merge 'er up, @javan

BTW I did the extraction with https://github.com/tmm1/emoji-extractor -- thanks @tmm1 🤘

javan · 2013-11-06T03:27:12Z

150c509 ❤️

jeremy · 2013-11-06T03:36:42Z

:man_in_business_suit_levitating:

josh · 2013-11-06T04:12:54Z

Nice!

Have a few small comments.

josh · 2013-11-06T04:13:41Z

Rakefile

-  unicodes = Dir["./images/emoji/unicode/*.png"].map { |fn| File.basename(fn) }
-  aliases = Dir["./images/emoji/*.png"].select { |fn| File.symlink?(fn) }.map { |fn| File.basename(fn) }
-  used_unicodes = aliases.map { |name| File.basename(File.readlink("./images/emoji/#{name}")) }.uniq
-  puts unicodes - used_unicodes


I was like "hey, why did you delete this!" at first, but I actually ❤️ the idea of making it a test. It should always be true.

josh · 2013-11-06T04:17:39Z

I like the method names here better than #35.

josh · 2013-11-06T04:18:59Z

Ah, rad, I wanted to merge #26 a long time ago, but I think I hit that issue as well. Nice.

javan · 2013-11-06T14:37:04Z

I figured out now to snarf Apple's emoji dictionary (https://github.com/github/gemoji/blob/unicode-mapping/Rakefile#L10-L16) and comparing their emoji to ours, it looks like we've got a bunch of code points that need updating. 😐

trevorturk · 2013-11-06T14:44:15Z

Weird -- just looks like we need to remove the "-fe0f" from all of those. Anyone know what that's about?

javan · 2013-11-06T15:01:04Z

My test output isn't super clear. The format is:

[Apple's unicode char] (Apple's code points) - [Our unicode char] (our code points) [our name]

So, we need to add "fe0f" to many of ours. Looks like it's to disambiguate emoji from its plain text twin. Like ♣️ and ♣. Not sure we ever had the code points right.

josh · 2013-11-06T16:52:36Z

Possible related history #20.

…ojis in iOS 7 / OS X 10.9. http://www.unicode.org/L2/L2011/11438-emoji-var.pdf http://unicode.org/reports/tr28/tr28-3.html#13_7_variation_selectors

javan · 2013-11-10T16:14:43Z

56da9a8 explains the "fe0f" variation selector and adds support for it. We should be all caught up with the latest emojis and their code points now.

josh · 2013-11-10T22:18:53Z

Rad.

Feel free to merge whenever you like.

Sent from my iPhone (in China)

On Nov 11, 2013, at 12:14 AM, Javan Makhmali notifications@github.com wrote:

56da9a8 explains the "fe0f" variation selector and adds support for it. We should be all caught up with the latest emojis and their code points now.

—
Reply to this email directly or view it on GitHub.

Unicode mapping, codepoint fixes, and tests!

javan · 2013-11-11T02:47:48Z

Merged and tagged. I don't have permission to push a new gem though.

trevorturk · 2013-11-11T02:51:58Z

Done.

🤘

javan · 2013-11-15T00:25:18Z

Unfortunately, installing gemoji from rubygems (i.e. a .gem file) kills the unicode mapping because the symlinks are lost. It appears that building the gem copies all of the files specified in the gemspec (s.files = Dir["README.md", "images/**/*.png", ...), but follows the symlinks and copies the actual destination files. So, we're not missing any files, just symlinks, and we have twice the number of bytes necessary since the files are essentially duped.

Anyone know if this is a bug or feature, and if there's a way around it?

Installing from github works so that's one workaround.

# gem 'gemoji'
>> File.symlink? "/usr/local/var/rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/gemoji-1.5.0/images/emoji/cry.png"
=> false

# gem 'gemoji', github: 'github/gemoji'
>> File.symlink? "/usr/local/var/rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/bundler/gems/gemoji-7755a8a18197/images/emoji/cry.png"
=> true

aroben · 2014-01-24T15:58:24Z

It doesn't look to me like there's a way to store symlinks in a gem. Seems like we'll need to get that information elsewhere.

javan · 2014-01-24T17:27:17Z

We could generate the mapping in some kind of pre-release task. For now,
installing from GitHub is a reasonable workaround.

On Friday, January 24, 2014, Adam Roben notifications@github.com wrote:

It doesn't look to me like there's a way to store symlinks in a gem. Seems
like we'll need to get that information elsewhere.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/37#issuecomment-33234608
.

lautis and others added 5 commits January 26, 2013 13:40

Fix digit/keycap unicode code points

f9f0527

Emoji digits and # are normal ASCII digits followed by COMBINING ENCLOSING KEYCAP (0x20e3). http://www.unicode.org/~scherer/emoji4unicode/20091221/emojidata.html

Convert emoji names to unicode and vice versa. Add some *gasp* tests.…

bfbc8c5

… h/t @mbillard

Tests to ensure the gem is in good shape. Leave the unicode tests int…

cb85b7b

…entionally failing. Fix to follow.

Merge branch 'keycaps' of git://github.com/lautis/gemoji into unicode…

968373b

…-mapping

Fix incorrect codepoint for 🌠

8e09854

Add missing 🌃

150c509

josh reviewed Nov 6, 2013
View reviewed changes

josh mentioned this pull request Nov 6, 2013

Added methods to convert emojis to and from unicode characters #35

Closed

josh mentioned this pull request Nov 6, 2013

Fix digit/keycap unicode code points #26

Closed

Compare our emoji to emoji straight from the horse's mouth

c180447

javan added 4 commits November 10, 2013 10:04

Add support for the unicode variation selector (fe0f) added to 107 em…

56da9a8

…ojis in iOS 7 / OS X 10.9. http://www.unicode.org/L2/L2011/11438-emoji-var.pdf http://unicode.org/reports/tr28/tr28-3.html#13_7_variation_selectors

Add missing train emoji

9cd56f6

Fix up incorrect/mising speaker emojis

19ddee2

Make mappings private pending better names

13bcea9

javan added a commit that referenced this pull request Nov 11, 2013

Merge pull request #37 from github/unicode-mapping

69f625f

Unicode mapping, codepoint fixes, and tests!

javan merged commit 69f625f into master Nov 11, 2013

javan deleted the unicode-mapping branch November 11, 2013 02:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unicode mapping, codepoint fixes, and tests! #37

Unicode mapping, codepoint fixes, and tests! #37

javan commented Nov 6, 2013

trevorturk commented Nov 6, 2013

trevorturk commented Nov 6, 2013

trevorturk commented Nov 6, 2013

javan commented Nov 6, 2013

trevorturk commented Nov 6, 2013

javan commented Nov 6, 2013

jeremy commented Nov 6, 2013

josh commented Nov 6, 2013

josh Nov 6, 2013

josh commented Nov 6, 2013

josh commented Nov 6, 2013

javan commented Nov 6, 2013

trevorturk commented Nov 6, 2013

javan commented Nov 6, 2013

josh commented Nov 6, 2013

javan commented Nov 10, 2013

josh commented Nov 10, 2013

javan commented Nov 11, 2013

trevorturk commented Nov 11, 2013

javan commented Nov 15, 2013

aroben commented Jan 24, 2014

javan commented Jan 24, 2014

Unicode mapping, codepoint fixes, and tests! #37

Unicode mapping, codepoint fixes, and tests! #37

Conversation

javan commented Nov 6, 2013

trevorturk commented Nov 6, 2013

trevorturk commented Nov 6, 2013

trevorturk commented Nov 6, 2013

javan commented Nov 6, 2013

trevorturk commented Nov 6, 2013

javan commented Nov 6, 2013

jeremy commented Nov 6, 2013

josh commented Nov 6, 2013

josh Nov 6, 2013

Choose a reason for hiding this comment

josh commented Nov 6, 2013

josh commented Nov 6, 2013

javan commented Nov 6, 2013

trevorturk commented Nov 6, 2013

javan commented Nov 6, 2013

josh commented Nov 6, 2013

javan commented Nov 10, 2013

josh commented Nov 10, 2013

javan commented Nov 11, 2013

trevorturk commented Nov 11, 2013

javan commented Nov 15, 2013

aroben commented Jan 24, 2014

javan commented Jan 24, 2014