New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract PNG images from Apple's "Apple Color Emoji.ttf" font #63

Merged
merged 6 commits into from Oct 16, 2014

Conversation

Projects
None yet
6 participants
@javan
Collaborator

javan commented Sep 21, 2014

The images currently included in this gem were extracted using https://github.com/tmm1/emoji-extractor and then renamed by hand (I think) to match their unicode hex values. emoji-extractor.rb uses a regex to parse out png data for each emoji, and isn't able to match them with their codepoints so they come out named sequentially (1.png, 2.png, etc.). See tmm1/emoji-extractor#1.

Today I dug in and figured out where and how all the bitmap bits are stored in Apple's emoji font (hint, right here) and added support to the ttfunk gem for parsing it all out (PR: prawnpdf/ttfunk#22). Combined with other character data in the font file, we now have enough information to identify and extract each emoji png at every available size (20x20, 32x32, 40x40, 48x48, 64x64, 96x96, 160x160).

You can see from this PR that many of the images have subtle color changes. I'm guessing that's because the originals were extracted long ago and Apple has since refined them.

One thought I had is that we could remove the included images from the gem and have people extract them on their own with a rake task.This would give them a choice of which size(s) to extract and wipe our hands clean of any copyright issues. What do you think?

cc: @mislav, @josh, @trevorturk, @aroben, @tmm1

馃帀

@practicingruby

This comment has been minimized.

practicingruby commented Sep 21, 2014

Related to the colors, I'm not sure if this is at all relevant but since I ended up on this page when reviewing the TTFunk pull request, I figured I'd leave a link here: http://en.wikipedia.org/wiki/OpenType#Color

@tmm1

This comment has been minimized.

tmm1 commented Sep 21, 2014

Today I dug in and figured out where and how all the bitmap bits are stored in Apple's emoji font (hint, right here) and added support to the ttfunk gem for parsing it all out (PR: prawnpdf/ttfunk#22). Combined with other character data in the font file, we now have enough information to identify and extract each emoji png at every available size (20x20, 32x32, 40x40, 48x48, 64x64, 96x96, 160x160).

Very very cool.

One thought I had is that we could remove the included images from the gem and have people extract them on their own with a rake task.This would give them a choice of which size(s) to extract and wipe our hands clean of any copyright issues. What do you think?

馃憤

@mislav

This comment has been minimized.

Member

mislav commented Sep 21, 2014

So awesome. Three things before we can accept this PR:

  1. The dependent ttfunk gem has to have a release available from RubyGems.org so we can drop the git dependency.
  2. Please edit gemoji code and update the image PNGs in separate commits so it's easier to follow through changes.
  3. This broke the build by failing one of my integrity checks. Apparently two different PNGs ended up with the same checksum, indicating that your extractor script needs tweaking: https://travis-ci.org/github/gemoji/jobs/35836344

One thought I had is that we could remove the included images from the gem and have people extract them on their own with a rake task.

馃憥 I want this library to be usable on all platforms not just the Mac.

@practicingruby

This comment has been minimized.

practicingruby commented Sep 21, 2014

I just cut a new release of TTFunk (1.4.0) that includes @javan's patch. Hope that helps!

@javan javan force-pushed the extractor branch from 1f5a310 to 1b35fbd Sep 21, 2014

@javan

This comment has been minimized.

Collaborator

javan commented Sep 21, 2014

@mislav

  1. 鈽戯笍
  2. 鈽戯笍
  3. Turns out Apple now uses the exact same PNG for :black_medium_square: and :black_large_square:. Here's my full investigation: https://gist.github.com/javan/512ad91917bf2ba24555. We either need to add a special case to the test for those emojis or remove the test. Do you have a preference?

Re: removing images. I personally don't care either way. The gem is hosted within your github account so you all can make the call. It would be nice to allow users to pick which size they want so I think we should either include all of the sizes in the gem or move the images:extract task to lib/tasks/emoji.rake and adjust it for copying to your app.

@mislav

This comment has been minimized.

Member

mislav commented Sep 22, 2014

We either need to add a special case to the test for those emojis or remove the test. Do you have a preference?

Yeah, please add a special case. We also have a special case for shipit squirrel being the only image that's not 64x64 px. I like having a test that verifies there are no duplicates.

@mislav

This comment has been minimized.

Member

mislav commented Sep 22, 2014

This increases the size of images slightly. I was able to shave it down locally to almost half size by means of:

$ imageOptim -a -d images/emoji/unicode/
...
TOTAL was: 4280.370kb now: 2234.615kb saving: 2045.755kb (47.00%) quality: 99.236776%

I would like to make this a built-in task that follows the extraction of images, so the size doesn't go back up the next time someone runs the extract task. However, what makes it tricky is that the above command relies on ImageOptim and ImageAlpha OS X applications being installed on the system. Ideally, our image optimization step should work without you having to install anything in your system. I'm pondering whether the optimization task could download these OS X apps in a temporary location and invoke binaries in them directly. Does this sound like a sane idea?

@javan

This comment has been minimized.

Collaborator

javan commented Sep 22, 2014

Thinking on this more, my preference would be to remove the emoji images from the gem and ship it with emoji:images:extract and emoji:images:optimize Rake tasks. Then people can extract whatever size(s) they want and optimize using our task (and whatever dependencies it has) or using their own optimizer.

For non-Mac compatibility, we could create a separate gemoji-images gem or package them all up on S3 somewhere.

@mislav

This comment has been minimized.

Member

mislav commented Sep 22, 2014

For non-Mac compatibility, we could create a separate gemoji-images gem or package them all up on S3 somewhere.

Why do that? I thought that the whole point of removing images from this gem is to stop re-distributing what we don't have rights to.

@javan

This comment has been minimized.

Collaborator

javan commented Sep 22, 2014

Well, I'd prefer not distributing the images at all, but you mentioned wanting to maintain compatibility with non-Mac platforms so I see that idea being a compromise. Splitting the images and the gem gives us a separate repo to take down should there ever be a copyright issue.

@mislav

This comment has been minimized.

Member

mislav commented Sep 22, 2014

On Mon, Sep 22, 2014 at 12:43 PM, Javan Makhmali notifications@github.com
wrote:

you mentioned wanting to maintain compatibility with non-Mac platforms so
I see that idea being a compromise

I don't think we can compromise. If we want user-friendliness then we
should distribute images. If we don't want to distribute images then we
should take them out completely. Packaging them separately is less
user-friendly and still copyright-infringing.

I could be convinced that we can drop the images in the next major release,
but that means telling people that they will need access to a Mac at the
time they're adding emoji to their app.

@javan

This comment has been minimized.

Collaborator

javan commented Sep 22, 2014

How do you feel about adding other-sized images then? They'd be useful to anyone designing for multiple resolutions / font sizes and they're something we (Basecamp) have an immediate need for. Perhaps adding 32x32 and 160x160? Or go wild and add them all.

@mislav

This comment has been minimized.

Member

mislav commented Sep 22, 2014

On Mon, Sep 22, 2014 at 1:23 PM, Javan Makhmali notifications@github.com
wrote:

How do you feel about adding other-sized images then?

I'm not too happy about adding different sizes. We'd have to name alternate
sizes differently, and we'd significantly increase the size of the gem.
Maybe you're right that we should make a release without images in it and
have people include the rake task in their apps which could have an option
to pick the image size.

Let's make this PR about just the extractor code, but without actually
updating the images? Then we can think about ripping out the images and
making a new major release.

@javan javan force-pushed the extractor branch from 9c89b43 to 1a836d3 Sep 25, 2014

@javan

This comment has been minimized.

Collaborator

javan commented Sep 25, 2014

Yeah, please add a special case. We also have a special case for shipit squirrel being the only image that's not 64x64 px. I like having a test that verifies there are no duplicates.

鈽戯笍

If anything, I think we should update the images and continue to have just the one size. Most of the current images were extracted long ago and have been subtly tweaked by Apple since (presumably to make them look better). Let's give gemoji a fresh coat of paint.

javan and others added some commits Sep 21, 2014

Extract and optimize PNGs from "Apple Color Emoji.ttf"
Commands used:

    $ rake images:extract
    $ imageOptim -a -d images/emoji/unicode/

This required both ImageOptim and ImageAlpha to be installed as OS X apps.
Run the rest of the PNGs through ImageOptim
Command used:

    $ imageOptim -a -d images/emoji/
@javan

This comment has been minimized.

Collaborator

javan commented Oct 14, 2014

@mislav, @josh wdyt about merging this as-is or after a pass through an image optimizer?

If you're 馃憥, I think I'll build a separate gem just for extracting the images and gemoji depend on it later.

@mislav mislav force-pushed the extractor branch from 8c21eef to f5092b7 Oct 16, 2014

@mislav

This comment has been minimized.

Member

mislav commented Oct 16, 2014

@javan: I've ran the updated images through the optimizer and rebased this PR. Please take a look. I'm 馃憤 for packaging this as a next minor release, but in the future it might be better to not distribute the images at all but provide a task, like you originally suggested. We could also offer that you download one of the open source emoji sets as an alternative to using Apple's.

@javan

This comment has been minimized.

Collaborator

javan commented Oct 16, 2014

Thanks for the 馃憖, @mislav! :shipit:

Happy to help get an image-less release going too.

mislav added a commit that referenced this pull request Oct 16, 2014

Merge pull request #63 from github/extractor
Extract PNG images from Apple's "Apple Color Emoji.ttf" font

@mislav mislav merged commit 757584d into master Oct 16, 2014

@mislav mislav deleted the extractor branch Oct 16, 2014

@bbharier

This comment has been minimized.

bbharier commented Dec 1, 2015

Is this extract feature functional? I have little experience with Ruby and can't figure out how to run the extractor.

@mislav

This comment has been minimized.

Member

mislav commented Dec 1, 2015

@bbharier:

require 'emoji/extractor'
Emoji::Extractor.new(64, "path/to/emoji/images").extract!
@bbharier

This comment has been minimized.

bbharier commented Dec 2, 2015

Thank you!

@Danny182

This comment has been minimized.

Danny182 commented May 21, 2016

I found a way by terminal running "rake images:extract" it extracts the emojis but it doesn't extracts some of them.
"馃檮 1F644" doesn't get extracted. I updated manually the "db/NamesList.txt" (the info of that emoji is in there).
Can someone help me?

Thanks in advance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment