Bundle icu4c #4

josh · 2011-10-14T16:53:06Z

The extra setup steps are getting to me now. Really, really don't want to have to remember to brew install icu4c again. Its also breaking my travis builds. :(

Another approach would be to spin off a libicu gem for just the source. We could also attempt binary builds that way as well.

/cc @brianmario @tmm1 @rtomayko @tanoku

brianmario · 2011-10-14T17:03:46Z

fwiw you can add this to your .travis.yml too:

before_script:
  - "apt-get install libicu-dev"

But, I'd still like to get around having to install it :)

weppos · 2011-10-26T18:33:36Z

I'm really interested in this topic. I'm trying to install the Gem on Heroku, where unfortunately the libicu-dev library is not available.

Just to let you know, I tried with this branch but it's still raising an error.

Using charlock_holmes (0.6.8) from https://github.com/brianmario/charlock_holmes.git (at bundle-icu) with native extensions Unfortunately, a fatal error has occurred. Please report this error to the Bundler issue tracker at https://github.com/carlhuda/bundler/issues so that we can fix it. Thanks!
/usr/local/lib/ruby/1.9.1/rubygems/installer.rb:483:in `rescue in block in build_extensions': ERROR: Failed to build gem native extension. (Gem::Installer::ExtensionBuildError)
/usr/local/bin/ruby extconf.rb
-- tar zxvf icu4c-4_8_1-src.tgz
-- ./configure --prefix=/tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/bundler/gems/charlock_holmes-0fa4d3ff22e3/ext/charlock_holmes/dst/ --disable-tests --disable-samples --disable-icuio --disable-extras --disable-layout --enable-static
-- make install
checking for main() in -licui18n... no
***************************************************************************************
********* error compiling and linking icu4c. please report issue on github *********
***************************************************************************************
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.
Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/usr/local/bin/ruby
--with-icu-dir
--without-icu-dir
--with-icu-include
--without-icu-include=${icu-dir}/include
--with-icu-lib
--without-icu-lib=${icu-dir}/lib
--with-icui18nlib
--without-icui18nlib
Gem files will remain installed in /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/bundler/gems/charlock_holmes-0fa4d3ff22e3 for inspection.
Results logged to /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/bundler/gems/charlock_holmes-0fa4d3ff22e3/ext/charlock_holmes/gem_make.out
from /usr/local/lib/ruby/1.9.1/rubygems/installer.rb:486:in `block in build_extensions'
from /usr/local/lib/ruby/1.9.1/rubygems/installer.rb:446:in `each'
from /usr/local/lib/ruby/1.9.1/rubygems/installer.rb:446:in `build_extensions'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/source.rb:443:in `generate_bin'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/source.rb:564:in `install'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/installer.rb:73:in `block in install_gem_from_spec'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/rubygems_integration.rb:93:in `with_build_args'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/installer.rb:72:in `install_gem_from_spec'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/installer.rb:56:in `block in run'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/installer.rb:55:in `run'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/installer.rb:12:in `install'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/cli.rb:219:in `install'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/vendor/thor/task.rb:22:in `run'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/vendor/thor.rb:263:in `dispatch'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/lib/bundler/vendor/thor/base.rb:386:in `start'
from /tmp/build_22v1vaioxw0z6/vendor/bundle/ruby/1.9.1/gems/bundler-1.1.rc/bin/bundle:13:in `<top (required)>'
from vendor/bundle/ruby/1.9.1/bin/bundle:19:in `load'
from vendor/bundle/ruby/1.9.1/bin/bundle:19:in `<main>'

josh · 2011-10-26T18:44:23Z

Good point. We really want this to help support heroku.

josh · 2011-11-09T20:31:16Z

I can't seem to get to work anymore.

$ bundle exec rake
/Users/josh/.rbenv/versions/1.9.3-p0/bin/ruby -I. ../../../../ext/charlock_holmes/extconf.rb
  -- tar zxvf icu4c-4_8_1-src.tgz
  -- ./configure --prefix=/Volumes/Scratch/charlock_holmes/ext/charlock_holmes/dst/ --disable-tests --disable-samples --disable-icuio --disable-extras --disable-layout --enable-static
  -- make install
checking for main() in -licui18n... yes
checking for unicode/ucnv.h... yes
  -- tar zxvf file-5.08.tar.gz
  -- ./configure --prefix=/Volumes/Scratch/charlock_holmes/ext/charlock_holmes/dst/ --disable-shared --enable-static --with-pic
  -- make -C src install
  -- make -C magic install
checking for main() in -lmagic_ext... yes
checking for magic.h... yes
creating Makefile
make
compiling ../../../../ext/charlock_holmes/converter.c
../../../../ext/charlock_holmes/converter.c: In function ‘rb_converter_convert’:
../../../../ext/charlock_holmes/converter.c:19: warning: implicit conversion shortens 64-bit value into a 32-bit value
../../../../ext/charlock_holmes/converter.c:7: warning: unused parameter ‘self’
../../../../ext/charlock_holmes/converter.c: At top level:
../../../../ext/charlock_holmes/common.h:24: warning: ‘charlock_new_str’ defined but not used
../../../../ext/charlock_holmes/common.h:33: warning: ‘charlock_new_str2’ defined but not used
compiling ../../../../ext/charlock_holmes/encoding_detector.c
../../../../ext/charlock_holmes/encoding_detector.c: In function ‘rb_encdec_detect’:
../../../../ext/charlock_holmes/encoding_detector.c:100: warning: implicit conversion shortens 64-bit value into a 32-bit value
../../../../ext/charlock_holmes/encoding_detector.c: In function ‘rb_encdec_detect_all’:
../../../../ext/charlock_holmes/encoding_detector.c:149: warning: implicit conversion shortens 64-bit value into a 32-bit value
../../../../ext/charlock_holmes/encoding_detector.c: At top level:
../../../../ext/charlock_holmes/common.h:15: warning: ‘charlock_new_enc_str’ defined but not used
compiling ../../../../ext/charlock_holmes/ext.c
../../../../ext/charlock_holmes/common.h:15: warning: ‘charlock_new_enc_str’ defined but not used
../../../../ext/charlock_holmes/common.h:24: warning: ‘charlock_new_str’ defined but not used
../../../../ext/charlock_holmes/common.h:33: warning: ‘charlock_new_str2’ defined but not used
linking shared-object charlock_holmes/charlock_holmes.bundle
/Users/josh/.rbenv/versions/1.9.3-p0/bin/ruby -S bundle exec rspec ./spec/converter_spec.rb ./spec/encoding_detector_spec.rb ./spec/string_method_spec.rb
/Volumes/Scratch/charlock_holmes/lib/charlock_holmes.rb:1:in `require': dlopen(/Volumes/Scratch/charlock_holmes/lib/charlock_holmes/charlock_holmes.bundle, 9): Library not loaded: libicui18n.48.dylib (LoadError)
  Referenced from: /Volumes/Scratch/charlock_holmes/lib/charlock_holmes/charlock_holmes.bundle
  Reason: image not found - /Volumes/Scratch/charlock_holmes/lib/charlock_holmes/charlock_holmes.bundle
    from /Volumes/Scratch/charlock_holmes/lib/charlock_holmes.rb:1:in `<top (required)>'
    from /Volumes/Scratch/charlock_holmes/spec/spec_helper.rb:3:in `require'
    from /Volumes/Scratch/charlock_holmes/spec/spec_helper.rb:3:in `<top (required)>'
    from /Volumes/Scratch/charlock_holmes/spec/converter_spec.rb:3:in `require'
    from /Volumes/Scratch/charlock_holmes/spec/converter_spec.rb:3:in `<top (required)>'
    from /Users/josh/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rspec-core-2.6.4/lib/rspec/core/configuration.rb:419:in `load'
    from /Users/josh/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rspec-core-2.6.4/lib/rspec/core/configuration.rb:419:in `block in load_spec_files'
    from /Users/josh/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rspec-core-2.6.4/lib/rspec/core/configuration.rb:419:in `map'
    from /Users/josh/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rspec-core-2.6.4/lib/rspec/core/configuration.rb:419:in `load_spec_files'
    from /Users/josh/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rspec-core-2.6.4/lib/rspec/core/command_line.rb:18:in `run'
    from /Users/josh/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rspec-core-2.6.4/lib/rspec/core/runner.rb:80:in `run_in_process'
    from /Users/josh/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rspec-core-2.6.4/lib/rspec/core/runner.rb:69:in `run'
    from /Users/josh/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rspec-core-2.6.4/lib/rspec/core/runner.rb:11:in `block in autorun'
rake aborted!
ruby -S bundle exec rspec ./spec/converter_spec.rb ./spec/encoding_detector_spec.rb ./spec/string_method_spec.rb failed

Tasks: TOP => default => spec
(See full trace by running task with --trace)

jch · 2012-08-06T22:05:20Z

@josh did you come up with a solution, or did you just package it with a pre-built binary? I'm trying to get this working on heroku.

josh · 2012-08-06T22:17:39Z

@jch never figured it out. PDI :D

jch · 2012-08-06T22:22:39Z

@josh 💩, thanks anyways

juliamae · 2012-10-03T00:13:48Z

This is still uninstallable on Heroku, which is a bummer since http://github.com/github/linguist depends on it. @jch, did you figure anything out?

jch · 2012-10-03T00:29:55Z

@juliamae haven't looked at it again. At some point, I'll need to take a look again since https://github.com/github/github-html uses linguist and I'm working on open sourcing that.

EDIT: url is now https://github.com/jch/html-pipeline

brianmario · 2012-10-03T00:32:00Z

I'm taking a look right now, will hopefully post progress soon.

scottjg · 2012-10-03T04:07:14Z

i was able to fix this, but i don't have push access to update the pr.

brianmario · 2012-10-03T04:52:27Z

rad! you should have push now

On Oct 2, 2012, at 9:07 PM, "Scott J. Goldman" notifications@github.com wrote:

i was able to fix this, but i don't have push access to update the pr.

—
Reply to this email directly or view it on GitHub.

scottjg · 2012-10-03T05:04:47Z

hmm i guess the PR doesn't update with new commits when it's closed? but i did push the fix to this branch with 3a916ab

brianmario · 2012-10-03T05:17:28Z

fuck yeah thanks man!

On Oct 2, 2012, at 10:04 PM, "Scott J. Goldman" notifications@github.com wrote:

hmm i guess the PR doesn't update with new commits when it's closed? but i did push the fix to this branch with 3a916ab

—
Reply to this email directly or view it on GitHub.

jch · 2012-10-03T09:20:06Z

@scottjg ✨ I'll update the gem to point at this.

jch · 2012-10-04T00:32:30Z

@brianmario is there anything holding this back from being merged into master and bumping the version?

brianmario · 2012-10-04T00:48:02Z

Not really... I just still feel icky about a 19MB gem :
I'll get over it and post a new gem soon though ;)

juliamae · 2012-10-12T20:36:01Z

@brianmario I've been specifying this branch in the community app (for use with html-pipeline) and it is ccrraazzyy slow to install during CI and deploy. It adds about 5 minutes to our CI and over 5 minutes to the Heroku deploy 😒

However, it's successfully deploying to Heroku 🎉

Rather than have it be this slow for people who don't need it, maybe offer a tag that has icu4c bundled or maintain this branch with updates. Maintaining a branch is a pain, but I don't think it makes sense to blow up the size & ruin the performance for everyone.

brianmario · 2012-10-12T21:21:49Z

@juliamae woohoo! Yeah I'm in favor of maintaining the branch. I don't think there will be many changes to master very often so it shouldn't be hard to maintain.

Thanks for the update!

maca · 2013-01-10T04:54:50Z

Is it possible to use heroku buildpacks? I could use this buildpack (https://github.com/maca/mutli-buildpack-icu4c) to compile icu4c on the heroku instance but I am having issues with the lib path while compiling the gem ext.

malsharman · 2013-01-18T02:19:18Z

It seems very difficult to get this gem to work on heroku. I've used the bundle-icu branch. This option got the furthest, and got to a point of require 'charlock_holmes' working - then failed when attempting to use any functions with an error of "could not find magic files".
Any ideas?

6 · 2013-01-24T06:00:41Z

I'm having the same issue as @malsharman. I added the following to my Gemfile:

gem 'charlock_holmes', :git => "git://github.com/brianmario/charlock_holmes.git", :branch => "bundle-icu"

This deploys to Heroku, and require 'charlock_holmes' works, but I get the same "could not find magic files" error when using CharlockHolmes::EncodingDetector.detect.

It doesn't help us determine if the file is binary or not, and just seems to create a bunch of problems like the ones in #4

scottjg · 2013-01-27T09:54:54Z

the magic files issue should be resolved now that #26 is merged into the bundle-icu branch. can you guys try again if you were having that problem?

6 · 2013-01-27T17:38:25Z

Awesome, it works now on Heroku!! Thanks @scottjg!

geothird · 2013-02-26T17:27:43Z

For anyone who needs it, I added a gem for the bundle-icu branch.

gem 'charlock-holmes-bundle-icu', '~> 0.6.9.2'

Also created a fork of github-linguist for deploying to heroku with this as its charlock_holmes dependency.

gem 'github-linguist', git: 'https://github.com/geothird/linguist.git'

jredburn · 2013-02-27T20:00:36Z

@geothird The bundle-icu gem fails for me on Heroku with:

       Installing charlock-holmes-bundle-icu (0.6.9.2)
       Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.
       /tmp/build_2lwy2n16fwqzz/vendor/ruby-1.9.3/bin/ruby extconf.rb
       checking for main() in -licui18n... no
       checking for main() in -licui18n... no
       ***************************************************************************************
       *********** icu required (brew install icu4c or apt-get install libicu-dev) ***********
       ***************************************************************************************
       *** extconf.rb failed ***
       Could not create Makefile due to some reason, probably lack of
       necessary libraries and/or headers.  Check the mkmf.log file for more
       details.  You may need configuration options.

geothird · 2013-03-02T02:18:32Z

I didn't include the lib, this one should work, tested on heroku.

gem "charlock_holmes_bundle_icu", "~> 0.6.9.2"

It doesn't help us determine if the file is binary or not, and just seems to create a bunch of problems like the ones in #4

…nmario/charlock_holmes#4

Bundle icu4c

1b1699c

disable some icu components we're not using

0fa4d3f

josh closed this Nov 27, 2011

chrislloyd mentioned this pull request Dec 27, 2012

Optionally require github-linguist gjtorikian/html-pipeline#33

Closed

scottjg pushed a commit that referenced this pull request Jan 27, 2013

Don't use the "soft" magic db

7d02997

It doesn't help us determine if the file is binary or not, and just seems to create a bunch of problems like the ones in #4

scottjg mentioned this pull request Jan 27, 2013

Don't use the "soft" magic db #26

Merged

scottjg pushed a commit that referenced this pull request Mar 8, 2013

Don't use the "soft" magic db

0989ca9

It doesn't help us determine if the file is binary or not, and just seems to create a bunch of problems like the ones in #4

harlantwood added a commit to harlantwood/html_massage that referenced this pull request Sep 16, 2014

Install dependency needed by charlock_holmes; for more info see: bria…

7697559

…nmario/charlock_holmes#4

benogle mentioned this pull request Jul 9, 2015

Correctly detect the encoding atom/atom#4160

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bundle icu4c #4

Bundle icu4c #4

josh commented Oct 14, 2011

brianmario commented Oct 14, 2011

weppos commented Oct 26, 2011

josh commented Oct 26, 2011

josh commented Nov 9, 2011

jch commented Aug 6, 2012

josh commented Aug 6, 2012

jch commented Aug 6, 2012

juliamae commented Oct 3, 2012

jch commented Oct 3, 2012

brianmario commented Oct 3, 2012

scottjg commented Oct 3, 2012

brianmario commented Oct 3, 2012

scottjg commented Oct 3, 2012

brianmario commented Oct 3, 2012

jch commented Oct 3, 2012

jch commented Oct 4, 2012

brianmario commented Oct 4, 2012

juliamae commented Oct 12, 2012

brianmario commented Oct 12, 2012

maca commented Jan 10, 2013

malsharman commented Jan 18, 2013

6 commented Jan 24, 2013

scottjg commented Jan 27, 2013

6 commented Jan 27, 2013

geothird commented Feb 26, 2013

jredburn commented Feb 27, 2013

geothird commented Mar 2, 2013

Bundle icu4c #4

Bundle icu4c #4

Conversation

josh commented Oct 14, 2011

brianmario commented Oct 14, 2011

weppos commented Oct 26, 2011

josh commented Oct 26, 2011

josh commented Nov 9, 2011

jch commented Aug 6, 2012

josh commented Aug 6, 2012

jch commented Aug 6, 2012

juliamae commented Oct 3, 2012

jch commented Oct 3, 2012

brianmario commented Oct 3, 2012

scottjg commented Oct 3, 2012

brianmario commented Oct 3, 2012

scottjg commented Oct 3, 2012

brianmario commented Oct 3, 2012

jch commented Oct 3, 2012

jch commented Oct 4, 2012

brianmario commented Oct 4, 2012

juliamae commented Oct 12, 2012

brianmario commented Oct 12, 2012

maca commented Jan 10, 2013

malsharman commented Jan 18, 2013

6 commented Jan 24, 2013

scottjg commented Jan 27, 2013

6 commented Jan 27, 2013

geothird commented Feb 26, 2013

jredburn commented Feb 27, 2013

geothird commented Mar 2, 2013