New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New process for building, distributing Shoes gems #72

Closed
ccoupe opened this Issue Feb 20, 2015 · 11 comments

Comments

Projects
None yet
2 participants
@ccoupe
Contributor

ccoupe commented Feb 20, 2015

It's really annoying to build all of Shoes just to fix a gem. It's really annoying to build gems (and ext) just to fix something unrelated to them. It will even more annoying if we wanted to replace or add gems in the future.

My half baked idea is that there would be no ext - so ftsearch and chipmunk would be converted to gems. Those and the gem hpricot and sqlite3 would be in some gem archive file - Shoes.gems for example that would be part of a download and install. In the install phase when Shoes runs for the first time, it see it doesn't have it's built in gems and install them. We add a button to Cobbler to download and install newer Shoes.gems if available.

Since gems can have pre-compiled binary payloads we would have to provide Shoes.gem for each platform to be included when we do rake package.

The task of building and updating the gems is mostly separated from Shoes and very dependent on rubygems doing what I think it can do.

@ccoupe ccoupe added the Wishlist label Feb 20, 2015

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Feb 20, 2015

Collaborator

This change is certainly appealing. We already talked about the possibility to add new gems and replace old ones (Nokogiri instead of hpricot, Picky instead of ftsearch).

There might be one caveat in the scenario proposed above — during development, I would often test things in tightmingw directory where the gems should be readily available. It means that they might have to be preinstalled instead of installing them at installation time.

Collaborator

BackOrder commented Feb 20, 2015

This change is certainly appealing. We already talked about the possibility to add new gems and replace old ones (Nokogiri instead of hpricot, Picky instead of ftsearch).

There might be one caveat in the scenario proposed above — during development, I would often test things in tightmingw directory where the gems should be readily available. It means that they might have to be preinstalled instead of installing them at installation time.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Feb 21, 2015

Contributor

Indeed, that is a weakness. Something that might be easier and still provide some separation between gem building and shoes building would be to copy the gem contents into the resulting shoes (tightmingw in this case) similar to what it does now. The gems source/build would not live in the shoes3.2 directories. For the gem building side , building a gem could copy it into the an existing tightmingw (if it exists) replacing what was there.. Sadly, this wouldn't simplify much, if at all. More thinking.

Contributor

ccoupe commented Feb 21, 2015

Indeed, that is a weakness. Something that might be easier and still provide some separation between gem building and shoes building would be to copy the gem contents into the resulting shoes (tightmingw in this case) similar to what it does now. The gems source/build would not live in the shoes3.2 directories. For the gem building side , building a gem could copy it into the an existing tightmingw (if it exists) replacing what was there.. Sadly, this wouldn't simplify much, if at all. More thinking.

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Feb 21, 2015

Collaborator

It should definitively copy the gems into the resulting Shoes. The advantage here is that it would be possible to build them separately. Rake tasks could be written in such way to only build those gems if not built, when changed and/or when the specific tasks are invoked.

Collaborator

BackOrder commented Feb 21, 2015

It should definitively copy the gems into the resulting Shoes. The advantage here is that it would be possible to build them separately. Rake tasks could be written in such way to only build those gems if not built, when changed and/or when the specific tasks are invoked.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Feb 21, 2015

Contributor

Yes, the gems should be (re)built only if needed. The scheme needs to deal with the cross compile problems and dependent so/dll/dylib that might not be in the Shoes location. I don't know how well rakecompiler can handle what I'm thinking of. Just don't know.

Contributor

ccoupe commented Feb 21, 2015

Yes, the gems should be (re)built only if needed. The scheme needs to deal with the cross compile problems and dependent so/dll/dylib that might not be in the Shoes location. I don't know how well rakecompiler can handle what I'm thinking of. Just don't know.

@BackOrder

This comment has been minimized.

Show comment
Hide comment
@BackOrder

BackOrder Feb 21, 2015

Collaborator

Rake provides rule and file rules that would allow to do something like that. It won't build the rule when the target of the rule is already present.

Collaborator

BackOrder commented Feb 21, 2015

Rake provides rule and file rules that would allow to do something like that. It won't build the rule when the target of the rule is already present.

ccoupe added a commit that referenced this issue Mar 31, 2015

more rakefile cleanup (hah!) for #70 and prep for #72
* Use TGT_ARCH from Rakefile to pick a target specific extconf.rb
  if it exists. Other wise use the monster ugly extconf.rb
* Should have done this months ago.
* Still need to invent/discover a way to delete existing RbCongfig::
  constants and reload
* had to modify .gitignore more better.

@ccoupe ccoupe added Enhancement and removed Wishlist labels Apr 3, 2015

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 3, 2015

Contributor

I'm going to start this effort but trying to get Nokogiri to build/cross compile and replace hpricot in Shoes and see what happens and learn what has to be done.

Contributor

ccoupe commented Apr 3, 2015

I'm going to start this effort but trying to get Nokogiri to build/cross compile and replace hpricot in Shoes and see what happens and learn what has to be done.

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 4, 2015

Contributor

Yikes! Nokogiri is complicated. I still haven't found the incantation to cross compile the windows binary gem. The good news is that Shoes Cobbler had no problem download and installing the binary gem on Windows. That wasn't working the last time @BackOrder and I tried.

It's important to note that they appear to have statically linked/copied libxml2 and libxslt1 - in Shoes we'd prefer to use the libxml2.dll that we distribute. Nokogiri (on Windows) also includes the nokogiri.so for 1.9.3, 2.0.0 and 2.1.0. Only one of which we could use and of course it includes all the source and doc so it's a huge gem.

They appear to include their own copy of libxml2 and libxslt1 source code (and patches to them) but I haven't found the incantation to build a gem with and x86_64_linux binary inside. I see hints that it can be done with difficulty. OSX? I'm even less sure how that would work w/o xcode installed. What I can see is a very difficult task ahead. Thinking about the Picky dependencies (and dll duplications) suggest it would be an even harder task.

There only two things that require built-in gems - samples/expert-funnies.rb and samples/simple-sqlite.rb. XML parsing and composing SQL statements from Ruby are not beginning programming topics. In my simple mind, if folks need to do those tasks from Shoes it's not too much to ask for them to install devkit. xcode or issue the apt-get/yum/zypper commands. Shoes already knows how use existing gems on a users system if told to. So, rather than enhance the built-in gems I'm leaning towards not including them.

For the rare situation when a developer wants to package a shoes app with binary gems of their choosing, there would be a way to do that in Shoes if the developer is willing to build the binary gem.

Contributor

ccoupe commented Apr 4, 2015

Yikes! Nokogiri is complicated. I still haven't found the incantation to cross compile the windows binary gem. The good news is that Shoes Cobbler had no problem download and installing the binary gem on Windows. That wasn't working the last time @BackOrder and I tried.

It's important to note that they appear to have statically linked/copied libxml2 and libxslt1 - in Shoes we'd prefer to use the libxml2.dll that we distribute. Nokogiri (on Windows) also includes the nokogiri.so for 1.9.3, 2.0.0 and 2.1.0. Only one of which we could use and of course it includes all the source and doc so it's a huge gem.

They appear to include their own copy of libxml2 and libxslt1 source code (and patches to them) but I haven't found the incantation to build a gem with and x86_64_linux binary inside. I see hints that it can be done with difficulty. OSX? I'm even less sure how that would work w/o xcode installed. What I can see is a very difficult task ahead. Thinking about the Picky dependencies (and dll duplications) suggest it would be an even harder task.

There only two things that require built-in gems - samples/expert-funnies.rb and samples/simple-sqlite.rb. XML parsing and composing SQL statements from Ruby are not beginning programming topics. In my simple mind, if folks need to do those tasks from Shoes it's not too much to ask for them to install devkit. xcode or issue the apt-get/yum/zypper commands. Shoes already knows how use existing gems on a users system if told to. So, rather than enhance the built-in gems I'm leaning towards not including them.

For the rare situation when a developer wants to package a shoes app with binary gems of their choosing, there would be a way to do that in Shoes if the developer is willing to build the binary gem.

@ccoupe ccoupe added Wishlist and removed Enhancement labels Apr 4, 2015

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe Apr 18, 2015

Contributor

I've managed to separate gem and extension building from building Shoes for targets xwin7, win7 and xarm6hf. Basic documentation at https://github.com/Shoes3/shoes3/wiki/Gems-for-Shoes. Commits will follow shortly. I still have to do OSX changes and testing to do but the design 'doesn't suck much'.

Then I can try to build nokogiri and picky and all it's deps - many, many many gems.

Contributor

ccoupe commented Apr 18, 2015

I've managed to separate gem and extension building from building Shoes for targets xwin7, win7 and xarm6hf. Basic documentation at https://github.com/Shoes3/shoes3/wiki/Gems-for-Shoes. Commits will follow shortly. I still have to do OSX changes and testing to do but the design 'doesn't suck much'.

Then I can try to build nokogiri and picky and all it's deps - many, many many gems.

ccoupe added a commit that referenced this issue Apr 18, 2015

Separate gems and shoes building #72 and #70.
* still has backwards/default just in case code
* expect changes - doesn't suck much as is.

ccoupe added a commit that referenced this issue Apr 19, 2015

rakefile OSX for seperate gems location/compile #72, #70
* start the move of common_build from make/*/tasks to make/common-build.rb
* differentiate between '.bundle' and '.so' for exts/gems

ccoupe added a commit that referenced this issue Apr 19, 2015

ccoupe added a commit that referenced this issue Apr 20, 2015

@ccoupe ccoupe added Normal and removed Wishlist labels May 6, 2015

@ccoupe ccoupe modified the milestone: 3.2.23 May 6, 2015

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe May 6, 2015

Contributor

This took a lot longer and more effort than I thought. It's a tolerable process, for patient and observant. It should be easier. Nokogiri-1.6.6.2 is included in Shoes 3.2.23 betas - trimmed to just the essentials to reduce bloat (a lot of that gem).

Updates https://github.com/Shoes3/shoes3/wiki/Gems-for-Shoes.

Contributor

ccoupe commented May 6, 2015

This took a lot longer and more effort than I thought. It's a tolerable process, for patient and observant. It should be easier. Nokogiri-1.6.6.2 is included in Shoes 3.2.23 betas - trimmed to just the essentials to reduce bloat (a lot of that gem).

Updates https://github.com/Shoes3/shoes3/wiki/Gems-for-Shoes.

ccoupe referenced this issue May 15, 2015

WIP: Build/load pre-compiled gems w/o needing dev tools for the
Shoe end user.  It's still clumsy but it works and the number of people
needing the feature is small.
@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe May 16, 2015

Contributor

I started the explanation for why I want this feature and think it is something that Shoes 3.2 (MRI Ruby) can do that is completely platform specific but oddly generic. https://github.com/Shoes3/shoes3/wiki/Gempacks

Contributor

ccoupe commented May 16, 2015

I started the explanation for why I want this feature and think it is something that Shoes 3.2 (MRI Ruby) can do that is completely platform specific but oddly generic. https://github.com/Shoes3/shoes3/wiki/Gempacks

ccoupe added a commit that referenced this issue May 20, 2015

Updating win7 and xwin7 for many things: #72, #124
* move to Ruby 2.1.6
* Windows ruby gotcha's
* Added Rake constant SHOES_GEM_ARCH (for xwin7 and xarm6hf need this)
* Accomodate central directory of gems (old skool like sqlite3) and newer
  gems (copied by CopyGem.rb or extract1gem.rb or built by elfs with magic)
* Added byebug and deps to built in gems. No one is forced to use them and
  it's a bit clunky to use and developers only and there could be OSX huh?
  -- just getting started.
* Most built-in Shoes gems are no longer in 'specications/default.
  Sqlite3 is still in between - but it works.
* expect similar changes for other platforms and sqlite

ccoupe added a commit that referenced this issue May 23, 2015

@ccoupe

This comment has been minimized.

Show comment
Hide comment
@ccoupe

ccoupe May 29, 2015

Contributor

I'm thinking of closing this issue unless folks think it should stay open. 3.2.23 does have new handling for gem building and distribution when building Shoes. Gem handling when users want to package gems or build gems from source is a different issue.

Contributor

ccoupe commented May 29, 2015

I'm thinking of closing this issue unless folks think it should stay open. 3.2.23 does have new handling for gem building and distribution when building Shoes. Gem handling when users want to package gems or build gems from source is a different issue.

@ccoupe ccoupe added this to the 3.2.23 milestone Jun 13, 2015

@ccoupe ccoupe closed this Jun 13, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment