Proposition: to separate addons from Io

[ales-tsurko]

Hi Steve! As I can see the main problem with building Io are its dependencies. When some of addons won't build, a user didn't get Io too. Also not all the addons equally important for all users. It would be better to allow user to install what he/she needs. So why don't separate the addons from Io? We could convert the addons into Eerie packages. And make a repository for the each one. A user then could install what (s)he needs with Eerie. This also would make the Io's build process a lot easier, reduce dependencies and make Io itself smaller. And I'm not speaking about such features as addons updates and environments.
Moreover, Eerie not only makes it simple to install Io addons and all of its dependencies, but it's also has a Docio — the documentation generator for the packages. Docio generates documentation from the same comments format, that is already used in addons, but also supports markdown. After the generating process it provides you with a site for a package (this site is generated by Docio, for example: http://tonikasoft.com/docio/) and methods to look up for an addon documentation right in the Io's interpreter.
I could convert all the addons into packages right after we'll finish fixing cross platform problems (there are not many).
This also assumes that Eerie will be the official Io's package manager, that will be installed with Io itself.
What do you think?

[stevedekorte]

Hey Ales, I'm open to patches that would make Io easier to install. I'm also open to alternative ways to generate docs as long as they are in the same format as the current docs.

[ales-tsurko]

Great! Could you then make repositories for the each of the addons? Then I could fork them and make PRs to make them packages. Or you could make an organization for the addons (and, maybe, Io itself) and temporally allow me rights to create repositories in there.
Otherwise I should make it under https://github.com/tonikasoft profile.

[ales-tsurko]

The repositories may not contain anything — I'd copy the sources myself. But they should be properly named (had the same names as in the addons directory).

[stevedekorte]

Why are separate repos required?

[ales-tsurko]

Eerie keeps the package URL so you can update the package. It's also would be simpler to install packages separately.
For example, I'd like to install AVCodec addon, I'd do it so:
eerie install https://github.com/AVCodec.git
Then if I'd like to update it, I'd just call:
eerie update AVCodec
And Eerie would download the updated code and will recompile it if needed.
Not only Git supported, but also svn, mercurial, archives and local directories. But there is a drawback with the last one (local packages). They are good for development or, for example, to convert some Io project into a package. But if we ship the Io's source code with all the addons (so they all be installed from local directories), then in order to update some addon, the user should pull all the changes from the main repository (for all the addons and Io itself) and only after that (s)he could call eerie update for a specific addon. This is because eerie will update the addon using installed URL, that will be a local directory in this case.

A separate repository is just its own place. It makes it independent from the other addons and Io's sources.
Don't worry — if a package depends on another package, eerie will fetch and install it automatically from its repository. And there will not anymore a situtation when to compile Io, a user would have to install dependencies for all of the addons.

There are also other benefits, for example all the issues regarding specifical packages goes to each own repository. It would be also simpler for contributors to work on a specific addon on a separate repository.

[ales-tsurko]

That also corresponds to "encapsulate what varies" principle in some kind.

[ales-tsurko]

I mean to allow a user to choose which addons to install. And by default there would be not many addons — just those, that required. After installation a user could install any additional addons (s)he needs (or don't install anything). But if you'd like it's not so hard to add all of them as dependencies, so they all will be installed, in this case Eerie at least would allow a user to remove what (s)he doesn't need.

Eerie packages have the same directory structure as the existing addons, except of the package manifest file (package.json). So it'll be not so hard to convert them. Eerie also uses the good old (and fixed) AddonBuilder to build native packages that makes it independent of and simpler to use than Cmake.

Packages could contain an executable scripts, so they could be installed as apps. For example, jasmineio package is shipping with jasmineio command line tool: https://github.com/bekkopen/jasmineio
In this sense Eerie is more like homebrew for Io — not just an "addon builder and updater". That means that Eerie is also a manager for apps written in Io (i.e. a package could be not just an addon, but a whole application or game).

There are a lot of benefits of this design. Do you see any drawbacks?

[ales-tsurko]

While fixing one of the addons, I've found yet another benefit: there is no need to recompile and reinstall Io with Eerie — just the package:)

[stevedekorte]

Ales, Sorry for the delay responding. This sounds promising.

One concern I have is if the API for addons changes, it would require dealing with a lot of different repos to update everything. I don't see that happening though, so maybe it's not an issue.

Another concern is that it would clutter my git repo namespace, but maybe it's time to move Io to a separate Io github organization. That might also make granting privileges to maintainers easier too. I'll look into it.

[ales-tsurko]

Even if there are chances that API will change in the near future, the repos are just like the separate directories in the addon directory in that case. The only difference is that all they should be cloned to local directory, but this task could be automated and after that the potential work would be the same as if that happened now.

+1 for organization! I could make all the work with addons then, if you allow me rights to create repositories in there.

[stevedekorte]

I've set up an https://github.com/IoLanguage and made you a member. You should have permissions to create repos for the addons. Want to give that a try?

[ales-tsurko]

Hooray!
BTW, github has an option to move an existing repo to an organization. It's somewhere in settings.

[ales-tsurko]

The plan is next:

1. Right now I'll create empty repos for addons so I could fork and change them into Eerie packages;
2. I'll make a fork of Io, that configured to compile without the addons, but with Eerie;
3. I'll convert all the addons to Eerie packages.

After the last step we could merge the Io's fork.

[stevedekorte]

"BTW, github has an option to move an existing repo to an organization. It's somewhere in settings."

Ok, I'll try to move the repo.

[stevedekorte]

Moved.

[ales-tsurko]

Partly done second step. Io compiles without addons except of those needed by Eerie. The fork I'm working on is here: https://github.com/AlesTsurko/io
I'd like to reduce default dependencies. I'll try to make it independent of Regex (PCRE). Than the only dependency would remain is Yajl...

[ales-tsurko]

Working on this ticket now to make it independent of Yajl.

[ales-tsurko]

Hi @stevedekorte
Sorry for the delay, I'm a little busy at work now.
The second step is almost done. I just would like to know your opinion about how to add Eerie as a dependency. There are few options:

1. Make it independent at all and just give the instructions how to grab (git clone https://path-to/eerie.git) and install it (cd eerie && io setup.io) in order to get additional packages;
2. Add it as a submodule to the Io's repository and also add instructions how to install it (cd eerie && io setup.io);
3. Automatically run Eerie setup script after installation of Io.

The disadvantage of the second option is that it adds an additional step in the Io installation process.

The disadvantages of the third option are:
to install just the core part of Io without any additional packages a user have to modify CMakeLists.txt;
to develop Eerie a user also have to modify CMakeLists.txt to make it possible to install Eerie independently of Io.

Add a Cmake flag to install (or do not) Eerie is also an option, but for me it also looks like an additional step. Moreover I guess a lot of developers don't even look at additional flags when building with CMake:)

[ales-tsurko]

Or I could add a CMake flag to DO NOT install Eerie. That is install it automatically by default, but give an option to install it manually or don't install it at all.

[stevedekorte]

Hey Ales, I'll leave this up to you. I'd just suggest minimizing the amount of effort and knowledge needed by a new user to get things working. In this case would you say that's the 3rd option?

[ales-tsurko]

Yes. And I'll add a flag to make it possible to compile without Eerie. So it will be the 2nd (install manually) and 3rd (install automatically) option at the same time.

[ales-tsurko]

The second step is done. Eerie added as a git submodule and installs by default after the Io installation, but I've also added a flag WITHOUT_EERIE to make it possible to install Io without Eerie if needed.

I need to fix some issues on Windows for Eerie, I hope I'll do it during the next week. After that I'll start to convert addons into packages.

By the way the CMakeLists.txt was already configured for CPack — that makes it simple to compile Io into a binary package for distribution, it could be used to update binary releases regularly (for even simpler installation of Io).

[ales-tsurko]

Sorry for the delay. Work + music — I was really busy during the last few weeks. I've done some steps to take Eerie onto Windows. But "thankfully" to the platform there's a lot of restrictions on the former. The first one is about executable scripts. We haven't such a feature on Windows by default (so we should run all the scripts with io command, giving a script as an input argument). The second problem is a lot of available compilers/environments (so I finished with shipping some dependencies with binaries). In overall, the Windows out-of-the-box is awful platform for development. But there is WSL and Cygwin...
I'm using mingw-w64 now and will do my best to support "out-of-the-box" Windows as fully as I can, but I can't imagine a developer on Windows who wouldn't use WSL (or Cygwin) to be honest. Also there will be platform-related restrictions, which will make Eerie not so much usable as on macOS and Linux. But there're no any restrictions with WSL, of course. So I would depend on the last options (WSL/Cygwin) on Windows and forget about anything else. Would you agree?
After resolving of these we could merge my fork with upstream, but we can use a new branch for that until I'll convert the addons. I'd also move Eerie repo (and it's dependencies) to the IoLanguage organization. Would you mind?
BTW, Io itself hadn't compiled on Windows, but I've fixed it.
You can view the progress of work here: https://github.com/AlesTsurko/io
Eerie lives here: https://github.com/tonikasoft/eerie

[stevedekorte]

"So I would depend on the last options (WSL/Cygwin) on Windows and forget about anything else. Would you agree?"

Yeah, that seems reasonable.

"I'd also move Eerie repo (and it's dependencies) to the IoLanguage organization. Would you mind?"

Sounds good. Just send me any instructions for what you need done on my side.

"BTW, Io itself hadn't compiled on Windows, but I've fixed it."

Great. Looking good.

[ales-tsurko]

"I'd also move Eerie repo (and it's dependencies) to the IoLanguage organization. Would you mind?"

Sounds good. Just send me any instructions for what you need done on my side.

I just moved Eerie and its dependencies myself. It's already on IoLanguage.

[ales-tsurko]

Hey Steve! Eerie finally compiles and works on Windows! Hurray! The only thing I'd like to fix before I'll dive into the addons conversion process is System ioPath method — it returns wrong value on Windows and now I'm rely on System installPrefix .. "/Io" which is not great idea... But anyway after we've done so far it's just a little step to finish. I hope to fix that today or tomorrow.

[stevedekorte]

Alex, nice work! Looking forward to it.

[ales-tsurko]

I've fixed the bug related to System installPrefix on Windows and I'm going to start converting addons tomorrow.

[ales-tsurko]

Hi Steve. Glad to inform you, that all the addons have been converted to Eerie packages. Moreover I did it manually, so I've checked the each one. The new-old system brings back some addons, that are commented in CMakeLists.txt. And I fixed some of the addons during the conversion. I also deleted the Postgres, because it looks like that was a duplicate of PostgreSQL.

Here is the full report:

• AVCodec [won't compile]
• AppleSensors [works]
• AsyncRequest
• Beanstalk
• BigNum
• Bitly
• Blowfish
• Box
• CFFI [fixed]
• CGI
• Cairo [compiles]
• Clutter [partly fixed]
• ContinuedFraction
• Curses
• DBI
• DistributedObjects
• EditLine
• Facebook
• Flux
• Fnmatch
• Font
• GLFW
• GoogleSearch
• HttpClient
• Image
• LZO
• LibSndFile
• Libxml2
• Loki
• Loudmouth
• MD5
• Memcached [works]
• MySQL [fixed]
• NetworkAdapter [fixed]
• NotificationCenter
• NullAddon
• ODE [compiles]
• Oauth [compiles]
• ObjcBridge [compiles]
• Obsidian
• Ogg
• OpenGL [fixed bugs that didn't allow to init, but anyway doesn't work due to problems with glut on macos]
• PortAudio [compiles and runs, but may have problems]
• PostgreSQL
• Postgres [removed — duplicate of PostrgreSQL]
• Python
• QDBM
• Random
• Range
• Rational
• ReadLine
• Regex
• SGML
• SHA1
• SQLite [didn't tested (can't compile with sqlite2)]
• SQLite3
• SampleRateConverter [fixed]
• SecureSocket
• SkipDB
• Socket
• SoundTouch [fixed build on macOS]
• SqlDatabase
• Syslog
• SystemCall
• TagDB
• TagLib [works]
• Theora
• Thread
• TokyoCabinet
• Twitter
• UUID
• User
• VertexDB
• Volcano
• Vorbis
• Yajl
• Zlib

I'm going to make the PR with Io changes now.

[stevedekorte]

Nice work Alex :)

[ales-tsurko]

I've opened the pull request: #411