Gradle/Git: Extract non-core plugins / allow optional plugins

[binwiederhier]

Right now, the master branch contains many plugins in the syncany-plugins directory, namely these plugins: syncany-plugin-ftp, syncany-plugin-sftp, syncany-plugin-webdav, syncany-plugin-rest.

Also, the plugins appear in the gradle configuration for the CLI (and the GUI): settings.gradle and in the dependencies of syncany-cli.

In the future, the number of plugins will grow, and I won't have enough time to maintain them all. That's why we need the possibility to add optional plugins from other github repos -- which can be hosted at the Syncany organization space, but maintained by others.

Something like git subtree or submodule could be used, but maybe something completely loosely coupled would be even better.

I'm looking for someone to implement this. Please let me know if you're interested.

[christf]

Whoever is implementing this should have a look at the build system
for openwrt and the Freifunk Firmware for Lübeck / Cologne or
Frankfurt. This system is highly configurable and enables the above
use case.

[binwiederhier]

I think adding another build system wouldn't be beneficial. Gradle does the job perfectly fine and I'm sure it's very capable to do this. I think the more interesting part is that the plugins depend on some classes in the syncany-lib (e.g. Connection, Plugin, TransferManager), and if they are not part of the same project (or git repo) dependencies might be more interesting.

Maybe let syncany-lib or syncany-plugin-api (does not exist yet) be a git subtree of syncany-plugin-{sftp,rest,...}. Something like that.

[binwiederhier]

To the lucky one who gets to develop this: Vincent has already done some work that could be reused: https://github.com/vwiencek/syncany-plugin-hazelcast

Related thread on the mailing list:

• https://lists.launchpad.net/syncany-team/msg00689.html
• https://lists.launchpad.net/syncany-team/msg00692.html

[binwiederhier]

This has to be done for 0.1, because SFTP and S3 shouldn't be in the first release.

[vwiencek]

Dear all,

1/ First Step
I created 3 new repositories on my github space "
github.com/vwiencek/syncany-plugin(rest|sftp|hazelcast)"

All these plugin repositories work with submodules, meaning that they
"fetch" a specific syncany branch (currently plugin-split) from a specific
repo (currently my syncany fork), and all of this can be defined into
.gitmodule file.

2/ Second Step
I removed rest|sftp plugins (and updated .gradle files) from syncany master
under a new "plugin-split" branch.
All you have to do is to merge plugin-split into master and the plugin will
disapear

3/ Todo

• find a convenient and easy way to use plugins from syncany master branch,
  with something like, "gradle add-plugin rest and gradle remove-plugin
  rest" for example. These tasks will invoke 'git submodule add|rm' and do
  the git stuff to fetch plugin
• find a way to define plugins to be included in release, fo example in a
  release.properties file containing a line like "plugins=rest, sftp,
  hazelcast, amazon ...."

Vincent

On Wed, Mar 5, 2014 at 10:45 PM, Philipp C. Heckel <notifications@github.com

wrote:

This has to be done for 0.1, because SFTP and S3 shouldn't be in the first
release.

Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-36797886
.

Vincent Wiencek
vwiencek@gmail.com

[binwiederhier]

Hi Vincent,

very nice. Two things:

1. My goal is to make the optional plugins downloadable on their own -- either as ZIP file (with readme and stuff), or as a simple jar (with dependencies). It should then be possible to add plugins by copying them to ~/.syncany/plugins/ or by calling something like "sy plugin add syncany-plugin-sftp-0.1.jar/zip". If we do it like this, a gradle task would not be necessary.
2. If you want, you can transfer the plugins to the syncany organization space; I'll give you ownership to the corresponding repos.

What do you think about that?

[binwiederhier]

I merged your branches into this ones here; @vwiencek @pimotte and @guitarlum should have full write access to them:

• https://github.com/syncany/syncany-plugin-rest
• https://github.com/syncany/syncany-plugin-sftp
• https://github.com/syncany/syncany-plugin-hazelcast

[binwiederhier]

I started the development of a PluginCommand in this branch: https://github.com/binwiederhier/syncany/tree/feature/easyplugin

Syntax will be sy plugin (list|rlist|get|activate|deactivate). If anyone wants to develop this, please shout out. There are many other things for me to do. Stability and such ...

[binwiederhier]

Talked to Pim on IRC; decided that the syntax should be different:

sy plugin list [--remote|--local|--all]
   Lists local plugins (active/inactive) and/or remotely available plugins 
   on api.syncany.org

sy plugin install <URL | local file | plugin-id>
   Installs a plugin from an arbitrary URL, local file or from the available plugins
   on api.syncany.org (with a plugin identifier)

sy plugin activate <plugin-id>
   Activates the plugin with the given plugin identifier

sy plugin deactivate <plugin-id>
   Deactivates the plugin with the given plugin identifier

The sy plugin install command could be used like this:

sy plugin install http://some-location.tld/syncany-plugin-s3.jar
sy plugin install localfile.jar
sy plugin install s3 --> gets from api.syncany.org

[binwiederhier]

I continued working on this. Commit a817fab now talks to http://api.syncany.org/v1/plugins/list.php (mock data), parses the response and can display it as shown below.

Obviously the sy plugin list command should display all plugins:

1. Active, i.e. loaded by the classloader (a) from lib/ dir (already done) and (b) from ~/.syncany/plugins/enabled/ (To Do)
2. Inactive, i.e. in ~/.syncany/plugin/available/ (To Do)
3. Remotely available (done)

pheckel@platop /tmp/a $ sy plugin list
Id       Name            Version        Status
----------------------------------------------------
ftp      FTP             0.1            inst/..
local    Local           0.1            inst/..
webdav   WebDAV          0.1            inst/..
s3       Amazon S3       0.1.0          inst/..
webdav   WebDAV          0.1.0          inst/..
sftp     SFTP            0.1.0          inst/..

pheckel@platop /tmp/a $ sy plugin list --remote-only
Id       Name            Version        Status
----------------------------------------------------
s3       Amazon S3       0.1.0          inst/..
webdav   WebDAV          0.1.0          inst/..
sftp     SFTP            0.1.0          inst/..

pheckel@platop /tmp/a $ sy plugin list --local-only
Id       Name            Version        Status
----------------------------------------------------
ftp      FTP             0.1            inst/..
local    Local           0.1            inst/..
webdav   WebDAV          0.1            inst/..

[pimotte]

Using ~/.syncany might give some namespace issues, since that is the same folder used if a user inits in ~. Though it doesn't cause any real issues afaik, would it be cleaner to use ~/.config/syncany instead?

[binwiederhier]

Agreed.

[binwiederhier]

So again, @pimotte opened my mind about things: We decided to not allow plugins to be activated/deactivated. Instead, "install" and "deinstall" should be enough. If anyone can think of a reason why we should allow deactivated plugins, speak now or forever hold your peace.

The new help file would look like this:

  plugin (list | install | uninstall) [<args>]
     Lists and manages available plugins. The following actions are 
     available:

     list [<args>]
       Lists locally installed plugins and/or remotely available plugins 
       on api.syncany.org

       Arguments: 
       -a, --all                     List all available plugins (default)
       -R, --remote-only             List remotely available plugins
       -L, --local-only              List locally available plugins

     install (<URL> | <file> | <plugin-id>)
       Installs a plugin from an arbitrary URL, local file or from the 
       available plugins on api.syncany.org (with a plugin identifier)

     uninstall <plugin-id>
       Uninstalls a plugin entirely (removes the JAR file). This action can
       only be used for plugins that were installed by the user, and not for
       system-wide plugins.

[binwiederhier]

I have another issue I would very much like feedback to.

After I have implemented the basics of the plugin installation, I am trying to compile an example for an external plugin. The SFTP plugin prepared by Vincent works code-wise, but there is currently no way to dynamically add it to Syncany.

The following things are necessary:

1. It must be possible to (a) create a single JAR file or (b) multiple JAR files (= including dependencies)
2. The plugin must be "auto-deployed" to the Syncany server (like the main project)
3. The Syncany shell/batch script must look for JARs not only in lib/, but also in ~/.config/syncany/plugins.

I don't see (2) as an issue, and (3) depends on (1). I am currently struggling with (1). In particular, I am not sure how to handle dependencies in plugins. There are two options:

(1a) Create a single JAR including all dependencies. This might be a bit tricky gradle-wise, but the 'uberjar' target already does that (lib/util dependencies need to be excluded though).

(1b) Create one JAR for the plugin without dependencies. This would have a couple of impacts:

• For sy plugin install <plugin-id>, this would mean that not only the plugin JAR would have to be downloaded, but also the dependencies (either from a maven repo, or from the Syncany server).
• Also, the dependencies must be available in the API structure
• To allow deinstallation of plugins, some sort of dependency management in ~/.config/syncany/plugins/ is necessary, or there must be sub-folders like ~/.config/syncany/plugins/sftp/. In the latter case, the classpath defined in the syncany/sy shell/batch script would have to look for JARs in all subfolders (non-trivial gradle/shell-script magic).

In short, in (1a) the sy plugin install sftp command does this:

• Get plugin info from api.syncany.org
• Download http://.../syncany-plugin-sftp-0.1.0.jar and place it in ~/.config/syncany/plugins/syncany-plugin-sftp-0.1.0.jar
• The shell script's classpath looks like this: java -cp "...:~/.config/syncany/plugins/*".

For (1b), the sy plugin install sftp command does this:

• Get plugin info from api.syncany.org (including dependencies
• Download http://.../syncany-plugin-sftp-0.1.0.jar and place it in ~/.config/syncany/plugins/sftp/syncany-plugin-sftp-0.1.0.jar (subfolder!)
• Download other dependencies from Maven repos and place them in ~/.config/syncany/plugins/sftp/<some-jar>.jar
• The shell script's classpath must include subfolders of ~/.config/syncany/plugins/*/* (non-trivial, for me)

[pimotte]

The final bullet point: Perhaps something like in: https://github.com/binwiederhier/syncany/blob/develop/gradle/arch/syncany/syncany

However, my intuition is in favour of 1a. It reduces complexity and I can't imagine that plugins being so large that space becomes an issue. Is the .jar in syncany.org/dist/plugins/ftp the uberjar or the normal one?

[vwiencek]

ok maybe one point regarding class loading issues (waiting for java
jigsaw): what happens if two plugin reference two distinct versions of the
same library ?

ex :

• ftp depends on liba-1.0.jar
• sftp depends on liba-2.0.jar

I'm not sure whether java 7 or 8 can handle such cases ....
The only solution would be to closely look at plugin dependencies no ?

On Mon, Apr 14, 2014 at 9:59 AM, pimotte notifications@github.com wrote:

The final bullet point: Perhaps something like in:
https://github.com/binwiederhier/syncany/blob/develop/gradle/arch/syncany/syncany

However, my intuition is in favour of 1a. It reduces complexity and I
can't imagine that plugins being so large that space becomes an issue. Is
the .jar in syncany.org/dist/plugins/ftp the uberjar or the normal one?

Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-40341489
.

Vincent Wiencek
vwiencek@gmail.com

[vwiencek]

Instead of hacking shell scripts, on other solution could be to write a
custom ClassLoader which could
1/ test whether the jar is present in the .config/syncany/libs
2a/ if yes, then nothing,
2b/ if no, then download from api.syncany.org, put it in
.config/syncany/libs
3/ then load the jar

what do you think ?

jcl tool : https://github.com/kamranzafar/JCL/

On Mon, Apr 14, 2014 at 10:04 AM, Vincent Wiencek vwiencek@gmail.comwrote:

ok maybe one point regarding class loading issues (waiting for java
jigsaw): what happens if two plugin reference two distinct versions of the
same library ?

ex :

• ftp depends on liba-1.0.jar
• sftp depends on liba-2.0.jar

I'm not sure whether java 7 or 8 can handle such cases ....
The only solution would be to closely look at plugin dependencies no ?

On Mon, Apr 14, 2014 at 9:59 AM, pimotte notifications@github.com wrote:

The final bullet point: Perhaps something like in:
https://github.com/binwiederhier/syncany/blob/develop/gradle/arch/syncany/syncany

However, my intuition is in favour of 1a. It reduces complexity and I
can't imagine that plugins being so large that space becomes an issue. Is
the .jar in syncany.org/dist/plugins/ftp the uberjar or the normal one?

Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-40341489
.

Vincent Wiencek
vwiencek@gmail.com

Vincent Wiencek
vwiencek@gmail.com

[binwiederhier]

@pimotte

• shell script: Something like that shell script, yes, but added to the prerolled Gradle-shell script via a gradle task.
• JARs: The JAR on the server is without dependencies. I's 8 KB after all :-). I uploaded one of the uberjars I created. It includes all the dependencies, including the syncany-lib and syncany-util dependencies (so all of Syncany + the SFTP plugin depenencies). If you look inside, it looks really messy. Even though it might be easier to manage, I am actually in favor of having separate JARs for the depencies.

@vwiencek

• liba-1.0 vs liba-2.0: I think we won't ever be able to avoid that unless we reinvent dependency management to Syncany. Plugins might be incompatible. I don't think we can avoid that.
• The custom class loader idea looks interesting; maybe that is indeed the way to go. Instead of downloading dependencies on demand, I would still do it during sy plugin install though. The class loader would just load all JARs in ~/.config/syncany/plugins/ (including subdirs). For by (1b) solution, this would only solve the "hacking the shell script part though. We'd still need to download the dependencies from the maven repo.

[binwiederhier]

So after a discussion with @pimotte we decided that it'd be easier for now to have a single JAR. I prepared the SFTP plugin to create a JAR including the plugin dependencies.

It's not quite as automatic as I had wished. The dependencies for the JAR have to be manually added as a configuration in the build file.

Also, the plugins can have their own version, which is essential I think.

I also added a description for plugin development here:
https://github.com/binwiederhier/syncany/wiki/Plugin-development

Using the easyplugin branch, downloading sy plugin install sftp works already 👍

I also managed to adjust the classpath: https://github.com/binwiederhier/syncany/blob/69203310c4b11def96c3ff240fd92c8d2999795e/syncany-cli/build.gradle#L39

Reviews and feedback of any kind are welcome!

[binwiederhier]

I think we're almost there.

 $ sy plugin list
Id       Name            Local Version  Remote Version  Inst.  Upgr.
---------------------------------------------------------------------------------
ftp      FTP             0.1            0.1.1-alpha     yes    yes  
local    Local           0.1                            yes         
s3       Amazon S3                      0.1.0                  yes  
sftp     SFTP                           0.1.0-SNAPSH..         yes      <<<< NOT INSTALLED
webdav   WebDAV          0.1            0.1.0           yes    yes  
pheckel@platop ~/s $ sy plugin install sftp
pheckel@platop ~/s $ sy plugin list
Id       Name            Local Version  Remote Version  Inst.  Upgr.
---------------------------------------------------------------------------------
ftp      FTP             0.1            0.1.1-alpha     yes    yes  
local    Local           0.1                            yes         
s3       Amazon S3                      0.1.0                  yes  
sftp     SFTP            0.1            0.1.0-SNAPSH..  yes    yes    <<<< INSTALLED
webdav   WebDAV          0.1            0.1.0           yes    yes  
$ sy init
Choose a storage plugin. Available plugins are: ftp, local, sftp, webdav <<<< AVAILABLE!
(...)

pheckel@platop ~/s $ sy plugin uninstall sftp
pheckel@platop ~/s $ sy plugin list
Id       Name            Local Version  Remote Version  Inst.  Upgr.
---------------------------------------------------------------------------------
ftp      FTP             0.1            0.1.1-alpha     yes    yes  
local    Local           0.1                            yes         
s3       Amazon S3                      0.1.0                  yes  
sftp     SFTP                           0.1.0-SNAPSH..         yes      <<<< NOT INSTALLED
webdav   WebDAV          0.1            0.1.0           yes    yes  

[vwiencek]

nice :)

On Tue, Apr 15, 2014 at 9:27 AM, Philipp C. Heckel <notifications@github.com

wrote:

I think we're almost there.

$ sy plugin list

Id Name Local Version Remote Version Inst. Upgr.

ftp FTP 0.1 0.1.1-alpha yes yes
local Local 0.1 yes
s3 Amazon S3 0.1.0 yes
sftp SFTP 0.1.0-SNAPSH.. yes <<<< NOT INSTALLED
webdav WebDAV 0.1 0.1.0 yes yes
pheckel@platop ~/s $ sy plugin install sftp
pheckel@platop ~/s $ sy plugin list

Id Name Local Version Remote Version Inst. Upgr.

ftp FTP 0.1 0.1.1-alpha yes yes
local Local 0.1 yes
s3 Amazon S3 0.1.0 yes
sftp SFTP 0.1 0.1.0-SNAPSH.. yes yes <<<< INSTALLED
webdav WebDAV 0.1 0.1.0 yes yes
$ sy init
Choose a storage plugin. Available plugins are: ftp, local, sftp, webdav <<<< AVAILABLE!
(...)

pheckel@platop ~/s $ sy plugin uninstall sftp
pheckel@platop ~/s $ sy plugin list

Id Name Local Version Remote Version Inst. Upgr.

ftp FTP 0.1 0.1.1-alpha yes yes
local Local 0.1 yes
s3 Amazon S3 0.1.0 yes
sftp SFTP 0.1.0-SNAPSH.. yes <<<< NOT INSTALLED
webdav WebDAV 0.1 0.1.0 yes yes

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-40453002
.

Vincent Wiencek
vwiencek@gmail.com

[binwiederhier]

(1) Download/install from syncany.org/dist

pheckel@platop ~/s $ sy plugin install sftp    
Plugin successfully installed from https://www.syncany.org/dist/plugins/snapshots/sftp/syncany-plugin-sftp-0.1.0-SNAPSHOT+1404152304git8bd0176.jar
Install location: /home/pheckel/.config/syncany/plugins/syncany-plugin-sftp-0.1.0-SNAPSHOT+1404152304git8bd0176.jar

Plugin details:
- ID: sftp
- Name: SFTP
- Version: 0.1.0-SNAPSHOT+1404152304git8bd0176

pheckel@platop ~/s $ sy plugin remove sftp
Plugin successfully removed.
Original local was /home/pheckel/.config/syncany/plugins/syncany-plugin-sftp-0.1.0-SNAPSHOT+1404152304git8bd0176.jar

(2) Install from local file

pheckel@platop ~/s $ sy plugin install syncany-plugin-sftp-0.1.0-SNAPSHOT+1404152304git8bd0176.jar 
Plugin successfully installed from syncany-plugin-sftp-0.1.0-SNAPSHOT+1404152304git8bd0176.jar
Install location: /home/pheckel/.config/syncany/plugins/syncany-plugin-sftp-0.1.0-SNAPSHOT+1404152304git8bd0176.jar

Plugin details:
- ID: sftp
- Name: SFTP
- Version: 0.1.0-SNAPSHOT+1404152304git8bd0176

pheckel@platop ~/s $ sy plugin remove sftp
Plugin successfully removed.
Original local was /home/pheckel/.config/syncany/plugins/syncany-plugin-sftp-0.1.0-SNAPSHOT+1404152304git8bd0176.jar

(3) Install from any URL (!)

pheckel@platop ~/s $ sy plugin install https://www.syncany.org/dist/plugins/snapshots/sftp/syncany-plugin-sftp-0.1.0-SNAPSHOT+1404152304git8bd0176.jar
Plugin successfully installed from https://www.syncany.org/dist/plugins/snapshots/sftp/syncany-plugin-sftp-0.1.0-SNAPSHOT+1404152304git8bd0176.jar
Install location: /home/pheckel/.config/syncany/plugins/syncany-plugin-sftp-0.1.0-SNAPSHOT+1404152304git8bd0176.jar

Plugin details:
- ID: sftp
- Name: SFTP
- Version: 0.1.0-SNAPSHOT+1404152304git8bd0176

[binwiederhier]

This is finally ready to be merged. Any comments?
develop...feature/easyplugin

I'll merge it tonight unless you guys have any comments.

[pimotte]

Nicely done, no objections to merging.

Minor-minor nitpick: PluginCommand, line 91. The comment in the middle of the if/else if construction threw me off slightly. I think it's cleaner to move that one into the elseif-clause.

[binwiederhier]

Sounds good. I'll merge it then, and externalize the webdav plugin.

Regarding the "comment-before-else-if" thing: I've always done it this way, but Steffen also told me once that it's odd. Maybe I have to change my habits...