Initial impressions/questions: Default FAQ, ssh urls,

[yarikoptic]

I am going through the https://web.gin.g-node.org/G-Node/Info/wiki/InHouseWithoutDocker . Thank you for the nice documentation -- it seems has worked out for me quite splendidly! Here I want to outline a few points which I have observed and which you might (or might not) want to address or just explain (decided not to separate into separate issues for now, added empty checkboxes to signal "something to address"). Some issues might relate more to upstream gogs .

• There is no default FAQ page (http://localhost:3000/G-Node/Info/wiki/FaqTroubleshooting gives 404 page)
• no options to initialize git-annex while creating a repo In above FAQ I was hoping to find on how to initialize "git annex" repository. I could not find an obvious setting somewhere while initializing a new repo or after to invoke "git annex init" on the server side. If such settings group existed (during creation) would be nice to be able to state desired backend for the annex (e.g. MD5) to partially mitigate those problems under windows. E.g. in datalad, by default, we just now use md5 backend:

$> datalad create /tmp/test; cat /tmp/test/.gitattributes
[INFO   ] Creating a new annex repo at /tmp/test 
create(ok): /tmp/test (dataset)                                                  
* annex.backend=MD5E

which should be sufficient for any scientific needs.

• incomplete ssh urls I have configured to not use builtin ssh. ssh urls provided by web interface "aren't complete" and rely on ssh account having entering the shell at the level of ~/gogs-repositories. I guess, whenever no internal ssh server is configured, I guess ssh urls need to be adjusted to reflect that . Anyways, I guess I better need to try the deployment with builtin ssh server, since I guess otherwise no multiple users would be able to push data
• http://... does not support git-annex I "git annex init"ed server side repo by adding an ssh remote, and letting git annex sync to init it, which doesn't work since

[Macaron] 2018-01-23 11:15:16: Started GET /yarikoptic/test1.git/config for [::1]
[Macaron] 2018-01-23 11:15:16: Completed GET /yarikoptic/test1.git/config 404 Not Found in 4.42299ms

i.e. git-annex trying to access that /config but it is not "interfaced" by the http/web interface. I wondered if it might become sufficient to make /config provided for git-annex so it could deduce uuid of that remote repo and request content straight via http?

I guess some of those issues are addressed via custom handling in gin cmdline client, but I thought it might be nice if GIN could work via native "git-annex" commands

If no objections, I will keep adding notes here as I go along. And please take those above points not as a critic of some kind, and just as my humble attempt on feedback to make GIN even better ;)

[cgars]

Hey there
The "without docker documentation" is kind of experimental and work in progress (hence it's not directly linked as you probably already noticed). That of course also means your comments are even more welcome and the fact that you try to deploy without docker makes it way more interesting to really work on the Tutorial again...

Let me shortly comment on some of the points:

In principle the use of the vanilla gin repo is discouraged for a lab deployment, rather we have a special branch for it (gin@home). That's the one we base the docker upon... It's not mentioned in the without docker documentation and it really should...

Regarding deployment and ssh related issues: A lot of it is addressed in the docker build, we have just not written it for the "deploy without docker" yet (hence the tutorial is not linked).

We have no annex init option so far, however, it's an interesting idea. We also don't support annex via HTTP currently. It's on my list of things that would be nice, however.

[yarikoptic]

Thank you @cgars
Re "not directly linked" -- I got to it through a "direct link" (and thank for it!) from https://web.gin.g-node.org/G-Node/Info/wiki/InHouse
re "special branch" -- that one seems to have a completely separate history although some (majority) commits are cherry-picked ... I just wondered - how is it different? ;)
re ssh -- I will check out the docker build script to see what is done, thanks!

[cgars]

Re "not directly linked" -- I got to it through a "direct link" (and thank for it!) from https://web.gin.g-node.org/G-Node/Info/wiki/InHouse

Oh, my. Well, it should at least say work in progress then. I guess I'll give it some love later this week, including the steps for ssh direct setup, which are briefly:

• create an ssh user (eg. git)
• make sure its .ssh folder is linked or at the right place and
• contains an authorized_keys file to which the service can write
• permissions

currently, the gin@hoime branches start.sh might be a good starting point for an overview...

In any case i am glad that the Tutorial it did help to some degree though!

re "special branch" -- that one seems to have a completely separate history although some (majority) commits are cherry-picked ... I just wondered - how is it different? ;)

It's basically a cherry-picked version of master, not including the GnodeGin specific things (eg. Links to wikis, legal stuff, functions that require or imply other deployed services etc. ). Simply speaking its a GIN/gogs Server supporting annex and meant for a lab deploy.

Thank for checking back and commenting. Ill value the time you invest for it!

[c42f]

Hi, I'm just starting with data versioning (having been a long time git user for code) and I like what I see in both gin and datalad. Out of the data curation systems I've seen, I think these are on the cutting edge and are fundamentally the right thing. I thought I'd leave some first impressions of my own here, I hope that's ok!

A big problem I have is to allow both power users and non-technical users collaborate on the same dataset. This is really hard but I hope might be doable with a combination of datalad and gin for each of these user groups respectively. In particular, gin-cli and gin/gogs seem tantalizingly close to being simple enough for the average user. I'm considering making a GUI to complement gin-cli for this purpose.

So, first impression of gin-cli and g-node/gogs is that you've really put effort into making the model simple, but building on powerful tools in the backend. This is great.

First impression of datalad is that it's tackling some ambitious and difficult problems on the power user side which I'd also like a story for: efficient nested datasets, data provenance, metadata extraction and search. And doing a really good job of creating a sensible model for these. (But I won't go on further about that, as this is a g-node repository.)

Back to g-node/gogs - I've run a test instance of this and it seems to work quite well. However, on a practical level we probably need OpenID connect so people can log in using ORCID or other identities. From that point of view I'm considering trying a plain gitea instance (with patch for git-annex support) and seeing whether that's usable. Overall the development velocity at gitea looks higher than gogs at this stage - have you considered porting your work there, and whether it's vaguely doable in a reasonable time?

Overall I really like these where these projects are going. Thanks for making them.

CC @aswinnarayanan

[yarikoptic]

@c42f you should also try datalad run whenever you run anything ;)
I didn't know about gitea. So there is some "pending"/circulating patch for git-annex support there?

[c42f]

Regarding gitea - yes there's a pending patch for git-annex support, see go-gitea/gitea#3786. We haven't tried it with gin or datalad yet, but it looks like it's mainly held up by issues with the testing harness.

[achilleas-k]

Hi Chris,

Thanks for reaching out. We very much appreciate the feedback!

A big problem I have is to allow both power users and non-technical users collaborate on the same dataset. This is really hard but I hope might be doable with a combination of datalad and gin for each of these user groups respectively. In particular, gin-cli and gin/gogs seem tantalizingly close to being simple enough for the average user. I'm considering making a GUI to complement gin-cli for this purpose.

This was a big part of the goal for gin-cli and GIN in general. We really want to provide a service that's friendly to the average user without limiting the power user. It's a very common goal for code and data management software and I'm sure we've all seen hundreds of examples of projects trying to strike this balance, to varying degrees of success. I've come to greatly appreciate the difficulty of this a lot more than I used to.

I'm considering making a GUI to complement gin-cli for this purpose.

We recently had a GUI frontend developed for Windows that has yet to see a proper release. It supports the core operations (clone, push, pull) which is what we wanted for an initial release. It's in a state where we need to test it more thoroughly before sending it out into the world.

https://github.com/G-Node/GinUI

If you're considering a cross-platform or Linux GUI, let us know how you're thinking about it. I don't want to promise too much, but there's a plan in place to split out a lot of the internals of gin-cli into a client library which more projects can use. Even in the current state however, the gin-cli is mostly written to accommodate this and I've been planning to look into exporting these internals for use in C code, so something like a GUI can just be built as a shell around the gin-cli internals (instead of shelling out for instance). If you were to start thinking about this, I'd certainly see if I could move this plan ahead a bit faster.

So, first impression of gin-cli and g-node/gogs is that you've really put effort into making the model simple, but building on powerful tools in the backend. This is great.

That's great to hear. Thanks!

Back to g-node/gogs - I've run a test instance of this and it seems to work quite well. However, on a practical level we probably need OpenID connect so people can log in using ORCID or other identities. From that point of view I'm considering trying a plain gitea instance (with patch for git-annex support) and seeing whether that's usable. Overall the development velocity at gitea looks higher than gogs at this stage - have you considered porting your work there, and whether it's vaguely doable in a reasonable time?

We haven't considered gitea. Would be interesting to see how far it's been forked from gogs and what's been added since.

Overall I really like these where these projects are going. Thanks for making them.

Thanks again for checking it out and all the feedback and comments!

[c42f]

We recently had a GUI frontend developed for Windows [...]

This is great news and exactly what I was looking for. I hadn't thought it through thoroughly, but my thoughts were split between:

• A cross platform GUI, probably based on the go Qt wrapper, vs
• Native windows explorer integration - ideal for the local end users.

Naturally I was expecting to base any GUI work off gin-cli as a backend. I think leaving linux desktop users to the gin command line would be fine for now.

Anyway, I should check out GinUI, I'm not sure how I missed that.

[achilleas-k]

Anyway, I should check out GinUI, I'm not sure how I missed that.

It's pretty easy to miss, I guess. We haven't properly announced it yet and it's not mentioned anywhere in the docs. That should be changing soon.

• Native windows explorer integration - ideal for the local end users.

That was part of our original plan as well, but evolved a little differently. GinUI¹ has two parts:

1. a service that lives in the system tray for logging in and cloning repositories.
2. context menus for uploading/downloading and fetching the content of annexed data.

We also considered file browser extensions as you mentioned (Dolphin services and Nautilus scripts, for instance), but other features kept getting higher priority.

¹ I should note that the name of the project might change before it's finally released. Not an important point, but thought you should know since you're checking it out.