dot abapgit: requirements to other git repositories

[larshp]

some package manager style functionality?

[larshp]

or requirements that specific objects must exist in the system

[larshp]

also see https://github.com/abramsba/APM-SAP-Client

[abramsba]

I'll jump into slack sometime in the coming days to discuss the details but it's been quiet with APM as of lately. Here is a documentation explaining how to use that repository, but it requires an older and custom version of abapGit and was designed with an ecosystem in mind (not using SAP crap as dependencies but just other packages).

https://docs.google.com/document/d/1h4o3FpSEafZLJPflGmru2okN1lT1EIYCux_PeUKTJpQ/edit?usp=sharing

[larshp]

@flaiker suggested that the PINF object can be used for dependency management, this seems like a good idea.

A key is needed in order to know if the required repo is installed, the URL will not work as it is changed while forking, plus the URL will not work for offline projects.

Add the package interface names in https://github.com/larshp/abapGit/blob/master/docs/repos.json

And add required package interfaces in .abapgit.xml

[larshp]

http://dotabap.org

[larshp]

Overall description

It should be possible to define dependencies for a repository. When a repository is installed it will check that the dependencies are already installed, and help the user installing these if needed.

Design considerations

• Should work with forks. User forks repository A and installs the fork. User want to install repostiory B which has a dependency to A, this should be okay since a fork of A is already installed.
• Offline/zip support.
• No versioning, ABAP only supports one active version of an object, so users are forced to use the latest version.

Design suggestion

Add a list of

• Unique identifier
• Git URL
• branch/tag

in the .abapgit.xml file

The unique identifier will be used to determine if a dependency is installed or not, ie. it must be persisted along with the repo. Suggestion is to use package interfaces for the unique identifier, eg. ZABAPGIT package interface as part of the abapGit repo.
Git URL + branch will be used to guide the user for the online scenario.
dotabap.org already checks uniqueness of all objects across known open source projects, this will also check the uniqueness of the package interfaces.

[fabianlupa]

Seems good to me!
Another approach would be to not only use dotabap.org as a check for PINF-name uniqueness but also as a central package registry. So you could remove the Git-URL part from the .abapgit.xml file and in the online installation scenario abapGit asks dotabap.org for the git url that is assigned to the package interface name. This would allow urls to be changed later without having all dependent repos change their dependencies. When using the offline installation using ZIP-files the URL is of no use anyway. But it also introduces a single point of failure if the registry is not available for some reason.

[fabianlupa]

It might also be a good idea to combine this functionality with the existing requirement-checks, at least UI-wise so when you are installing a new repository you can see in one dialog what you need to have installed, both repositories and software components.

[jfilak]

Since you propose to be able to track branch/tag, I propose to be able to track also commit.

Wouldn't it make more sense to start another project called "abapPackageManger" instead of implementing this feature in "a git client for ABAP"?

[larshp]

yeah, its not typical to combine the 2

but how will it look? first install abapGit, which then can be used to install the abapPackageManager, which then installs objects via abapGit? Installed repos are listed in abapPackageManager and the same is listed in abapGit.

[jfilak]

What about this:

• apm can install programs via HTTP / HTTPS on its own
• apm is aware of abapGit and asks the first user to install abapGit via apm
• when abapGit is installed apm can install (abap)git projects

[abramsba]

Hi, these were my ideas and notes when I tried creating mine.

• Package and source control management are two completely different responsibilities and don't belong in the same application together. APM should only make use of abapGit to install code, not extend it. APM also shouldn't interfere with abapGit.

• APM Should only be used to control the version of packages on development environments, from there SAP transports need to deploy updates/removals/new libraries to acceptance and production.

• None of the customers I've ever worked at have been receptive to the idea of allowing their SAP landscape to communicate openly with the internet, especially if as a consequence code and objects are created. It needs to be able to work offline and with archives.

• Due to the centralized domain specific nature of how SAP works inherently multiple versions of the same code can't be actively present on the same application server, which defeats the purpose of having a program to manage versions anyway.

• HTTP Interface is possible to be able to integrate APM with CLI tools, but I had to make a lot of modifications to abapGit to be able to automatically assign objects to transports and packages which made it incompatible with the offical abapGit (for example, SAPGUI prompts)

I haven't also actually developed any ABAP libraries that had more than a handful of dependencies, or at least I've never built anything that had to deal with multiple level of dependencies either. While it's not automated or even management, might it be an idea to add these details to another file in the repository and then have developers/maintainers indicate which other git libraries are required?

[fabianlupa]

Keep in mind that if you split up abapGit and abapPackageManager the user needs to keep two repositories up to date to be able to install others instead of just one.

[abramsba]

I can't think of any other programming languages that work in that way though. npm/yarn are seperate from node, as is maven from java, cargo from rust, and so forth. There are the applications used for managing dependencies and don't use git other than being able to use a git repository as a dependency source. I also don't think it's very much to ask of an ABAP developer to look after two ABAP programs instead of one.

[larshp]

ABAP is a special snowflake ❄️ 😄

node, java, and rust, I would consider as runtimes.

abapGit runs on top of the ABAP runtime like nodegit jgit and git2-rs

when doing node, rust or java development it runs on top of a windows/linux runtime with a file system, which gives the possibility to use many tools interchangeably. ABAP runs only on a SAP ABAP Kernel.

in npm/rpm(dont know about maven and cargo), some(most?) of the packages are compiled, in ABAP there is no compilation, only the source code. So might as well install the source code via a git client?

Package managers also typically manage versions, a user can choose to install a specific version. This is not possible in ABAP, the user will be forced to always choose the latest(development?) version. This is also okay since STMS is used for provisioning.

[larshp]

Currently, when cloning https://github.com/larshp/abapGitServer into a ABAP system with abapGit, the clone will fail during activation if the dependency https://github.com/larshp/ABAP-Swagger is not already cloned/installed.

in a npm similar approach,
git clone abapGitServer
npm install
will work since nothing is activated/compiled when cloning. The user executes both commands using the same user interface: the command line

[larshp]

Above is basically the problem I'd like to solve, being able to specify git dependencies.

When talking about package management:

• Versioning is already lost
• We have STMS for provisioning
• Discovering "packages" can be done via http://dotabap.org
• A central repository approach is IMHO not the best in closed ERP environments
• A central approach will have to support decentralization(internal package registers)
• A central approach raises a lot of questions regarding security

[abramsba]

node, java, and rust, I would consider as runtimes.

Maybe as a better example, I don't use npm via git, so I shouldn't be using an apm via abapGit either. I don't publish my node packages via git. Regardless, I never use git for anything other than source control management.

in npm/rpm(dont know about maven and cargo), some(most?) of the packages are compiled, in ABAP there is no compilation, only the source code. So might as well install the source code via a git client?

APM forked abapGit and attempted to remove all SAPGUI dependence and to stop registering libraries as git repositories. I used abapGit as the 'installer' behind the scenes because the only way I could make sense of the serializers was to to make use of the ZIP importer class. When importing a library to use for a project, I don't need to know how it's maintained. Same as above, when I import koa or something via npm I can't access the git repository either:

Nor will I ever modify the files under /node_modules/ unless I need to debug an issue. If I want to contribute to the koa library, I'd fork it on github and create modifications in my local environment. This also presents a challenge though because I wouldn't be able to work on an alternative version unless I have access to a second server.

in npm/rpm(dont know about maven and cargo), some(most?) of the packages are compiled, in ABAP there is no compilation, only the source code. So might as well install the source code via a git client?

Touches on my above point, but in my mind the structure of a ZIP file via abapGit has become for me the 'standard' format for exporting/importing code, though I'd like to have that as seperate functionality that I can use outside of abapGit.

[jfilak]

Yeah, factoring out the layer dealing with objects from abapGit into a separate project that can be used by apm and abapGit would make sense to me.

As suggest by @abramsba, the "abap package" can have the format of ZIP archive used by abapGit.
An apm repository can be just an HTTP site with a repo metadata file and package files.

[sbcgua]

My 2 cents to add :)

I think there really should be some version management.
There is a scenario that some product (business critical) is developed externally and then deployed to several clients. Then stability of the code matters a lot both for the product itself and the prerequisites which might be used there. As well as predictable environment.

So:
a) I actively develop a project, dependent on other libs, and have some influence on those libs, and have no maintenance obligations - I need latest versions
b) I actively develop a project and use some libs which I don't control and don't want to control, just need 1-2 functions from them. I need stability. My clients need stability. I clone some version (tag) of the libs and want them to be fixed.
c) based on B, I want to deploy particular product to a client, dependent on some ext libs. And I want the very same environment that was in my dev system.

Only A assumes latest versions (although it is probably most typical case). For now ... What about future if AG become more widespread... in particular with abap paas...

Unique indentifier: just IMHO, should not be any kind of UUID or anything else unreadable. I'd say the URL of original repo (which is saved in-code, not a clone url from metadata) could be a good option. Or path pkgname at dotabap.org. Actually both can be used in some priority (if there is registered dotabap path - good, otherwise direct url to github, like in npm). I think @flaiker meant something similar above (at the beginning).

Package manager vs git client. IMHO. From architecture perspective I'd be totally for separation of roles. But indeed abap is special :) abapGit is already a simple package manager - it shows installed packages for all users and manages metadata. Git client is a set of internal classes, not the whole program. Imho from usability perspective it should be one substance. Internally it could be several "modules" to separate the logic and then compiled together with abapmerge.

[abramsba]

I disagree very, very strongly with your post. As a software developer you should be aware of single responsibility, and what people are asking for when they ask for package manager integration in abapGit is bad design.

abap is special

Sorry if this offends, but I have trouble figuring out if people use this in the sarcastic sense that it's retarded, or if people legitimately use this excuse to justify using software and techniques in an unintended manner because they don't know better or don't want to admit they don't know what they're doing. If trying to coerce a 1980's SQL business programming language into taking on a responsibility it was never originally designed to do (decentralized source code control, 'object oriented programming') is what makes a language special, than ABAP is indeed special.

Looking at javascript as an example:

• node can be run without git
• npm can be run without git
• node can be run without npm

All other platforms manage just find keeping these three responsibilities separate. ABAP doesn't work like Node of course, but why can't the same analogy apply? If the workflow is close enough to the workflow of Node, than Node developers will have a much easier time finding their way in SAP than if they had to also learn the special ABAP sauce that gets thrown on top of everything.

• abap can be run without abapGit
• apm can be run without abapGit
• abap can be run without apm

I don't want a bloated monster of git client because SAP developers figured it'd be easy to jam pack three things that don't belong together under a single presentable package. Ideally, as I also said before, I'd already be so much happier if the installer component of abapGit was seperated so that I could even use that without git and have a similar setup and workflow to modern platforms.

[larshp]

From my point of view, the installer component is separated and can be used without git, abapGit uses this for offline projects.

abapGit can be used to move code into a SAP system, apm can be used to move code into a SAP system, the responsibility overlaps here?

How will the installation procedure look like in the above scenario with abapGit and/or apm? abapGit can be used to install apm, or apm to install abapGit, or both installations are separate but somehow share some code?

[larshp]

According to https://blog.npmjs.org/post/178027064160/next-generation-package-management npm is trying to build it into node itself, so the runtime will contain the package manager. I'm not saying this is a good or bad idea, but something to take from the blog is that they want to make it easier for the developer.

[sbcgua]

@abramsba: I cannot say that you're wrong. These are all good points. (and the comment to "abap is special" is fair :) ). But speaking "from helicopter" we need to consider other consequences as well:

a) One of AG principles is simple installation. I personally think that this is one of the main reasons why AG managed to get traction. People don't like changes, especially in so corporate env like sap. So copy-paste-run to try is really an advantage. What will happen with this after "splitting"? No, it is solvable! Just need to remember it in the big picture.

b) Splitting is good ... but for whom ? Here is a big question ? Imho, it is good for developers. Of abapGit and co-products. From architecture perspective - yes, totally! But for consumer (developer of some other code)? He does not care! He wants to run the UI, press the button, ... magic happens ..., profit! When I install node, I know npm is already there. If the git hadn't been so widespread and universal already, I'm sure, they would have include it to the node distro as well.

To be constructive :) OK, let's think, components ...

1. git client = set of classes. mainly for re-use. + minimal UI to clone 1 single package (zip or online). With selection screen, no UI. Well, maybe with diff viewer. But! should it do the activation also ? What if prerequisites are missing? Or activation is a separate tool? Something to think on...

2. package manager. what is currently UI + apm features in some mixture.

2.1) Maybe UI and "manager" should also be separate... because now it is not clear what is UI in sap :))))

3. Online catalog for discovery - dotabap.org. Alternative - direct git repo urls.

4. Powered up abapmerge to compile 1+2+2.1 for easy install.

Does it make some sense ?

[abramsba]

That would be if the package manager functionality was built into, I guess, the kernel. They're still not modifying a git client to extend it's functionality specifically for package management. This can't be though in ABAP (by us in either case), so the comprise here is having the git client also 'install' technical objects which is fetched by getting a compiled version of the client.

Ideally, I'd say there would be three individual components of software that can all be obtained individually:

• Installer: converts arbitrary ABAP source code to SAP technical objects. (for example configure and make)

• Package manager: software to manage version of libraries installed, libraries or software intended to be used in productive products (apt, npm, etc)

• SCM: software for library developers to manage the history of ABAP code (git)

• The installer is the API to creating ABAP technical objects.

• The installer installs the package manager.

• The installer installs the SCM.

Consider the typical ABAP assignment. Generally we're not asked to create frameworks, but to implement some kind of functional specification for a customer. As long as it works and it's done in the time expected, I don't think it interests most customers how it got there. The majority of my time is spent developing software I will most likely never share for obvious reasons. Using git won't provide me in this scenario any additional benefit nor is it worth the effort to try to get people to use something that's not actually fit for the job. Though, a few people in the team might know of a few handy packages for doing something like a web framework they like using, or something that provides better API's to things like conversion routines. They'll want to get this productive ready package and build their software on top of it.

Developers in teams who really like these libraries can also decide to help build the software further. In this case the developer won't be experimenting with the packaged version of the code on the system everybody is working from, but will have their own bald application server or have access to a hobby system of sorts. Instead of installing the package via the package manager, they clone the repository via git, make the changes they see fit, test, and then either merge/pull to their public code repository (not published package). Back to the first example, if they want to use the next version they can either download a ZIP from a global registry or, if allowed by customer, fetch via HTTPS.

As a third extreme example, I might even want to use the package manager to SCM. I may be working with a way to generate ABAP code strings and I need a way simply to test and run the objects and be able to remove it later. I may want to use the installer as well for creating code that doesn't need to be versioned or controlled.

[larshp]

Also see https://github.com/denoland/deno, its a "nodejs" like thingy, but aims to remove npm and the build system, by integrating everything. Packages are loaded directly from URLs

[GoWale]

Hi there, if I my inserts my ideas on this topic.

One of the big disatvantage of ABAP (and with this abapgit) is that you can not dowload/save/install a repo if it dependencies (SAP or other z-ojects) are missing in the target system.

As in the discussion mentioned there is no real way to solve this problem especially with “only abapgit”.
From my position (as a silent user of abapgit) the only thing that would be great if we could save all dependencies (from outside the repositorie) within the meta data (or similar).

So abapgit cann warn me before install that there might be error cause a/b and f are missing.
This would a least address the issue on installing/downloading a repo that needs a special release or other dependencies.

[larshp]

for requirements to a special release there is a feature implemented, see example abapGit/ADT_Backend@0d44288

[larshp]

closing, moving to #2236