Building RPGLE from git without managing dependencies

[worksofliam]

Git is the way forward for the future of IBM i, we all know that. One of the biggest issues we face currently is not the developer tools honestly. We've come a great distance compared to where we were two years ago in regards to VS Code and local development for RPGLE code.

But the biggest issue is how we build our code. I am not talking about compiling a program here-or-there in VS Code or RDi. I am talking about how we can build our code in an automated fashion. Today there are two main tools for automating the build of ILE code out of git:

1. a makefile
2. ibmi-bob (Rules.mk)

They both can be powerful tools, but the more complicated or larger your application is, the harder it can be to maintain those files. Defining what every programs depends on for it to be compiled is a PITA.

rmake

We have the tech to understand how to build ILE objects from source, so it's time I put it to use. I am excited to write about rmake today. rmake is a tool that generates a makefile from source. This means that the developer no longer has to define a makefile with all the dependencies in them and it will instead get built automatically at build time.

Here's an action I have defined in VS Code, so after I make a change, it will rebuild what has changed for me:

  {
    "name": "Build all",
    "extensions": [
      "GLOBAL"
    ],
    "command": "rmake --verbose && /QOpenSys/pkgs/bin/gmake BIN_LIB=&CURLIB ERR=*EVENTF",
    "environment": "pase",
    "deployFirst": true
  }

Because it is just GNU Make under the hood, it will only rebuild the source that has changed, which is especially good for huge applications.

Usage

All the usage examples are against ibmi-company_system.

rmake without any parameters will just generate a makefile and go on with its day - not very complicated. But it has some very useful CLI options.

--verbose

Verbose will print everything out that it finds. In this example, it's printing all the base objects it finds, lists is source path and other objects it depends on.

-bash-5.1$ rmake --verbose
Found 3 files with '**/*.{SQLRPGLE,sqlrpgle,RPGLE,rpgle}'.
DEPTS.PGM
        Source: qrpglesrc/depts.pgm.sqlrpgle
        Depends on: EMPLOYEES.PGM DEPARTMENT.FILE DEPTS.FILE DEPTS.FILE
EMPLOYEES.PGM
        Source: qrpglesrc/employees.pgm.sqlrpgle
        Depends on: EMPLOYEE.FILE EMPS.FILE
MYPGM.PGM
        Source: qrpglesrc/mypgm.pgm.rpgle
        Depends on: $(APP_BNDDIR).BNDDIR

-l (finding usage)

-l is a way to find out where an object is used, and what changing it will impact/require a rebuild. In this instance, I look for employee.file

-bash-5.1$ rmake -l employee.file
EMPLOYEE.FILE (qddssrc/employee.table)
        EMPLOYEES.PGM (qrpglesrc/employees.pgm.sqlrpgle)
                DEPTS.PGM (qrpglesrc/depts.pgm.sqlrpgle)

This shows that EMPLOYEE.FILE is used by EMPLOYEES.PGM and that is used by DEPTS.PGM. Of course, if EMPLOYEES.FILE was used by multiple objects they would be listed here also.

Doing the build

As I mentioned already, rmake just generates the makefile, so gmake is still required to do this build. Because of this, the makefile should not be tracked in git. In this instance, I had already built my application:

-bash-5.1$ rmake && gmake BIN_LIB=LIAMA
gmake: Nothing to be done for 'all'.

But if I touch the file to make it seem updated, then it will rebuild it automatically:

-bash-5.1$ rmake && gmake BIN_LIB=LIAMA
liblist -c LIAMA;\
system "CRTBNDRPG PGM(LIAMA/MYPGM) SRCSTMF('qrpglesrc/mypgm.pgm.rpgle') OPTION(*EVENTF) DBGVIEW(*SOURCE) TGTRLS(*CURRENT) TGTCCSID(*JOB) BNDDIR((LIAMA/APP)) DFTACTGRP(*no)"
# spool file is printed here
          * * * * *   E N D   O F   F I N A L   S U M M A R Y   * * * * *
 Program MYPGM placed in library LIAMA. 00 highest severity. Created on 05/16/23 at 21:29:35.
           * * * * *   E N D   O F   C O M P I L A T I O N   * * * * *

Availability

It's not ready yet. Sorry!

[AG1965]

That's interesting, very interesting, as i'm too silly for BOB and struggling with gmake for quite some time right now, and just as i'm seeing the horizon there's this rising - could be perfect timing!
Interesting that you seem to avoid the hen/egg problem by using the sources instead of the objects to generate recipes.
Is there the possibility to handle special cases, e.g., adopted authority and changing the owner for CLLE, or some display files needing some non-default parameters?
How do you make the "where used"? Do you keep a DSPPGMREF-file or is this created each time on the fly? (Which would be rather slow with big source packages. Ours is a bit more than 45.000 source members, and i consider that big.)
Could one edit the generated gmake file(s?) to add triggers on the right side of the colon that rmake (or DSPPGMREF) can't figure out? Like dynamical SQL, but i know which files the statement might use and want to add them to the recipe?
Thanks for your great work(s), really appreciate it!

[jwoehr]

Good idea @worksofliam ... we have traditionally called this genre of tool a "makemake". And as a life expert in gmake I can affirm that this staple of C/C++ development is both:

• adequate, and
• obscure to most IBM i programmers.

So providing a more complete makemake than bob, one congenial to IBM i programming, will be an awesome contribution to the practice!

"Go for it!"

[ataylorkato]

In your example, specifically DEPTS.PGM, how are these 4 dependencies being determined? Is it just doing source analysis on qrpglesrc/depts.pgm.sqlrpgle and catching Dcl-F/Dcl-Pr/etc, or some other method?

[worksofliam]

@ataylorkt

how are these 4 dependencies being determined?

Good question. Everything rmake does is by source.

• EMPLOYEES.PGM is defined as EXTPGM prototype.
• DEPARTMENT.FILE is used in two places: as a EXTNAME struct and in an embedded SQL query
• DEPTS.FILE is defined as a workstation file. It's also listed twice - thus why it's not released yet.

You can see the source code for DEPTS.PGM here.

[edmundreinhardt]

Would it be simple to generate a Rules.mk for bob since that is a subset? I.e we just need to specify the dependencies in the exact same format without specifying the recipe to do the compile.

This would fill the existing hole of discovering the dependencies which requires a lot of work.

[worksofliam]

@edmundreinhardt Yup, generating the Rules.mk is up next!

[AG1965]

In my ongoing struggle i have decided to call make with option -n to do nothing and redirect the output to a make.sh-file and reading it line by line and executing it with QCMDEXC.
This should solve my problems with the library list which doesn't work (although i included .ONESHELL in my makefile) and my inability to escape member names with $ signs in them.
(And this opens new possibilities: when i replace the object library with a branch library it could help me with creating branches into branch libraries with the production libraries as equivalent to the main branch.)

[JDubbTX]

One thing that I haven't seen a tool like this do - and maybe because this is a bit advanced and lots of IBM i shops may not really need this - is dynamic dependencies for ILE service programs based on the signature and parameters of the service program.

By specifying a static signature in binder language, calling programs don't usually need a rebuild. A change to a service programs signature would mean all calling programs would need a rebuild.

You can also specify a '*prev' and a '*current' program level (like in this example) instead of a static signature in binder language, but many shops just opt for a static signature - that's the standard at our shop.

I'd like a tool to warn the user if the signature of a service program changes, and give the user some idea of impact. By impact, I mean number of objects to rebuild due to the change in signature. This is all in an effort to avoid the dreaded signature violation error.

If the tool senses no change to the signature of a service program, no calling programs / modules / service programs would need to be rebuilt. The exception to this would be if a parameter changed in one of the procedures of the service program.

You would have to build in a bunch of smarts here to know 1) There is a static signature for this service program and 2) the module that was changed before creation of the service program had a change to one of its parameters and therefor 3) All calling programs that make use of that particular procedure with the parameter change should be rebuilt.

The rules for this could get quite hairy, which is probably why I haven't seen anyone attempt this. Might be difficult to do this with source info alone - you could more definitively know which objects need a rebuild by querying BOUND_SRVPGM_INFO or make use of the signature list in PROGRAM_INFO For either you would need an object library list.

[worksofliam]

@JDubbTX

By specifying a static signature in binder language, calling programs don't usually need a rebuild. A change to a service programs signature would mean all calling programs would need a rebuild.

In theory it'd be possible. If you ensured that each service depended on a binder source for it to build, it would detect that the service program wouldn't need recompiling if the binder source didn't change. Of course, that's basic as it's looking at file change and not signature change, but it's a step in the right directory.

I will share more later about how you can use -l to see how changing a specific object/source will impact other objects.