AL Rest Client

[ajkauffmann]

Provides functionality to easily call REST web services

The module contains methods to support

• Calling web services with just one line of code
• Creating request content from Text, JSON, XML or binary data
• Read the response as Text, JSON, XML or binary data
• Authenticate using basic authentication

Some examples:

// Getting text
ResponseText := ALRestClient.Get(Url).Content().AsText();

// Getting binary data
TempBlob := ALRestClient.Get(Url).Content().AsBlob();

// Posting binary data and returning binary data
ALHttpContent.Create(TempBlob)
TempBlob := ALRestClient.Post(Url, ALHttpContent).Content().AsBlob();

// Getting a JsonObject
JsonObject := ALRestClient.GetAsJson(Url).AsObject();

// Posting a JsonObject and returning the result as JsonObject
JsonObject := ALRestClient.PostAsJson(Url, JsonObject).AsObject();

// Using Basic Authentication
HttpAuthenticationBasic.Initialize('user01', 'Password123');
ALRestClient.Initialize(HttpAuthenticationBasic);
ALHttpResponseMessage := ALRestClient.Get('https://httpbin.org/basic-auth/user01/Password123');

Remarks:

• Support for OAuth can be added later. A module is already available, but it requires some internal discussion.
• Telemetry signal RT0019 (Outgoing Web Service Request) will be lost for the ISV because the platform only sends it to the app that uses HttpClient directly. This needs to be addressed somehow.
• Documentation (like a readme.md file) can be generated from the XML comments, but maybe it's better to write a more extensive document how to use this new module.

[PeterConijn]

This looks really awesome, AJ. I did find some security considerations, but I can't wait until this is in BC

[JesperSchulz]

Looks like a logo is missing. I'll remove the reference to it (no need for logos in modules). Would love to make the compilation succeed to start with :-)

[ajkauffmann]

Replaced the usage of enum "Http Request Type" from the OAuth module with a new enum "Http Method". I figured it was more appropriate for the module to have its own enum instead of taking a dependency on the OAuth module.
Also added a dependency on Base64 Convert to solve a compiler error.

[IhorHandziuk]

Overall a very good module. We could benefit a lot from using it.

[ajkauffmann]

Thank you all for the comments so far. Much appreciated and great feedback!
I'm going to work through all of it this week and will provide a number of updates. Including an update to solve the issue with telemetry signal RT0019 and the popup question in sandboxes for allowing or blocking outgoing web requests (which was now only once for the system app instead of per app that actually started the request).

[ajkauffmann]

Just create a fairly large commit to resolve almost all comments.

But most importantly, I've introduced an interface "Http Client Handler". A codeunit "Default Client Handler" is also provided, that can also serve as boilerplate code for apps. The main purpose of this handler codeunit is to let the original app send the actual http request. As a result, telemetry signal RT0019 will be sent to the ISV's AppInsight and the feature to block outgoing web requests is applied to the original app.

The Http Client Handler must be specified when initializing the Rest Client. In case no handler codeunit is specified, it will take the default one provided in the module.

This pattern can also be used for tests to create fake responses or a mocking codeunit to handle the request differently.

[JesperSchulz]

@JesperSchulz It seems the pipeline is not ready for platform 12.0 at the moment.

It is not, no. But don't sweat it. As long as it compiles on 2023 Wave 2 / 12.0, it's all good. Hopefully I'll soon be able to give this a spin in our internal build system!

[ajkauffmann]

@JesperSchulz It seems the pipeline is not ready for platform 12.0 at the moment.

It is not, no. But don't sweat it. As long as it compiles on 2023 Wave 2 / 12.0, it's all good. Hopefully I'll soon be able to give this a spin in our internal build system!

I've tested with a 23.0 preview build, and all tests succeed. It's great to have the SecretText implemented from the beginning, that's why I needed v23.

[ajkauffmann]

One small issue is missing, which was reported to @KennieNP a while ago: the HttpRequest object in the AL platform doesn't support relative URLs, while the underlying .Net object does. I've worked around it, but I'd be more happy if it would be supported in the platform object.

[JesperSchulz]

@KennieNP also suggested we add some retry-logic to this wrapper, while at it (I could not spot it when rushing through the code today). It would greatly improve the HttpClient call stability in the app if we did. He even provided some sample code to illustrate his intend ;-)

`

// RetryStrategy can be any of these strings (should be an enumeration)
// RegularIntervalRetry
// ExponentialBackoff
// Randomization


local procedure SendRequest(
    Client: HttpClient;
    RequestMessage: HttpRequestMessage;
    UserErrorMessage: Text[200];
    IsCollectable: boolean;
    MaxNumberOfRetries: Integer;
    RetryStrategy: Text;
    RetrySleeptimeInMilliseconds: Integer
) ResponseMessage: HttpResponseMessage
var
    IsSuccessful: Boolean;
    ErrorInfoObject: ErrorInfo;
    ErrorCustomDimensions: Dictionary of [Text, Text];
    vResponseMessage: HttpResponseMessage;
    HttpStatusCode: integer;
begin
    IsSuccessful := Client.Send(RequestMessage, vResponseMessage);

    if not IsSuccessful then begin
        ErrorInfoObject.DetailedMessage := 'The HttpClient.Send could not be sent, this is likely an error in the AL code.';
        ErrorInfoObject.Title := 'Could not call external service';
        ErrorInfoObject.Message := UserErrorMessage;
        ErrorInfoObject.Collectible := IsCollectable;
        ErrorCustomDimensions.Add('HTTPMethod', RequestMessage.Method);
        ErrorInfoObject.CustomDimensions := ErrorCustomDimensions;
        Error(ErrorInfoObject);
    end;

    if not vResponseMessage.IsSuccessStatusCode() then begin
        HttpStatusCode := vResponseMessage.HttpStatusCode();

        // Handling status codes 429 and 503 requires the client to adopt a retry logic while providing a cool off period. 
        case HttpStatusCode of
            419:
                vResponseMessage := SendRequestRetry(Client, RequestMessage, MaxNumberOfRetries, RetryStrategy, RetrySleeptimeInMilliseconds);
            503:
                vResponseMessage := SendRequestRetry(Client, RequestMessage, MaxNumberOfRetries, RetryStrategy, RetrySleeptimeInMilliseconds);
        end;

        if not vResponseMessage.IsSuccessStatusCode() then begin
            HttpStatusCode := vResponseMessage.HttpStatusCode();
            // final error handling after possible retries
            if ((HttpStatusCode = 429) or (HttpStatusCode = 503)) then begin
                ErrorInfoObject.DetailedMessage := 'The HttpClient.Send call was successful, but the external service responded with a ' + Format(HttpStatusCode) + ' HTTP status code even after retrying.';
                ErrorCustomDimensions.Add('RetryAttempts', Format(MaxNumberOfRetries));
            end;
            if (HttpStatusCode = 504) then begin
                ErrorInfoObject.DetailedMessage := 'The HttpClient.Send call was successful, but the external service timed out (responded with a ' + Format(HttpStatusCode) + ' HTTP status code). You probably need to refactor your AL code.';
            end;
            if (HttpStatusCode > 500) then begin
                ErrorInfoObject.DetailedMessage := 'The HttpClient.Send call was successful, but the external service responded with a 5xx error. This call will not be retried.';
            end;

            ErrorInfoObject.Title := 'Could not call external service';
            ErrorInfoObject.Message := UserErrorMessage;
            ErrorInfoObject.Collectible := IsCollectable;
            ErrorCustomDimensions.Add('HttpStatusCode', Format(HttpStatusCode));
            ErrorCustomDimensions.Add('HTTPMethod', RequestMessage.Method);
            ErrorInfoObject.CustomDimensions := ErrorCustomDimensions;
            Error(ErrorInfoObject);
        end
        else // retry worked! :)
            ;

    end;

    ResponseMessage := vResponseMessage
end;



// Asserts that Client.Send(RequestMessage, vResponseMessage) is successfull (that the AL code is sound)
local procedure SendRequestRetry(
    Client: HttpClient;
    RequestMessage: HttpRequestMessage;
    MaxAttempts: Integer;
    RetryStrategy: Text;
    SleeptimeInMilliseconds: Integer
) ResponseMessage: HttpResponseMessage
var
    vResponseMessage: HttpResponseMessage;
    Attempt: Integer;
    IsSuccessful: Boolean;
begin
    Attempt := 1;
    while Attempt <= MaxAttempts do begin
        IsSuccessful := Client.Send(RequestMessage, vResponseMessage);
        if (IsSuccessful) then
            break
        else begin
            case RetryStrategy of
                'RegularIntervalRetry':
                    RegularIntervalRetry(SleeptimeInMilliseconds);
                'ExponentialBackoff':
                    ExponentialBackOff(SleeptimeInMilliseconds, Attempt);
                'Randomization':
                    Random(SleeptimeInMilliseconds);
                else
                    RegularIntervalRetry(SleeptimeInMilliseconds);
            end;
            // if RetryStrategy = 'RegularIntervalRetry' then 
            // else if ;
        end;
        ;
        Attempt := Attempt + 1;
    end;
    ResponseMessage := vResponseMessage
end;

local procedure RegularIntervalRetry(SleeptimeInMilliseconds: Integer)
begin
    Sleep(SleeptimeInMilliseconds);
end;

local procedure ExponentialBackOff(SleeptimeInMilliseconds: Integer; Attempt: Integer)
begin
    Sleep(SleeptimeInMilliseconds * Power(2, Attempt));
end;

local procedure Random(SleeptimeInMilliseconds: Integer)
begin
    Sleep(System.Random(SleeptimeInMilliseconds));
end;

}
`

This doesn't have to be part of the initial PR, but we'll probably add it eventually, as it's a great suggestion.

[ajkauffmann]

That's a great suggestion! I agree it can be added later. If time permits I may add it to the current PR.
For the retry strategy: if that's an enum, we could consider adding an interface to it. Then it can easily be extended by others with alternative strategies.
You may want to already increase the number range so we have enough objects reserved! 😄

[JesperSchulz]

Feel free to extend the number range 😊 There should be plenty of available numbers...
Love the idea of making the retry strategy extensible. I guess retying can be done in many different ways.
Adding a retry strategy to the module will further boost the value proposition of this module. We are indeed seeing a lot of failures in telemetry due to the lack of this capability.
This module is coming along nicely 🥳 I'd love to get it on the road with 23.1 (November release) - let's see if we can make that happen!

[pri-kise]

over all this PR looks very promising.
Personally I'd like to have an Generi POST method on the rest client, that accepts an InStream, but as a starter the current Generic Send is okay, too.

[JesperSchulz]

Let's see what our build system has to say to this PR! Processing internally now.

[JesperSchulz]

Short update: Tomorrow we are internally opening up for PRs through the new BCApps repository. I will make this PR the first official AL PR to be processed in the new repository 🥳 From November 1st, the BCApps repository will open to the public too. By then, I hope this PR has successfully been processed though.

[JesperSchulz]

This is in a good enough shape to get submitted. Further improvements can be made in chaser PRs. With this module in, we can start uptake 🥳

[JesperSchulz]

We'll have to wait with merging this, until we've rolled out the 23.0 bits on Monday, as this doesn't compile against 22.5 due to namespaces and the use of secure string.

[JesperSchulz]

The PR went into the main branch (24.0) 🥳 As soon as we release the 23 bits, we can merge this PR. Further improvements can then be done to the added module.