The information of Request policy is missing in "Document"

[lyr3051]

I call a API which is register in API Manager and add Request policy "IP White-list".
I found the information of request policy is missing in traffic-summary index:

I try to find my open traffic log, the Request policy (IP White-list) is still exist.

Is the information about Request policy will transfer to elasticsearch by default ? or I should do some configuration to make it work?

[cwiechmann]

Hi @lyr3051, The API-Details, incl. what kind of Policies are configured is not directly taken from the OpenTraffic-Event Log. It is retrieved by Logstash during ingestion using the API-Builder REST-API "API-Lookup".

However, for obvious performance reasons during ingestion into Elasticsearch, API-Details aren't looked up for every single API-Request processed by Logstash. Instead the information is stored in Memcache and APIBuilder4Elastic for 10 minutes by default.

So, maybe you have called the same API, but without a policy configured, before. With that the API-Details are cached and any further request to the same API in the next 10 minutes, will use the cached API-Details, without your policy.
In other words, wait some time, and you should see the API incl. your policies. There is no additional configuration required.

FYI: What you see in the screenshot is the entire policy-execution path, which is shown in Traffic-Monitor when selecting an API. It's way too complicated to extract anything useful out of it, as it may contain any weird combination of policies.

[Wei-Yo]

oh~ I get it, thanks Chris!
you explained in great detail~

Another question I want to ask is,

Regarding the size of opentrafficlog and the size of payload, is there a file size limit or a limit on the number of files?
Where can I go to adjust this limit?

The relevant settings I can see are only here, are there any other places?

[cwiechmann]

You need to make sure, to configure the OpenTraffic-Eventlog disk space big enough. Read more here:
https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk#enable-open-traffic-event-log

The payload is out of band, as you are storing them on a shared volume, which make the payload available to API-Builder4Elastic:
https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk#traffic-payload
There is no limit for the payload size.
However, one small difference fyi:
When using the "Show" payload button in the Traffic-Monitor UI, by default only the first 20.000 bytes are returned, because more data breaks the UX in the UI anyway. If a user really needs to see the entire file, he can use the "Save" option.

[Wei-Yo]

I did not find a place to configure the number of OpenTrafficlog and the size and number of a single file of the payload from the link you provided.

Could it be that OpenTraffic log and payload do not rotate when the capacity limit is reached?

[cwiechmann]

The size of the OpenTraffic files are static. I think 10MB. You only configure the total disk space for all OpenTraffic files. They are rotated based on the disk space setting you see in the dialog.

Payload files are indeed not rotated. You need to run a script that removes files older than x days.

[Wei-Yo]

Thanks for your reply, I see.

In addition, I had some tests for Traffic Monitor earlier, and my configuration is as follows:

When I searched Traffic Monitor for 2021/12/12 Traffic, I didn't find any matching data.

But I do find 2021/12/12Traffic in the opsdb folder

To sum up the above, I would like to confirm two things.

1. Do you know if the source of Traffic Monitor is opsdb, what is the upper limit of the data that the UI can extract?
2. Will I encounter similar problems if the source of Traffic Monitor is Elasticsearch?

[lyr3051]

Hi @cwiechmann
There is another question about the API-Details.
When I call the API with API Key, the apiSecurity still "Passthrough".
Is the API Security also retrieved by Logstash during ingestion using the API-Builder REST-API "API-Lookup" ?

[cwiechmann]

Yes, The API-Details that are looked up also contain the API-Security. It's essentially everything what you configured for API in the API-Manager Frondend-API section.

[lyr3051]

Hi @cwiechmann

I wonder why the request policy is missing in traffic-summary index before I restart the API-builder and memcached,
but it is still exist in API Gateway Traffic monitoring?

Which index is restrieved from API Builder to show the data in API Gateway Traffic Monitoring?

[cwiechmann]

Hi @lyr3051, I believe you discovered a bug in the solution. I realized the same behavior and the root cause, seems the internal In-Memory caching in APIBuilder4Elastic, which never expired.
I fixed this and it will be delivered as part of the next release.