Skip to content

Commit

Permalink
XRDCAT-12: Improve Company and Organization handling processes (#28)
Browse files Browse the repository at this point in the history
  • Loading branch information
@raits
raits committed Jun 18, 2024
1 parent a3a62cb commit 35d33bf
Show file tree
Hide file tree
Showing 18 changed files with 425 additions and 535 deletions.
23 changes: 12 additions & 11 deletions doc/xroad_catalog_installation_guide.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# X-Road Catalog Installation Guide
Version: 1.3.1
Version: 1.3.2
Doc. ID: IG-XRDCAT

---
Expand All @@ -12,6 +12,7 @@ Doc. ID: IG-XRDCAT
| 09.09.2023 | 1.2.0 | Remove instructions to install the `xroad-conflient` module manually | Petteri Kivimäki |
| 24.09.2023 | 1.3.0 | Add instructions to disable the automatic backup job run by the `xroad-conflient` module | Petteri Kivimäki |
| 10.06.2024 | 1.3.1 | Add information about default values for configurable properties | Raido Kaju |
| 14.06.2024 | 1.3.2 | Update information about company and organization task properties | Raido Kaju |

## Table of Contents <!-- omit in toc -->

Expand Down Expand Up @@ -189,16 +190,16 @@ Optional parameters which can be configured in the same file are described below
When using the `xroad-catalog-collector` module with the `FI` profile, the following additional optional parameters are
in effect:

| Parameter | Defaults | Description |
|--------------------------------------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `xroad-catalog.max-organizations-per-request` | 100 | A parameter for setting the maximum amount of organizations that can be fetched from the organizations API in one request, e.g. value `100` means `100 organizations`. |
| `xroad-catalog.fetch-companies-limit` | 1000 | A parameter for setting the maximum amount of companies that can be fetched from the companies API, e.g. value `1000` means `1000 companies`. |
| `xroad-catalog.fetch-organizations-limit` | 2000 | A parameter for setting the maximum amount of organizations that can be fetched from the organizations API, e.g. value `2000` means `2000 organizations`. |
| `xroad-catalog.fetch-companies-run-unlimited` | false | A parameter for setting whether the X-Road Catalog Collector should try to fetch data from the companies API continuously during a day or only between certain hours, e.g. value `true` means `continously`. |
| `xroad-catalog.fetch-companies-time-after-hour` | 3 | A parameter for setting the start of time interval during which the X-Road Catalog Collector should try to fetch data from the companies API continuously (this parameter will be ignored if the parameter `xroad-catalog.fetch-companies-run-unlimited` is set to `true`), e.g. value `18` means starting from `18:00`. |
| `xroad-catalog.fetch-companies-time-before-hour` | 4 | A parameter for setting the end of time interval during which the X-Road Catalog Collector should try to fetch data from the companies API continuously (this parameter will be ignored if the parameter `xroad-catalog.fetch-companies-run-unlimited` is set to `true`), e.g. value `23` means ending at `23:00`. |
| `xroad-catalog.fetch-organizations-pool-size` | 10 | A parameter for setting the amount of virtual threads in the pool for fetching organizations from the organizations API, e.g. value `10` means `10 virtual threads`. |
| `xroad-catalog.fetch-companies-pool-size` | 10 | A parameter for setting the amount of virtual threads in the pool for fetching companies from the companies API, e.g. value `10` means `10 virtual threads`. |
| Parameter | Defaults | Description |
|--------------------------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `xroad-catalog.fetch-external-run-unlimited` | false | A parameter for setting whether the X-Road Catalog Collector should try to fetch data from the external API continuously during a day or only between certain hours, e.g. value `true` means `continously`. |
| `xroad-catalog.fetch-external-time-after-hour` | 3 | A parameter for setting the start of time interval during which the X-Road Catalog Collector should try to fetch data from the companies API continuously (this parameter will be ignored if the parameter `xroad-catalog.fetch-companies-run-unlimited` is set to `true`), e.g. value `18` means starting from `18:00`. |
| `xroad-catalog.fetch-external-time-before-hour` | 4 | A parameter for setting the end of time interval during which the X-Road Catalog Collector should try to fetch data from the companies API continuously (this parameter will be ignored if the parameter `xroad-catalog.fetch-companies-run-unlimited` is set to `true`), e.g. value `23` means ending at `23:00`. |
| `xroad-catalog.fetch-external-interval-min` | 20 | A parameter for setting the amount of time in minutes after which the X-Road Catalog Collector should start re-fetching data from the external API, e.g. value `20` means `every 20 minutes`. This works together with the following two parameters to determine how often data is checked for staleness and updated in a batch. |
| `xroad-catalog.fetch-external-limit` | 500 | A parameter for setting the maximum amount of Members that should be fetched per external API in one run, e.g. value `500` means `500 members`. In the current implementation the example value would fetch `500` members information from both the `company` and `organization` API. |
| `xroad-catalog.fetch-external-update-after-days` | 7 | A parameter for setting the amount of days after which the X-Road Catalog Collector should consider Company and Organization data stale and try to fetch data from the external API again, e.g. value `7` means `after 7 days`. |
| `xroad-catalog.fetch-organizations-pool-size` | 10 | A parameter for setting the amount of virtual threads in the pool for fetching organizations from the organizations API, e.g. value `10` means `10 virtual threads`. This controls how many parallel requests will hit the organizations API. |
| `xroad-catalog.fetch-companies-pool-size` | 10 | A parameter for setting the amount of virtual threads in the pool for fetching companies from the companies API, e.g. value `10` means `10 virtual threads`. This controls how many parallel requests will hit the companies API. |

In addition, update the `xroad-catalog.shared-params-file` property value in `/etc/xroad/xroad-catalog/lister-production.properties`.
The value must point to the `/etc/xroad/globalconf/<INSTANCE_IDENTIFIER>/shared-params.xml` X-Road global configuration file:
Expand Down
33 changes: 22 additions & 11 deletions ...catalog-collector/src/main/java/fi/vrk/xroad/catalog/collector/XRoadCatalogCollector.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
import fi.vrk.xroad.catalog.collector.tasks.FetchWsdlsTask;
import fi.vrk.xroad.catalog.collector.tasks.ListClientsTask;
import fi.vrk.xroad.catalog.collector.tasks.ListMethodsTask;
import fi.vrk.xroad.catalog.collector.tasks.UpdateExternalsTask;
import fi.vrk.xroad.catalog.collector.util.XRoadRestServiceIdentifierType;
import fi.vrk.xroad.catalog.collector.wsimport.ClientType;
import fi.vrk.xroad.catalog.collector.wsimport.XRoadServiceIdentifierType;
Expand Down Expand Up @@ -65,47 +66,57 @@ public static void main(String[] args) throws MalformedURLException, URISyntaxEx
}
}

final TaskPoolConfiguration taskPoolConfiguration = context.getBean(TaskPoolConfiguration.class);

final boolean isFIProfile = Arrays.stream(env.getActiveProfiles())
.anyMatch(str -> str.equalsIgnoreCase(FI_PROFILE));

final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
final BlockingQueue<ClientType> listMethodsQueue = new LinkedBlockingQueue<>();
final BlockingQueue<ClientType> fetchCompaniesQueue = isFIProfile ? new LinkedBlockingQueue<>() : null;
final BlockingQueue<ClientType> fetchOrganizationsQueue = isFIProfile ? new LinkedBlockingQueue<>() : null;
final BlockingQueue<String> fetchCompaniesQueue = isFIProfile ? new LinkedBlockingQueue<>() : null;
final BlockingQueue<String> fetchOrganizationsQueue = isFIProfile ? new LinkedBlockingQueue<>() : null;
final BlockingQueue<XRoadServiceIdentifierType> fetchWsdlsQueue = new LinkedBlockingQueue<>();
final BlockingQueue<XRoadRestServiceIdentifierType> fetchRestQueue = new LinkedBlockingQueue<>();
final BlockingQueue<XRoadRestServiceIdentifierType> fetchOpenApiQueue = new LinkedBlockingQueue<>();

if (isFIProfile) {
log.info("FI profile detected, starting up organizations and companies fetchers");

final FetchCompaniesTask fetchCompaniesTask = new FetchCompaniesTask(context, fetchCompaniesQueue);
Thread.ofVirtual().start(fetchCompaniesTask::run);
Thread.ofVirtual().start(fetchCompaniesTask);

final FetchOrganizationsTask fetchOrganizationsTask = new FetchOrganizationsTask(context,
fetchOrganizationsQueue);
Thread.ofVirtual().start(fetchOrganizationsTask::run);
Thread.ofVirtual().start(fetchOrganizationsTask);

final UpdateExternalsTask updateExternalsTask = new UpdateExternalsTask(context, fetchCompaniesQueue,
fetchOrganizationsQueue);
long externalInterval = taskPoolConfiguration.getFetchExternalInterval();
log.info("Starting up external sources updater with interval of {}", externalInterval);

scheduler.scheduleWithFixedDelay(updateExternalsTask, 0, externalInterval, TimeUnit.MINUTES);
}

final FetchWsdlsTask fetchWsdlsTask = new FetchWsdlsTask(context, fetchWsdlsQueue);
Thread.ofVirtual().start(fetchWsdlsTask::run);
Thread.ofVirtual().start(fetchWsdlsTask);

final FetchRestTask fetchRestTask = new FetchRestTask(context, fetchRestQueue);
Thread.ofVirtual().start(fetchRestTask::run);
Thread.ofVirtual().start(fetchRestTask);

final FetchOpenApiTask fetchOpenApiTask = new FetchOpenApiTask(context, fetchOpenApiQueue);
Thread.ofVirtual().start(fetchOpenApiTask::run);
Thread.ofVirtual().start(fetchOpenApiTask);

final ListMethodsTask listMethodsTask = new ListMethodsTask(context, listMethodsQueue, fetchWsdlsQueue,
fetchRestQueue,
fetchOpenApiQueue);
Thread.ofVirtual().start(listMethodsTask::run);
fetchRestQueue, fetchOpenApiQueue);
Thread.ofVirtual().start(listMethodsTask);

// The ListClientsTask is the main task that starts the whole process and
// gathers information that the other tasks will react on to do work
final ListClientsTask listClientsTask = new ListClientsTask(context, listMethodsQueue, fetchCompaniesQueue,
fetchOrganizationsQueue);

long collectorInterval = context.getBean(TaskPoolConfiguration.class).getCollectorInterval();

long collectorInterval = taskPoolConfiguration.getCollectorInterval();
log.info("Starting up catalog collector with collector interval of {}", collectorInterval);

scheduler.scheduleWithFixedDelay(listClientsTask::run, 0, collectorInterval, TimeUnit.MINUTES);
Expand Down
42 changes: 23 additions & 19 deletions ...tor/src/main/java/fi/vrk/xroad/catalog/collector/configuration/TaskPoolConfiguration.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,23 +74,31 @@ public class TaskPoolConfiguration {
@Value("${xroad-catalog.fetch-companies-url}")
private String fetchCompaniesUrl;

@Value("${xroad-catalog.max-organizations-per-request:100}")
private int maxOrganizationsPerRequest;
@Value("${xroad-catalog.fetch-external-limit:500}")
private int fetchExternalLimit;

@Value("${xroad-catalog.fetch-companies-limit:1000}")
private int fetchCompaniesLimit;
@Value("${xroad-catalog.fetch-external-update-after-days:7}")
private int fetchExternalUpdateAfterDays;

@Value("${xroad-catalog.fetch-organizations-limit:2000}")
private int fetchOrganizationsLimit;
@Value("${xroad-catalog.fetch-external-interval-min:20}")
private long fetchExternalInterval;

@Value("${xroad-catalog.fetch-companies-run-unlimited:false}")
private boolean fetchCompaniesRunUnlimited;
@Value("${xroad-catalog.fetch-external-run-unlimited:false}")
private boolean fetchExternalRunUnlimited;

@Value("${xroad-catalog.fetch-companies-time-after-hour:3}")
private int fetchCompaniesTimeAfterHour;
@Value("${xroad-catalog.fetch-external-time-after-hour:3}")
private int fetchExternalTimeAfterHour;

@Value("${xroad-catalog.fetch-companies-time-before-hour:4}")
private int fetchCompaniesTimeBeforeHour;
@Value("${xroad-catalog.fetch-external-time-before-hour:4}")
private int fetchExternalTimeBeforeHour;

@Value("${xroad-catalog.fetch-organizations-pool-size:10}")
private int fetchOrganizationsPoolSize;

@Value("${xroad-catalog.fetch-companies-pool-size:10}")
private int fetchCompaniesPoolSize;

// Parameters handling database log storage

@Value("${xroad-catalog.flush-log-time-after-hour:3}")
private int flushLogTimeAfterHour;
Expand All @@ -101,6 +109,8 @@ public class TaskPoolConfiguration {
@Value("${xroad-catalog.error-log-length-in-days:90}")
private int errorLogLengthInDays;

// Parameters controlling how often data is collected from the X-Road instance

@Value("${xroad-catalog.fetch-run-unlimited:false}")
private boolean fetchRunUnlimited;

Expand All @@ -110,7 +120,7 @@ public class TaskPoolConfiguration {
@Value("${xroad-catalog.fetch-time-before-hour:4}")
private int fetchTimeBeforeHour;

// Collector internal parameters
// Collector internal pool parameters

@Value("${xroad-catalog.collector-interval-min:20}")
private long collectorInterval;
Expand All @@ -127,10 +137,4 @@ public class TaskPoolConfiguration {
@Value("${xroad-catalog.fetch-rest-pool-size:10}")
private int fetchRestPoolSize;

@Value("${xroad-catalog.fetch-organizations-pool-size:10}")
private int fetchOrganizationsPoolSize;

@Value("${xroad-catalog.fetch-companies-pool-size:10}")
private int fetchCompaniesPoolSize;

}
2 changes: 1 addition & 1 deletion ...d-catalog-collector/src/main/java/fi/vrk/xroad/catalog/collector/tasks/BaseFetchTask.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
import lombok.extern.slf4j.Slf4j;

@Slf4j
public abstract class BaseFetchTask<T> {
public abstract class BaseFetchTask<T> implements Runnable {

protected final CatalogService catalogService;

Expand Down
Loading

0 comments on commit 35d33bf

Please sign in to comment.