Feat: added sync functionality

[harshithmullapudi]

No description provided.

[codestorybetabot]

✨ CodeStory generated PR Summary:

1: This pull request introduces a new SyncModule in engine-server.src.modules.sync.sync.module. The diff shows that the module is being added for the first time. As per the code graph, this module does not have any dependencies with other parts of the code in this pull request.
2: The pull request introduces a new interface GetTicketsQueryParams in the engine-server.src.modules.categories.ticketing.models.ticket.tickets.interface file. This interface doesn't seem to have any dependencies.
3: This PR introduces a new interface PathParamsWithCollectionId in the engine-server.src.modules.categories.ticketing.models.ticket.tickets.interface file. This interface has no dependencies on other changes in the PR.
4: The pull request introduces a new Transform function in engine-server.src.modules.categories.ticketing.models.ticket.tickets.interface. This function checks and returns whether the input value is 'true' in any of the following forms: 'true', 'True', or a boolean true. The Transform function does not have any dependencies.
5: The pull request includes modifications to the PathParams interface in the engine-server.src.modules.categories.ticketing.models.ticket.tickets.interface directory. The changes are isolated and do not affect any dependencies.
6: This pull request introduces changes to improve ticket retrieval in the ticketing module, and includes updates to the database query utility functions.

• The getCollectionTickets method in TicketsController now leverages the getCollectionTickets method from TicketService.
• The getCollectionTickets method in TicketService has been updated to decide between fetching tickets in real-time (getTicketsForRealtime) and fetching from the database (getTicketsFromDb).
• The getTicketsFromDb method now uses the getListFromDb method in both TicketService and CollectionService.
• The getListFromDb method in both TicketService and CollectionService have been updated to utilize the pagination, getBaseQuery, applyDateFilter, and getMetaParams utility functions from the knex.get module.
• The getBaseQuery function has been enhanced to use the getSelectKeys utility function.
• Finally, new utility functions getSelectKeys, applyDateFilter, and getMetaParams have been added to the knex.get module, and the pagination utility function has been added to the utils module.
  7: The pull request primarily involves refactoring the getTickets method in TicketsController and TicketService (under engine-server.src.modules.categories.ticketing.models.ticket).

In TicketsController.getTickets, the API call getDataFromAccount has been replaced with this.ticketService.getTickets for retrieving tickets. The parameters passed to the function have also been simplified.

In TicketService.getTickets, a new implementation has been provided which retrieves the ticket list directly from the database using the getListFromDb method. This involves querying based on integrationAccount.workspaceName, DATABASE_NAME, and integration_account_id.
8: The getTicketId method in TicketsController has been refactored. The original implementation of fetching data from the account has been replaced by a call to this.ticketService.getTicket. This simplifies the overall code and makes it more readable.
9: The init method in the IntegrationAccountService class (engine-server.src.modules.integration_account.integration_account.service.IntegrationAccountService.init) has been added. This method retrieves all integration accounts where syncEnabled is true using prismaService and initiates synchronization using syncService. No dependencies with other changes are noted.
10: The pull request primarily enhances the IntegrationAccountService.createIntegrationAccount function in the engine-server module. It introduces a check for syncEnabled and if true, it verifies the integration type and throws an exception if it's not a ticketing category. If the status is true, it creates an integrationAccount and if syncEnabled is true for the account, it triggers the SyncService.createScheduleIfNotExist and SyncService.runInitialSync methods.

The SyncService.createScheduleIfNotExist method tries to fetch a schedule and if it doesn't exist, it creates a new one by invoking the SyncService.createSchedule method. This method calculates the interval and offset for the schedule based on the syncPeriod and creates a schedule using these values.

The SyncService.runInitialSync method triggers an immediate sync operation.

The engine-worker module introduces a runSync workflow that executes preSync, runCloudQuerySync, and postSync activities. The preSync activity creates a new job and saves the configuration. The runCloudQuerySync activity executes the sync operation and updates the job status based on the outcome. The postSync activity updates the job logs.

The pull request also introduces several utility functions and interfaces in the engine-worker module to support the new activities and workflows. These include functions to generate and save configuration, create and update jobs, and extract information from logs. It also includes interfaces for IntegrationDefinition, Workspace, and IntegrationAccount.

In the engine-server module, utility functions are introduced to calculate the interval and offset for the sync schedule.
11: This PR involves two key modifications in our codebase. Firstly, the deleteSyncSchedule function is introduced in SyncService (engine-server.src.modules.sync.sync.service), which deletes a specific schedule based on its ID. It returns a boolean value indicating the success or failure of the operation. Secondly, the deleteIntegrationAccount function in IntegrationAccountService (engine-server.src.modules.integration_account.integration_account.service) has been updated to utilize this new deleteSyncSchedule function. It fetches the integration account, deletes the corresponding sync schedule, and throws an exception if the deletion fails.
12: The updateIntegrationAccount function in the IntegrationAccountService has been updated to support sync functionality. The function now checks if the syncEnabled flag in the updateIntegrationAccountBody is set. If so, it verifies the integrationType to ensure it is of TICKETING type. It then updates the integrationAccount with the new data, including syncEnabled and syncPeriod fields, and calls the updateSchedule function of the syncService with the updated integrationAccount.
13: The method getIntegrationAccountWithIntegrationType in IntegrationAccountService located at engine-server.src.modules.integration_account.integration_account.service has been updated. Now, it includes workspace in the returned object and also returns workspaceName which is the slug of the workspace with hyphens removed.
14: The pull request primarily involves changes to the callbackHandler method in OAuthCallbackService and createIntegrationAccount method in IntegrationAccountController.

In callbackHandler, the call to createIntegrationAccount has been replaced with createIntegrationAccountWithLink.

In createIntegrationAccount, the method now accepts a single createIntegrationAccountBody object as parameter instead of multiple parameters. This change is likely related to the modification in callbackHandler as it is a dependent of createIntegrationAccount.
15: The fetchData method in TicketsPath class (located in integrations.ticketing.github.src.models.ticket.tickets.path) has undergone a simplification. The parameters sort, direction, state, and assignee_id have been removed from the final_params object. This change doesn't have any dependencies.
16: The Transform function in the engine-server.src.modules.categories.ticketing.models.collection.collection.interface has been added. This function returns a boolean value based on the string comparison of the input value. There are no dependencies related to this change.
17: The init method in SyncService has been added in this PR. The method establishes a new connection to a server, with the address defaulting to 'localhost:7233' if no TEMPORAL_ADDRESS environment variable is provided. The connection is then used to create a new client, which is stored in this.client. No dependencies on other changes in this PR.
18: The pull request introduces a new function initiate in SyncService (engine-server.src.modules.sync.sync.service.SyncService.initiate). This function iterates over integrationAccounts and calls the createScheduleIfNotExist function for each account.
19: The newly added method runInitialSyncWithId in engine-server.src.modules.sync.sync.service.SyncService generates a scheduleId and triggers the schedule with ScheduleOverlapPolicy.BUFFER_ALL. This change doesn't have any dependencies.
20: The updateSchedule function in SyncService has been updated. This method now includes a new updateFunction which modifies the scheduleDescription based on the syncPeriod from the integrationAccount. It then uses this updated scheduleDescription to update the schedule handle. The getInterval function is used as a helper function within updateFunction to calculate interval and offset values.
21: The pull request introduces a new SyncService class in the engine-server.src.modules.sync.sync.service module. This class currently does not have any dependencies.
22: This PR introduces a new functionality in the TicketService class for fetching tickets. The getTicket function now supports fetching tickets in real-time mode or from the database, depending on the query.realtime parameter.

• If query.realtime is true, it calls getTicketForRealtime which fetches ticket data directly from the account using the getDataFromAccount function.
• If query.realtime is false, it calls getTicketFromDb which fetches ticket data from the database using the getObjectFromDb function from the knex.get module.

The getObjectFromDb function has also been updated to use knex for querying the Postgres database.
23: The pull request introduces a new service class named TicketService in the engine-server.src.modules.categories.ticketing.models.ticket.ticket.service directory. This class doesn't have any dependencies on other changes in the pull request.
24: The pull request introduces three new workflow methods: runCloudQuerySync, preSync, and postSync in the engine-worker.src.workflows file. These methods are proxied versions of the corresponding activities in engine-worker.src.activities, with a maximum retry limit of 3 attempts and a start-to-close timeout of 90 minutes.
25: This PR introduces a new interface Params in the engine-worker.src.workflows module. The Params interface includes a single property integrationAccountId of type string. There are no dependencies related to this change.
26: The Form component in new_integration_account_form has been updated to include two new fields: syncEnabled and syncPeriod that are now passed to the createIntegrationAccount method in both IntegrationAccountService and IntegrationAccountController. The integrationDefinitionId is now fetched from integrationDefinition object. These changes allow for the configuration of synchronization settings when creating a new integration account.
27: The pull request includes a modification in the NewIntegrationFormProps interface located in engine-webapp.src.modules.integration_account.new_integration_account.new_integration_account_form. The integrationDefinitionId field has been replaced with integrationDefinition of type IntegrationDefinition from engine-worker.src.utils.config. This change implies that instead of just the ID, the entire integration definition object is now required.
28: The FormProps in new_integration_account_form has been updated. The integrationDefinitionId property has been replaced with integrationDefinition which is of type IntegrationDefinition. This change aims to provide a more comprehensive integration definition in the form rather than just the ID.
29: The NewIntegrationForm component in engine-webapp.src.modules.integration_account.new_integration_account.new_integration_account_form has been updated. The integrationDefinitionId prop has been replaced with the integrationDefinition object, which is then used to extract the integrationDefinitionId. This change is isolated and does not affect any dependencies.
30: The getInitialValues function in new_integration_account_form_utils has been updated to include two new initial values: syncEnabled and syncPeriod. The syncEnabled is set to false by default and syncPeriod is an empty string.
31: The CreateIntegrationAccountParams class in engine-webapp.src.services.integration_account.create_integration_account has been extended with two optional parameters: syncEnabled (a boolean) and syncPeriod (an enum SYNC_OPTION_ENUM). These parameters are used to configure sync options for an integration account.
32: This pull request introduces new functionalities in the CollectionService class in the ticketing module. The getCollections method now supports real-time querying. If query.realtime is set to true, it fetches collections using the new getCollectionsForRealtime method, otherwise it uses the existing getCollectionsFromDb method. The getCollectionsFromDb method has been updated to fetch collections from a specific database and workspace, while getCollectionsForRealtime retrieves collections directly from the account using the getDataFromAccount method.
33: This PR introduces changes to the CollectionService within the ticketing module. The getCollection method is now capable of fetching data either in realtime or from the database, depending on the query.realtime parameter. It leverages two newly introduced methods: getCollectionForRealtime and getCollectionFromDb.

• The getCollectionForRealtime fetches data directly from the account.
• The getCollectionFromDb retrieves data from the database using getObjectFromDb from the common knex module.
  34: The pull request introduces a new service class, CollectionService, under the path engine-server.src.modules.categories.ticketing.models.collection.collection.service. There are no dependencies associated with this change.
  35: This PR modifies the NewIntegration component in engine-webapp.src.modules.integration_account.new_integration_account.new_integration_account. Instead of passing the integrationDefinitionId directly to NewIntegrationForm, the PR changes it to pass the entire integrationDefinition object that matches the selectedIntegrationDefinition. The integrationDefinition object is retrieved from integrationDefinitions array using the find method. This change is related to NewIntegrationForm component, which likely uses the passed integrationDefinition.
  36: The GetTicketsParams interface located in integrations.ticketing.github.src.models.ticket.ticket.interface has been simplified. Optional parameters such as sort, direction, state, and assignee_id have been removed from the queryParams object.
  37: The run function in engine-worker.src.worker.run has been added. It establishes a connection using NativeConnection.connect with the Temporal server, then creates and runs a worker using Worker.create. The worker uses the workflows defined in ./workflows and the tasks from DEFAULT_QUEUE.
  38: The bootstrap function in engine-server.src.main has been updated to initialize the IntegrationAccountService at the start of the application. This involves getting an instance of IntegrationAccountService and calling its init method. This change is related to engine-server.src.modules.integration_account.integration_account.service.IntegrationAccountService.init and engine-server.src.modules.sync.sync.service.SyncService.init, implying that synchronization services are also initialized at the start of the application.
  39: The method getCollections in the class CollectionController has been refactored. The call to getDataFromAccount has been replaced with a call to this.collectionService.getCollections. The query parameters have been simplified to just query instead of a combination of defaultQueryParams and query. The changes do not seem to have any dependencies.
  40: The getCollection method in CollectionController has been refactored to use collectionService.getCollection() instead of the previous getDataFromAccount. The method signature remains the same, but the internal implementation now relies on collectionService, simplifying the function call and removing the need for explicit type casting.

I might make mistakes, please give feedback to the devs at founders@codestory.ai or on slack