Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement the Jena- and Elasticsearch-Access in create/update/delete as transaction #1135

Open
helkv opened this issue Jun 4, 2019 · 4 comments
Assignees
Labels
Projects

Comments

@helkv
Copy link

@helkv helkv commented Jun 4, 2019

To manipulate data objects in Imeji both Jena (primary data store) and Elasticsearch (search index) have to be accessed.
The access to Jena and Elasticsearch is implemented in two consecutive steps within the create/update/delete operations (in the WriterFacade).

In Imeji these operations are not implemented as transactions.

The missing transaction-implementation can lead to synchronization problems when errors occur during the write process:

  1. In some cases if writing in Jena fails, Elasticsearch is nevertheless updated.
  2. Also if the access to Elasticsearch fails, the changes already made in Jena are not rolled back.

These scenarios will result in an inconsistency between Jena and Elasticsearch.

Task: Implement create/update/delete as true transactions.

@helkv helkv added the enhancement label Jun 4, 2019
@helkv

This comment has been minimized.

Copy link
Author

@helkv helkv commented Jun 4, 2019

Also see #1122 and #1089 for associated issues.

@MPDLbrede MPDLbrede added this to To do in imeji 4.7 Aug 27, 2019
@MPDLbrede MPDLbrede moved this from To do to In progress in imeji 4.7 Aug 27, 2019
@haarli haarli removed this from In progress in imeji 4.7 Oct 16, 2019
@haarli haarli added this to To do in imeji 4.8 Oct 16, 2019
@haarli

This comment has been minimized.

Copy link

@haarli haarli commented Oct 16, 2019

According to @MPDLbrede

  1. already fixed by other ticket
  2. Possible solution: Retry-queue for ElasticSearch

Possible implementation of 2. moved to imeji 4.8

@MPDLbrede MPDLbrede moved this from To do to In progress in imeji 4.8 Nov 5, 2019
@MPDLbrede

This comment has been minimized.

Copy link

@MPDLbrede MPDLbrede commented Nov 5, 2019

Details:
(1) "In some cases if writing in Jena fails, Elasticsearch is nevertheless updated."
Before #1122 when writing (create/update/delete) to Jena and ElasticSeach, we had the following scheme (shown for update):

public void create(List objects, User user) throws ImejiException {
[...]
try {
Future createTask = executor.submit(new CreateTask(objects, user));
createTask.get();
Future indexTask = executor.submit(new IndexTask(objects));
indexTask.get();
} catch (Exception e) {
throw new ImejiException("Error updating objects", e);
}
}

Meaning that given a list of objects these were

  • first written to Jena and
  • once the Jena writing thread had returned (without error) written to ElasticSearch

In #1122 this was changed to :

public List create(List objects, User user) throws ImejiException {
[...]

try {
  List<Object> createdObjectsInJena = executor.submit(new CreateTask(objects, user)).get();
  executor.submit(new IndexTask(createdObjectsInJena)).get();
  return createdObjectsInJena;
} catch (ExecutionException | InterruptedException | CancellationException execExept) {
  if (execExept.getCause() instanceof ImejiException) {
    throw (ImejiException) execExept.getCause();
  } else {
    throw new ImejiException(execExept.getMessage());
  }
}

}

Meaning that now

  • in a first thread the given objects are written to Jena
  • The thread then returns the latest version of the objects in Jena
  • The returned objects are written to ElasticSearch in a second thread

Implies:
Any object written to ElasticSearch exists in Jena. This is because any object written (copied) to ElasticSearch has been read directly from Jena.

@MPDLbrede

This comment has been minimized.

Copy link

@MPDLbrede MPDLbrede commented Nov 12, 2019

Documentation of development tasks and times

EntwicklungsStand_#1135.xlsx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
imeji 4.8
  
In progress
3 participants
You can’t perform that action at this time.