Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SO-2127 snomed rf2 importer issues #122

Merged
merged 30 commits into from
Jan 6, 2017
Merged

SO-2127 snomed rf2 importer issues #122

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
54b1ff8
SO-2127: fixing issues with RF2 importer
cmark Nov 24, 2016
3fe1824
SO-1782: store taxonomy defect information in data objects...
cmark Nov 25, 2016
0a46423
SO-2127: fix NPE when getting datatype for cd member rows
cmark Dec 6, 2016
c44cee9
SO-2127: skip cycle detection during import
cmark Dec 6, 2016
df2aa25
SO-2127: improve performance of in-memory ID service
cmark Dec 6, 2016
ce44373
SO-2127: extract relevant taxonomy information for nodes and edges...
cmark Dec 6, 2016
fb31a71
SO-2127: support bulk methods in import time component lookup impls
cmark Dec 6, 2016
7ebd046
SO-2127: refactor RF2 importer to support batch processing
cmark Dec 6, 2016
9613dd5
SO-2127: disable error reporting functionality
cmark Dec 6, 2016
923c222
SO-2127: batch RF2 processing improvements
cmark Dec 6, 2016
d38dcf8
SO-2127: get temp CDOResource outside the for loop (lang.refset import)
cmark Dec 6, 2016
0f9b4a1
SO-2127: remove unnecessary setId calls from member importers
cmark Dec 6, 2016
6de7228
Merge remote-tracking branch 'origin/develop' into issue/SO-2127-snom…
cmark Dec 7, 2016
51b9bc3
SO-2127: fix concrete domain member index entry building
cmark Dec 7, 2016
0331d77
SO-2127: use log message arguments instead of string concatenation
cmark Dec 7, 2016
c59ecd7
SO-2127: run index purge/optimize before creating version...
cmark Dec 7, 2016
9b96893
SO-2127: remove commit notification from notification queue
cmark Dec 8, 2016
a01efba
SO-2127: enqueue commit notifications when enabled in tx context
cmark Dec 8, 2016
59e33b5
SO-2127: remove temporary resource form lang. refset importer
cmark Dec 8, 2016
35eefec
SO-2127: change importer logger names to snomed.importer.rf2
cmark Dec 8, 2016
f7a3f56
SO-2127: reduce document load times during import
cmark Dec 9, 2016
8934bf7
SO-2127: rename constant from ON to TAB_SPLITTER
cmark Jan 4, 2017
07770db
SO-2127: fix comments in import time lookup services
cmark Jan 5, 2017
d2f5bce
SO-2127: include stack trace in log messages
cmark Jan 5, 2017
d3e48b6
SO-2127: use rows.clear instead of recreating List instance
cmark Jan 5, 2017
4105458
SO-2127: simplify processing of rows
cmark Jan 5, 2017
e3e04ed
SO-2127: rename component creation template methods
cmark Jan 5, 2017
efecf53
SO-2127: generate available identifiers in bulk load method for symmetry
cmark Jan 5, 2017
1b80174
SO-2127: avoid setting core component identifier twice
cmark Jan 5, 2017
b780da1
SO-2127: rename field extended to hasMapTargetDescription
cmark Jan 5, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
35 changes: 11 additions & 24 deletions ...rf2/src/com/b2international/snowowl/snomed/importer/rf2/model/AbstractSnomedImporter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,8 +174,8 @@ public AbstractSnomedImporter(final SnomedImportConfiguration<T> importConfigura
* Logs the given message to the dedicated import log file.
* @param message the message to log.
*/
protected void log(final String message) {
LogUtils.logImportActivity(IMPORT_LOGGER, userIdSupplier.get(), branchPathSupplier.get(), message);
protected void log(final String message, Object...arguments) {
LogUtils.logImportActivity(IMPORT_LOGGER, userIdSupplier.get(), branchPathSupplier.get(), message, arguments);
}

protected SnomedImportConfiguration<T> getImportConfiguration() {
Expand All @@ -197,7 +197,7 @@ protected final Collection<Component> getComponents(final Collection<String> com
@Override
public void preImport(final SubMonitor subMonitor) {

final String message = getPreImportMessage();
final String message = MessageFormat.format("Preparing {0} import", importConfiguration.getType().getDisplayName());
log(message);
subMonitor.beginTask(message, 1);

Expand All @@ -208,10 +208,6 @@ public void preImport(final SubMonitor subMonitor) {
}
}

private String getPreImportMessage() {
return MessageFormat.format("Preparing {0} import", importConfiguration.getType().getDisplayName());
}

private void createComponentStagingDirectory(final SubMonitor subMonitor) {

try {
Expand Down Expand Up @@ -406,8 +402,7 @@ private CellProcessor[] createValidatingCellProcessors() {
}

private ImportAction handlePreImportException(final SuperCSVException e) {
final String reason = null != e.getMessage() ? " Reason: '" + e.getMessage() + "'" : "";
log("Exception caught while reading release file. Continuing with next row." + reason);
log("Exception caught while reading release file. Continuing with next row. {}", e.getMessage());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the vararg Objects land in an SLF4J Logger, we could propagate exceptions as an additional parameter now, to get a stack trace along with the exception message in the log output. There are other opportunities to do so below.

importContext.getLogger().warn("Exception caught while reading release file. Continuing with next row.", e);
return ImportAction.CONTINUE;
}
Expand Down Expand Up @@ -441,8 +436,7 @@ private ComponentImportEntry getOrCreateImportEntry(final Map<String, ComponentI
try {
sliceStream = new FileOutputStream(sliceFile);
} catch (final FileNotFoundException e) {
final String reason = null != e.getMessage() ? " Reason: '" + e.getMessage() + "'" : "";
log("SNOMED CT import failed. Couldn't open output file '" + sliceFile.getAbsolutePath() + "' for writing." + reason);
log("SNOMED CT import failed. Couldn't open output file '{}' for writing. {}", sliceFile.getAbsolutePath(), e.getMessage());
throw new ImportException("Couldn't open output file '" + sliceFile.getAbsolutePath() + "' for writing.", e);
}

Expand Down Expand Up @@ -562,8 +556,7 @@ protected int getImportWorkUnits(final int recordCount) {
}

private ImportAction handleImportException(final Throwable e) {
final String reason = null != e.getMessage() ? " Reason: '" + e.getMessage() + "'" : "";
log("Exception caught while importing row from release file. Continuing with next row." + reason);
log("Exception caught while importing row from release file. Continuing with next row. {}", e.getMessage());
importContext.getLogger().warn("Exception caught while importing row from release file. Continuing with next row.", e);
return ImportAction.CONTINUE;
}
Expand Down Expand Up @@ -639,15 +632,13 @@ protected void preCommit(final InternalCDOTransaction transaction) throws Snowow
}

private ImportAction checkCommitException(final SnowowlServiceException e) {
final String reason = null != e.getMessage() ? " Reason: '" + e.getMessage() + "'" : "";
log("SNOMED CT import failed. Caught exception while import, aborting." + reason);
log("SNOMED CT import failed. Caught exception while import, aborting. {}", e.getMessage());
importContext.getLogger().warn("Caught exception while import, aborting.", e);
return ImportAction.BREAK;
}

private ImportAction checkCommitException(final CommitException e) {
final String reason = null != e.getMessage() ? " Reason: '" + e.getMessage() + "'" : "";
log("SNOMED CT import failed. Caught exception while import, aborting." + reason);
log("SNOMED CT import failed. Caught exception while import, aborting. {}", e.getMessage());
importContext.getLogger().warn("Caught exception while import, aborting.", e);
handleCommitException();
return ImportAction.BREAK;
Expand Down Expand Up @@ -740,9 +731,9 @@ protected Predicate<T> getRowFilter() {

@Override
public void postImport(final SubMonitor subMonitor) {

subMonitor.beginTask(getPostImportMessage(), 2);
log(getPostImportMessage());
final String message = MessageFormat.format("Finishing {0} import", importConfiguration.getType().getDisplayName());
subMonitor.beginTask(message, 2);
log(message);

// Tear down in opposite order
try {
Expand All @@ -753,10 +744,6 @@ public void postImport(final SubMonitor subMonitor) {
}
}

private String getPostImportMessage() {
return MessageFormat.format("Finishing {0} import", importConfiguration.getType().getDisplayName());
}

private void createIndexes(final SubMonitor subMonitor) {

final String message = MessageFormat.format("Creating indexes for {0} import.", importConfiguration.getType().getDisplayName());
Expand Down
2 changes: 1 addition & 1 deletion .../com/b2international/snowowl/snomed/importer/rf2/refset/AbstractSnomedRefSetImporter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ protected void createIdentifierConceptIfNotExists(final String identifierId) thr
if (identiferParentConcept == null) {
String message = MessageFormat.format("Reference set parent concept ''{0}'' not found in database.",
getIdentifierParentConceptId(identifierId));
log("SNOMED CT import failed. " + message);
log("SNOMED CT import failed. {}", message);
throw new ImportException(message);
}

Expand Down
31 changes: 13 additions & 18 deletions .../com/b2international/snowowl/snomed/importer/rf2/refset/SnomedLanguageRefSetImporter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,11 +204,9 @@ protected void preCommit(final InternalCDOTransaction transaction) throws Snowow


if (null == revisions) {

String message = "Cannot load revision for descriptions from store.";
log("SNOMED CT import failed. " + message);
log("SNOMED CT import failed. {}", message);
throw new SnowowlServiceException(message);

}

//mapping between SNOMED CT description storage key and the CDO revision
Expand All @@ -229,7 +227,7 @@ protected void preCommit(final InternalCDOTransaction transaction) throws Snowow

final BaseCDORevision memberRevision = (BaseCDORevision) newMember.cdoRevision();
final InternalCDORevision descriptionRevision =
(InternalCDORevision) Preconditions.checkNotNull(object, "Cannot find description revision. CDO ID: " + descriptionStorageKey);
(InternalCDORevision) Preconditions.checkNotNull(object, "Cannot find description revision. CDO ID: %s", descriptionStorageKey);

memberRevision.setContainerID(descriptionRevision.getID());
memberRevision.setContainingFeatureID(InternalEObject.EOPPOSITE_FEATURE_BASE - SnomedPackage.DESCRIPTION__LANGUAGE_REF_SET_MEMBERS);
Expand All @@ -240,21 +238,18 @@ protected void preCommit(final InternalCDOTransaction transaction) throws Snowow
//get revision delta for description
InternalCDORevisionDelta revisionDelta = (InternalCDORevisionDelta) lastSavepoint.getRevisionDeltas().get(descriptionRevision.getID());

if (null == revisionDelta) {

//create empty revision delta
revisionDelta = (InternalCDORevisionDelta) CDORevisionUtil.createDelta(descriptionRevision);
//register it
lastSavepoint.getRevisionDeltas().put(descriptionRevision.getID(), revisionDelta);


}
if (null == revisionDelta) {
//create empty revision delta
revisionDelta = (InternalCDORevisionDelta) CDORevisionUtil.createDelta(descriptionRevision);
//register it
lastSavepoint.getRevisionDeltas().put(descriptionRevision.getID(), revisionDelta);
}

//get current language reference set members for description
final CDOList cdoList = descriptionRevision.getList(SnomedPackage.eINSTANCE.getDescription_LanguageRefSetMembers());
//get current language reference set members for description
final CDOList cdoList = descriptionRevision.getList(SnomedPackage.eINSTANCE.getDescription_LanguageRefSetMembers());

//create add feature delta with the new member
revisionDelta.addFeatureDelta(createAddRevisionDelta(newMember, cdoList.size()));
//create add feature delta with the new member
revisionDelta.addFeatureDelta(createAddRevisionDelta(newMember, cdoList.size()));
}

final CDOResource resource = transaction.getOrCreateResource(TEMPORARY_LANGUAGE_MEMBER_ROOT_RESOURCE_NAME);
Expand Down Expand Up @@ -291,7 +286,7 @@ private String getReferencedComponentId(final SnomedLanguageRefSetMember member)

}

return Preconditions.checkNotNull(id, "Referenced component ID was null for reference set member. Member: " + member);
return Preconditions.checkNotNull(id, "Referenced component ID was null for reference set member. Member: %s", member);

}

Expand Down