Refactor to use store for all temp IO #3548

jelovirt · 2020-07-20T07:59:28Z

Description

Refactor to use Store for all temporary IO processing. Enable contributing Store implementations via service loader. Changes are not be visible to end users.

Motivation and Context

This separates refactoring needed for #3439. Then the in-memory and caching Store implementation can be merged without refactoring changes.

How Has This Been Tested?

All existing tests pass.

Type of Changes

New feature (non-breaking change which adds functionality)

Documentation and Compatibility

Contributing alternative Store implementations via org.dita.dost.store.StoreBuilder service loader may be documented, but it's very unlikely that plugin contributed Store implementations will be used.

jelovirt · 2020-07-20T08:01:47Z

src/main/java/org/dita/dost/ant/ExtensibleAntInvoker.java

@@ -316,10 +316,16 @@ public static Job getJob(final Project project) {
                project.addReference(ANT_REFERENCE_JOB, job);
            }
        } else {
+            XMLUtils xmlUtils = project.getReference(ANT_REFERENCE_XML_UTILS);


Store XMLUtils into Ant project reference, because we want to reuse the same Saxon Processor everywhere. This allows reusing the same name pool etc. that's needed for in-memory Store implementation.

Just curious -- in the past we had memory crashes in steps where Saxon ended up reading 5,000 or 10,000 topics (like the mappull step). Is reusing the same Saxon Processor going to have any impact on that sort of case, either better or worse, or is there no change?

No, the Processor class is what maintains the name pool etc. The XsltExecutable is the one that would be a problem and we don't reuse it by default.

jelovirt · 2020-07-20T08:03:58Z

src/main/java/org/dita/dost/ant/InitializeProjectTask.java

@@ -28,6 +28,11 @@
 * @since 3.5
 */
 public final class InitializeProjectTask extends Task {
+
+    private static ServiceLoader<StoreBuilder> storeBuilderLoader = ServiceLoader.load(StoreBuilder.class);


Use service loader to load StoreBuilder which in turn can be used to create the actual Store implementations.

jelovirt · 2020-07-20T08:04:33Z

src/main/java/org/dita/dost/ant/InitializeProjectTask.java

+
+    private static ServiceLoader<StoreBuilder> storeBuilderLoader = ServiceLoader.load(StoreBuilder.class);
+
+    private String storeType = "file";


Default StoreBuilder name, plain old temporary directory in disk implementation.

jelovirt · 2020-07-20T08:09:44Z

src/main/java/org/dita/dost/ant/types/JobSourceSet.java

@@ -58,9 +59,9 @@ public JobSourceSet() {
            for (final FileInfo f : job.getFileInfo(this::filter)) {
                log("Scanning for " + f.file.getPath(), Project.MSG_VERBOSE);
                final File tempFile = new File(job.tempDir, f.file.getPath());
-                if (tempFile.exists()) {
+                if (job.getStore().exists(tempFile.toURI())) {


All cases where we used to test file existance with File.exists(), we use Store.exists(URI) instead. The default StreamStore implementation will just test if the file exists. The added abstraction here allows in-memory Store to first check if the resource exists in memory, then fall back to checking if the resource exists in disk.

jelovirt · 2020-07-20T08:10:42Z

src/main/java/org/dita/dost/ant/types/JobSourceSet.java

                    log("Found temporary directory file " + tempFile, Project.MSG_VERBOSE);
-                    res.add(new FileResource(job.tempDir, f.file.toString()));
+                    res.add(new StoreResource(job, job.tempDirURI.relativize(f.uri)));


Custom Ant resource that can read from any Store, not just disk.

jelovirt · 2020-07-20T08:11:34Z

src/main/java/org/dita/dost/module/BranchFilterModule.java

@@ -374,10 +375,6 @@ private void generateCopies(final Element topicref, final List<FilterUtils> filt
                writer.setCurrentFile(dstAbsUri);
                final List<XMLFilter> pipe = singletonList(writer);

-                final File dstDirUri = new File(dstAbsUri.resolve("."));


No need to create directories into temporary directory in disk, the Store will handle that when resource is created.

jelovirt · 2020-07-20T08:16:38Z

src/main/java/org/dita/dost/module/DebugAndFilterModule.java

@@ -167,9 +161,9 @@ private void processFile(final FileInfo f) {

            in = new InputSource(f.src.toString());

-            final Serializer result = processor.newSerializer(outputFile);
+            final ContentHandler result = job.getStore().getContentHandler(outputFile.toURI());


Instead of manually creating a S9api Serializer ask for a ContentHandler from the Store; all IO to temp goes through Store.

jelovirt · 2020-07-20T08:18:29Z

src/main/java/org/dita/dost/module/MoveLinksModule.java

@@ -60,6 +62,8 @@ public AbstractPipelineOutput execute(final AbstractPipelineInput input) throws

            final XsltTransformer transformer = xsltCompiler.compile(new StreamSource(styleFile)).load();
            transformer.setErrorListener(toErrorListener(logger));
+            transformer.setURIResolver(new DelegatingURIResolver(CatalogUtils.getCatalogResolver(), job.getStore()));


The DelegatingURIResolver allow chaining XML catalog resolver and Store (which implements URIResolver), so that in the end Store will return the Source that is used to read the resource.

jelovirt · 2020-07-20T08:24:29Z

src/main/java/org/dita/dost/module/XsltModule.java

@@ -169,40 +151,62 @@ private void transform(final File in, final File out) throws DITAOTException {
            logger.debug("Set parameter " + filedirparameter + " to '" + v + "'");
            t.setParameter(new QName(filedirparameter), new XdmAtomicValue(v));
        }
+
+        if (properties.isEmpty()) {


If serialization properties (encoding, indentation etc.) from <xslt> element is empty, we can defer the transformation to Store. This means Store decides how to serialize the files. This allows in-memory Store to just create the resources and replace them in the internal cache, instead of first creating them in the cache with a temporary resource URI and then replacing them into the final resource URI.

jelovirt · 2020-07-20T08:28:08Z

src/main/java/org/dita/dost/module/XsltModule.java

+            if (same) {
+                destination.setDestinationBaseURI(out.toURI());
+            }
+            if (destination instanceof Serializer) {


Only set output properties if the destination is a Serializer.

Destination can be e.g. a tree builder that results in a S9api XdmNode, or then it can be something that serializes XML into a byte stream.

jelovirt · 2020-07-20T08:35:56Z

src/main/java/org/dita/dost/module/reader/TopicReaderModule.java

-                        break;
-                    default:
-                        break;
+            final XdmNode source = job.getStore().getImmutableNode(tmp);


Change reading resource from XML stream reader to S9api XdmNode. XdmNode is the XSLT/XPath document model that offers a fluent query API.

jelovirt · 2020-07-20T08:44:16Z

src/main/java/org/dita/dost/module/reader/TopicReaderModule.java

-                        break;
+            final XdmNode source = job.getStore().getImmutableNode(tmp);
+            logger.info("Reading " + tmp);
+            final Predicate<? super XdmNode> isTopicref = xdmItem -> MAP_TOPICREF.matches(xdmItem.getAttributeValue(QNAME_CLASS));


Off-topic, In cases like this, local variable in Java 10 would be nice for readability:

var isTopicref = xdmItem -> MAP_TOPICREF.matches(xdmItem.getAttributeValue(QNAME_CLASS));

jelovirt · 2020-07-20T08:47:57Z

src/main/java/org/dita/dost/reader/ChunkMapReader.java

@@ -232,7 +231,7 @@ private void chunkMap(final Element root) {
     * Create the new topic stump.
     */
    private void createTopicStump(final URI newFile) {
-        try (final OutputStream newFileWriter = new FileOutputStream(new File(newFile))) {
+        try (final OutputStream newFileWriter = job.getStore().getOutputStream(newFile)) {


Chunking code still uses byte streams to write the temporary XML files. Until chunking code is refactored or replaces, the Store API needs to support InputStream and OutputStream.

jelovirt · 2020-07-20T09:02:25Z

src/main/plugins/org.dita.base/xsl/preprocess/topicpullImpl.xsl

@@ -697,12 +705,12 @@ mode="topicpull:figure-linktext" and mode="topicpull:table-linktext"
      <xsl:when test="empty($targetElement)"/>
      <!--otherwise try pulling shortdesc from target-->
      <xsl:otherwise>
-        <xsl:variable name="shortdesc">
+        <xsl:variable name="shortdesc" as="node()*">


Without typing this variable will create a new document node as the root and we don't want that.

robander · 2020-08-06T14:58:56Z

src/main/java/org/dita/dost/store/Store.java

@@ -62,7 +97,7 @@
    Source getSource(URI path);

    /**
-     * Get immutable DOM document for file. If mutating methods are called,
+     * Get immutable DOM document for resouce. If mutating methods are called,


Really trivial comment, would be good to fix the typo "resouce", here and a few other spots

robander · 2020-08-06T15:03:54Z

src/main/java/org/dita/dost/store/ant/types/StoreResource.java

+     * @return true if this resource exists.
+     */
+    @Override
+    public boolean isExists() {


Another trivial comment that isExists feels like a weird because you're not checking if a file is exists, you're checking whether fileExists or uriExists or whatever?

It overrides a method from Ant's Resource class, so we can't rename it to anything that makes more sense.

robander · 2020-08-06T15:14:30Z

General question ... some of the most frustrating failures I've had with DITA-OT over the last couple years come from URI Exceptions, when the input is not a valid URI and the code crashes (without telling you what URI it was trying to validate).

I see a lot of URI variables in the new code -- is there any new testing for bad input URIs? Are external links going to be rewritten in any way, or are those left alone? For example:
<object data="https://www.youtube.com/watch?v=5gdIvjF99ac"> (object can't set scope=external)
<xref href="www.youtube.com/watch?v=5gdIvjF99ac"> (forgot the https)
<topicref href="missing file.dita"> (space instead of %20 )

jelovirt · 2020-08-07T06:31:06Z

All/most the places where we've added a URI field, are to replace direct InputStream use. So I don't see it as a risk that we start breaking old content that has invalid links. All those invalid links should be fixed during initial read, in the normalization/fix filters.

Signed-off-by: Jarno Elovirta <jarno@elovirta.com>

When the `store-type` parameter was added in b402e2f for #3548, it looks like the `@deprecated` attribute name may have been chosen from an IDE autocompletion process instead of the `@desc` attribute that provides the parameter description. This commit fixes the attribute name, so the description will appear in the docs at <https://www.dita-ot.org/dev/parameters/parameters-base.html#base__store-type>. Signed-off-by: Roger Sheen <roger@infotexture.net>

@Class

* Generate correct metadata wrapper for maps Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix getting language of a composite document without nested topic Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Throw warning on file reference what uses different case Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Update message text Co-authored-by: Roger Sheen <roger@infotexture.net> Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Split OASIS table test into individual tables Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Inline simple table normalization filter super class Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Refactor OASIS table normalization based on simpletable normalization Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Extract colspec generation into own method Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix vertical spanning Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Update version number Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Split simpletable tests Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Support nested simple table normalization Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add support for over 2 row spans Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix two equal parallel row spans Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Simplify test DITA resource Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Ignore foreign content When map contains flagging, ditaval-startprop element will contain prop elements without @Class. This will cause dita-ot:matches-shortdesc-class() in template patterns to throw errors due to missing @Class attribute. Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add rotate outputclass to entry output in HTML5 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Change rotate direction Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Save map after rewriting references Fixes regression from cf7d7be Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Update version to 3.5.2 and docs submodule Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Use argument long form in Docker deploy configuration Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Use unfiltered source topic as basis for topic fileinfo features Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Update file info fields for all topics that share the same source URI Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add steps wrapper for single step task Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Automatically bump Homebrew formula on release https://github.com/mislav/bump-homebrew-formula-action Signed-off-by: Roger Sheen <roger@infotexture.net> * Fix download URL generation Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Use manual dispatch Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Test with fork of homebrew-core repo Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Test workflow using forked action and tap Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Pass root map FileInfo to functions to reduce calls to Job for input map Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add support for multiple filter arguments Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add conversion arguments test Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add info on options that can be repeated Per dita-ot#3556 (comment) Signed-off-by: Roger Sheen <roger@infotexture.net> * Fix incorrect conversion usage line Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Refactor to use store for all temp IO Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add Store test Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Remove deprecated msgprefix variable Signed-off by: Robert D Anderson <gorodki@gmail.com> Update message DOTX071E to use proper params Signed-off-by: Robert D Anderson <gorodki@gmail.com> * Update version number and docs submodule Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Use Homebrew specific secret Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Use debug action for Homebrew workflow Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix input name in Brew bump action Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add second line to commit message The second line of the commit message is used for PR description body. * Retain mapref keyscope in retable Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Inject parent keyscope into reltable elements instead of using topicgroup Since reltable cannot have a @keyscope, we can push parent @keyscope there without worrying that we override the original @keyscope. Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Remove unused JAXP Transformer logging utility method Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add S9api message listener Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Throw exception on terminating xsl:message Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Log unsupported level as an info message and log an error about unsupported level Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix child key cascading to enable sibling keys Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Update version to 3.5.4 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Cache DitaClass instances Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix for dita-ot#3568, prefer public IDs when comparing descriptions for grammars stored in grammar cache. Signed-off-by: Radu Coravu <radu_coravu@sync.ro> * Use concurrent sets or lists Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add test for dita-ot#3573 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Reset table stack on tgroup dita-ot#3573 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add test case for broken table dita-ot#3566 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix indentation and remove change log comments Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Improve grammar pool test Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix indentation Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Enable nonworking dita.xsl.html5.cover extension point (dita-ot#2981) Signed-off-by: Shane Taylor <shane.taylor@cengage.com> * Use BufferedWriter to output Job for 200 fold performance improvement Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Make DitaClass package private and add tests Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Update docs submodule Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add support for reading debug information from S9api node Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add support for S9api node in DitaClass match method Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Use XdmNode for key reader Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Migrate tests to use XdmNode Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix bugs in keyref replacement filter Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix root element select expression Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix namespace binding in receiver output Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix walking document to pass all nodes, not just topicref elements Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix map processing on key rewrite Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Clean Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Allow FOP to generate changebars DITA-OT 3.4 upgraded the bundled Apache™ FOP library to version 2.4, which includes support for changebars, but did not enable that support. These changes remove the FOP-specific flagging overrides that disabled changebars in FOP, allowing the default PDF2 flagging routines to be applied. Fixes dita-ot#3511. Signed-off-by: Roger Sheen <roger@infotexture.net> * Fix build init order to initialise project after temp dir Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add generator=DITA-OT metadata to HTML header Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Extract Dublin Core into own stylesheet Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Clean metadata stylesheets Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Move Dublin Core to separate plug-in Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix tests for Dublin Core Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * [fix] note type notice missing from xhtml Signed-off-by: thendarion <16006640+thendarion@users.noreply.github.com> * Remove serialization of schemekeydef.xml and obsolete flag stylesheet Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Remove obsolete key processing from XHTML Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * In-memory Store that caches resources when possible Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Read and write subject scheme files with Store Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Generate subject scheme files into Store Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix HTML5 without list files Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add last modified to store Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Read and write job to store Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Don't create temp dir unless needed and verify temp dir URI has trailing slash Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Clean test Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Use store to get input/output stream Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix typo in parameter description attribute When the `store-type` parameter was added in dita-ot/dita-ot@b402e2f for dita-ot#3548, it looks like the `@deprecated` attribute name may have been chosen from an IDE autocompletion process instead of the `@desc` attribute that provides the parameter description. This commit fixes the attribute name, so the description will appear in the docs at <https://www.dita-ot.org/dev/parameters/parameters-base.html#base__store-type>. Signed-off-by: Roger Sheen <roger@infotexture.net> * Add missing terminal punctuation Information about default values is appended to the parameter description in the documentation, so it's important that each description is a complete sentence. Signed-off-by: Roger Sheen <roger@infotexture.net> * Fix setting html5.map.url property Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix broken Slack badge After DNS settings were changed a while back, the previous badge is no longer available at http://slack.dita-ot.org, which previously pointed to a self-hosted app on Heroku, but now redirects to the native Slack invite form at https://dita-ot.slack.com. This replaces the old badge with an SVG variant from https://shields.io. Signed-off-by: Roger Sheen <roger@infotexture.net> * Use constant for root scope ID Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add repeat argument to CLI to run process N times Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Share latest from oasis-tcs/dita and oasis-tcs/dita-techcomm Signed-off-by: Robert D Anderson <gorodki@gmail.com> * Add support for parallel XSLT Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add pool Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add parallel support to modules Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Refactor SAX pipe configuration Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Enable parallel XML filter pipe Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add parallel to topicpull Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add parallel processing to conref Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Use parallel processing in preprocess2 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add parallel processing to keyref Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Only allow parallel XSLT task without Ant's xmlcatalog Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Remove xmlcatalog from parallel XSTL task Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Document Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Clean Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Remove duplicate setter call Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Remove unused code Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix development build folder name Development versions of the distribution package are downloaded as `dita-ot-develop.zip`, which unpacks to reveal a folder named with the current version number and a suffix with the commit hash that the package is based on, such as `dita-ot-3.6.0-SNAPSHOT+61f95e5` Builds fail when run from this folder as described in dita-ot#2414. This commit changes the commit hash separator to the `@` sign, so the folder name is generated as `dita-ot-3.6.0-SNAPSHOT@61f95e5`, which allows builds to run from the folder without errors, and follows the convention GitHub uses to represent commits, such as dita-ot/dita-ot@61f95e5. In dita-ot#2414 (comment), @jelovirt expressed a preference for resolving the underlying issue rather than a workaround, but until that happens, we should ensure that our dev build packages can be run without renaming the folder. Fixes dita-ot#2414. Signed-off-by: Roger Sheen <roger@infotexture.net> * Fix workdir-uri generation bug Make sure URI for a directory will always end in a trailing slash. If the File points to a path that doesn't exist, File.toURI() will not output a trailing slash. Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix JobSourceSet cases where src is null Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix current file directory URI resolution when file doesn't exist Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Update Gradle wrapper to 6.7 Update Gradle distribution version & regenerate wrapper to pick up recent changes per https://docs.gradle.org/current/userguide/gradle_wrapper.html#sec:upgrading_wrapper ./gradlew wrapper --gradle-version=6.7 https://docs.gradle.org/6.7/release-notes.html Signed-off-by: Roger Sheen <roger@infotexture.net> * Fix for dita-ot#3558, update Apache FOP to 2.5, Batik to 1.13, PDF Box to 2.0.21, fop-pdf-images to 2.5, jcl-over-slf4j to 1.7.30 Signed-off-by: Radu Coravu <radu_coravu@sync.ro> * Add missing terminal punctuation Information about default values is appended to the parameter description in the documentation, so it's important that each description is a complete sentence. Signed-off-by: Roger Sheen <roger@infotexture.net> * Sort partial imports Signed-off-by: Roger Sheen <roger@infotexture.net> * Move hard-coded highlight domain styles to partial Signed-off-by: Roger Sheen <roger@infotexture.net> * Move hard-coded syntax diagram styles to partial Signed-off-by: Roger Sheen <roger@infotexture.net> * Move hard-coded long quote link styles to partial Signed-off-by: Roger Sheen <roger@infotexture.net> * Move hard-coded Boolean state styles to partial Signed-off-by: Roger Sheen <roger@infotexture.net> * Add Gradle Wrapper Validation Action https://github.com/marketplace/actions/gradle-wrapper-validation Signed-off-by: Roger Sheen <roger@infotexture.net> * Remove invalid code remnant from HTMLHelp plug-in - Amends f88cc9b - Fixes dita-ot#3627 Signed-off-by: Roger Sheen <roger@infotexture.net> * Only validate Gradle Wrapper on pull requests Per dita-ot#3629 (comment) Signed-off-by: Roger Sheen <roger@infotexture.net> * Remove hard-coded long quote citation alignment Amends 52e0b48, which adds the necessary rule to the Sass partial Per dita-ot#3632 (comment) Signed-off-by: Roger Sheen <roger@infotexture.net> * Fix repeat count in install and uninstall subcommands Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add TopicRefWriter unit test Signed-off-by: Jarno Elovirta <jarno@elovirta.com> Use parameterized test Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add GitHub Actions for unit tests Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add integration tests to GA and remove tests from Travis CI Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Remove disabled pdf2.index.skip param and deprecate old PDF indexing code Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Update Ant to 1.10.9 dita-ot#3613 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Update Commons IO to 2.8.0 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Move dist snapshot upload to GA Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix snapshot condition expression Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Remove snapshot GA condition Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix org/repo name in condition Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Catch exception from reading JPEG image dita-ot#3604 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Update version to 3.6 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add RELAX NG schemas to catalog Fixes dita-ot#3641. Signed-off-by: Roger Sheen <roger@infotexture.net> * Exclude RELAX NG schemas from 'dita-ot.jar' Signed-off-by: Roger Sheen <roger@infotexture.net> * Exclude resources catalog from 'dita-ot.jar' Signed-off-by: Roger Sheen <roger@infotexture.net> * Add license for DITA 2.0 grammar files dita-ot#3608 Signed-off-by: Robert D Anderson <gorodki@gmail.com> * Restore RELAX NG schemas to 'dita-ot.jar' Without these schemas in the .jar file, builds with project files fail with the following error: > Error: invalid input for CompactParseable - Amends 09c1d94 from dita-ot#3645 Signed-off-by: Roger Sheen <roger@infotexture.net> * Switch rotated table entries to 'vertical-rl' mode The 'sideways-lr' writing mode implemented in dita-ot#3541 is only supported in Firefox. Switching the writing mode to 'vertical-rl' should allow rotated text to appear in Chrome & Safari as well. While the initial 'sideways-lr' implementation rotated the contents of the cell 90 degrees counterclockwise as stipulated by the [DITA 1.3 spec](http://docs.oasis-open.org/dita/dita/v1.3/errata02/os/complete/part3-all-inclusive/langRef/base/entry.html#entry), that writing mode value is currently only supported in Firefox, so it seems better to rotate clockwise with 'vertical-rl' instead, which should be supported by more browsers (except IE). See https://developer.mozilla.org/en-US/docs/Web/CSS/writing-mode Signed-off-by: Roger Sheen <roger@infotexture.net> * Add support for multimedia oasis-tcs/dita#351 Signed-off-by: Robert D Anderson <gorodki@gmail.com> * Update doc to 3.6 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix NPE when initializing store Fixes dita-ot#3656 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Compile CSS with Dart Sass via sass-gradle-plugin The `jsass` compiler used up to DITA-OT 3.6 relies on LibSass, which has been [deprecated in favor of Dart Sass](https://sass-lang.com/blog/libsass-is-deprecated), and throws errors when building on the 64-bit extension of the ARM architecture. These changes replace jsass with the Sass Compile plugin for Gradle, which compiles sass or scss files using the official Dart Sass compiler: https://github.com/EtienneMiret/sass-gradle-plugin Signed-off-by: Roger Sheen <roger@infotexture.net> * Remove plug-in source comment Per dita-ot#3659 (comment) Co-authored-by: Jarno Elovirta <jarno@elovirta.com> Signed-off-by: Roger Sheen <roger@infotexture.net> * Update DOTX037W warning message Fixes dita-ot#3669. Signed-off-by: Roger Sheen <roger@infotexture.net> * Align `args.input` CLI description with docs Per dita-ot/docs#328 Signed-off-by: Roger Sheen <roger@infotexture.net> * Fix resolution of custom PDF resources. Fix for dita-ot#3688 Signed-off-by: Radu Coravu <radu_coravu@sync.ro> * Transform URIs which might have anchor. Fix for dita-ot#3677 Signed-off-by: Radu Coravu <radu_coravu@sync.ro> * Use sax source directly instead of toString() representation Fix for dita-ot#3689 Signed-off-by: Radu Coravu <radu_coravu@sync.ro> * Take into account URI may not be file when checking its existence Fix for dita-ot#3677 Signed-off-by: Radu Coravu <radu_coravu@sync.ro> * Properly compute absolute locations in merge map and topic parsers Fix for dita-ot#3687 Signed-off-by: Radu Coravu <radu_coravu@sync.ro> * Fix copy of topicmeta between keydef and keyref Fix for dita-ot#3694 Signed-off-by: Radu Coravu <radu_coravu@sync.ro> * Fix copy of topicmeta between keydef and keyref Fix for dita-ot#3694 Signed-off-by: Radu Coravu <radu_coravu@sync.ro> * Use delegating resolver dita-ot#3688 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Use utility method to null URI fragment and simplify test Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Refactor test to reduce setup boilerplate Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Test for file scheme for methods that use File operations Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Clean tests and remove tabs Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Fix formatting Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Refactor catalog setter code Extract catalog resolver into own variable to handle issue with varargs Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Add guard for unset logger Fixes dita-ot#3667 Signed-off-by: Jarno Elovirta <jarno@elovirta.com> * Update version to 3.6.1 and docs submodule Signed-off-by: Jarno Elovirta <jarno@elovirta.com> Co-authored-by: Jarno Elovirta <jarno@elovirta.com> Co-authored-by: Roger Sheen <roger@infotexture.net> Co-authored-by: Robert D Anderson <gorodki@gmail.com> Co-authored-by: Radu Coravu <radu_coravu@sync.ro> Co-authored-by: Shane Taylor <shane.taylor@cengage.com> Co-authored-by: thendarion <16006640+thendarion@users.noreply.github.com>

jelovirt added preprocess enhancement Changes to an existing feature labels Jul 20, 2020

jelovirt self-assigned this Jul 20, 2020

jelovirt added this to In progress in Next via automation Jul 20, 2020

jelovirt commented Jul 20, 2020

View reviewed changes

jelovirt force-pushed the feature/incremental-store-improvements branch 3 times, most recently from 3d15853 to 13d694e Compare July 20, 2020 09:35

jelovirt moved this from In progress to Review in progress in Next Jul 20, 2020

jelovirt requested a review from robander July 29, 2020 08:32

robander reviewed Aug 6, 2020

View reviewed changes

jelovirt force-pushed the feature/incremental-store-improvements branch from b7c6cc9 to 6cc6e58 Compare August 7, 2020 06:45

jelovirt added 2 commits August 7, 2020 10:00

Refactor to use store for all temp IO

b402e2f

Signed-off-by: Jarno Elovirta <jarno@elovirta.com>

Add Store test

e348e25

Signed-off-by: Jarno Elovirta <jarno@elovirta.com>

jelovirt force-pushed the feature/incremental-store-improvements branch from 6cc6e58 to e348e25 Compare August 7, 2020 07:00

jelovirt requested a review from robander August 7, 2020 07:56

robander approved these changes Aug 7, 2020

View reviewed changes

Next automation moved this from Review in progress to Reviewer approved Aug 7, 2020

jelovirt merged commit 6542add into develop Aug 7, 2020

Next automation moved this from Reviewer approved to Done Aug 7, 2020

jelovirt deleted the feature/incremental-store-improvements branch August 7, 2020 12:42

jelovirt added this to the Next milestone Aug 7, 2020

infotexture mentioned this pull request Oct 11, 2020

Store API documentation dita-ot/docs#309

Merged

infotexture mentioned this pull request Oct 11, 2020

Fix typo in parameter description attribute #3607

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor to use store for all temp IO #3548

Refactor to use store for all temp IO #3548

jelovirt commented Jul 20, 2020

jelovirt Jul 20, 2020

robander Aug 6, 2020

jelovirt Aug 7, 2020

jelovirt Jul 20, 2020

jelovirt Jul 20, 2020

jelovirt Jul 20, 2020

jelovirt Jul 20, 2020

jelovirt Jul 20, 2020

jelovirt Jul 20, 2020

jelovirt Jul 20, 2020

jelovirt Jul 20, 2020

jelovirt Jul 20, 2020

jelovirt Jul 20, 2020

jelovirt Jul 20, 2020

jelovirt Jul 20, 2020

jelovirt Jul 20, 2020

robander Aug 6, 2020

jelovirt Aug 7, 2020

robander Aug 6, 2020

jelovirt Aug 7, 2020

robander commented Aug 6, 2020

jelovirt commented Aug 7, 2020


		private static ServiceLoader<StoreBuilder> storeBuilderLoader = ServiceLoader.load(StoreBuilder.class);

		private String storeType = "file";

Refactor to use store for all temp IO #3548

Refactor to use store for all temp IO #3548

Conversation

jelovirt commented Jul 20, 2020

Description

Motivation and Context

How Has This Been Tested?

Type of Changes

Documentation and Compatibility

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robander commented Aug 6, 2020

jelovirt commented Aug 7, 2020