#18576 spring cleanup of starter import / reindex error handling by wezell · Pull Request #18584 · dotCMS/core

wezell · 2020-06-03T14:01:11Z

This is a bit of housekeeping (not a rewrite) on how the starter is generated and how it is imported.

removed most of the broken db locking/transaction stuff from the StartupTasksExecutor and importing a starter is now transactional.

moved the Import and Export Utils to their own classes and renamed them ImportStarterUtil and ExportStarterUtil

When serializing and deserializing objects as XML, I am using the BundleUtil which handles writing XML 1.0 correctly.

Heavy cleanup (though not a complete rewrite) on the ImportStarterUtil. It is better but there is risk that I might have been to aggressive and missed something.

I have starters and data sets that cannot be imported and/or reindexed without these changes. Reach out to me if you are looking to test one of them.

jdotcms · 2020-06-03T18:41:14Z

-		db.setSQL("select count(*) as test from inode");
+		return new DotConnect()
+                .setSQL("select count(*) as test from inode")
+                .getInt("test")>0;


Nice I didn't know this one

jdotcms · 2020-06-03T18:42:20Z

+
+
+
+            Try.run(() -> loadStarterSite()).getOrElseThrow(e->new DotRuntimeException(e));


you can replace this (e->new DotRuntimeException(e));
by this
(DotRuntimeException::new);

Kinda fancy

jdotcms · 2020-06-03T18:47:50Z

+                httpResponse.setHeader("Content-Disposition", "attachment; filename=" + zipFile.getName());
+            }
+
+            try(final OutputStream outStream = (download) 


…-xml

nollymar · 2020-06-08T16:31:42Z

+
+    public void executeStartUpTasks() throws DotDataException {
+
+        Logger.debug(this.getClass(), "Running Startup Tasks");


I think it's redundant because there is a Logger.info just below with the same info

…-xml

dev-dotcms · 2020-06-16T23:29:04Z

+                boolean identityOn = false;
+                String cName = _className.substring(_className.lastIndexOf(".") + 1);
+                String tableName = "";
+                if (classesWithIdentity.contains(cName) && DbConnectionFactory.isMsSql() && !cName.equalsIgnoreCase("inode")) {


Issue found: Position literals first in String comparisons for EqualsIgnoreCase

dev-dotcms · 2020-06-16T23:29:06Z

+     * @param out - Printwriter to write responses to Reponse Printwriter so this method can write to
+     *        screen.
+     */
+    private void doXMLFileImport(File f) throws Exception {


Issue found: Avoid variables with short names like f

dev-dotcms · 2020-06-16T23:29:07Z

    public Response indexList(@Context HttpServletRequest httpServletRequest, @Context final HttpServletResponse httpServletResponse, @PathParam("params") String params) {
        try {
-            InitDataObject init = auth(params,httpServletRequest, httpServletResponse);
+            InitDataObject init = auth(httpServletRequest, httpServletResponse);


Issue found: Avoid unused local variables such as 'init'.

dev-dotcms · 2020-06-16T23:29:08Z

+                }
+            } else if (_importClass.equals(User.class)) {
+                for (int j = 0; j < l.size(); j++) {
+                    User u = (User) l.get(j);


Issue found: Avoid variables with short names like u

dev-dotcms · 2020-06-16T23:29:09Z

+
+            if (_className.equals("Counter")) {
+                counter = true;
+            } else if (_className.equals("Image")) {


Issue found: Position literals first in String comparisons

dev-dotcms · 2020-06-16T23:29:10Z

+        }
+
+        // updating file_type on folder now that structures were added
+        DotConnect dc = new DotConnect();


Issue found: Avoid variables with short names like dc

dev-dotcms · 2020-06-16T23:29:11Z

+	private final String oraCreate = "CREATE TABLE \"DB_VERSION\" ( \"DB_VERSION\" INTEGER NOT NULL , \"DATE_UPDATE\" TIMESTAMP NOT NULL, PRIMARY KEY (\"DB_VERSION\") )";
+	private final String msCreate  = "CREATE TABLE db_version (	db_version int NOT NULL , date_update datetime NOT NULL, PRIMARY KEY (db_version) )";
+
+	private final String SELECT = "SELECT max(db_version) AS test FROM db_version";


Issue found: This final field could be made static

dev-dotcms · 2020-06-16T23:29:12Z

+                        return true;
+                    }
+                });
+            } catch (Exception e) {


Issue found: Avoid catching generic exceptions such as NullPointerException, RuntimeException, Exception in try-catch block

dev-dotcms · 2020-06-16T23:29:13Z

+                    final HashMap<String, String> dcResults = (HashMap<String, String>) l.get(j);
+
+                    DotConnect dc = new DotConnect();
+                    StringBuffer sb = new StringBuffer("insert into portlet values (?,?,?,?,");


Issue found: Avoid variables with short names like sb

dev-dotcms · 2020-06-16T23:29:15Z

+    @Path("/reindex")
+    @Produces({MediaType.APPLICATION_JSON})
+    public Response startReindex(@Context final HttpServletRequest request, @Context final HttpServletResponse response,
+                    @QueryParam("shards") int shards, @DefaultValue(DOTALL) @QueryParam("contentType") String contentType) throws DotDataException, DotSecurityException {


Issue found: Avoid reassigning parameters such as 'shards'

dev-dotcms · 2020-06-16T23:29:16Z


+	private final String createTableSQL() {
+
+	       return (DbConnectionFactory.isPostgres()) 


Issue found: Useless parentheses.

dev-dotcms · 2020-06-16T23:29:17Z

-            indexName = create(indexName, shards, live);
+            if(indexName == null)
+                indexName=ContentletIndexAPIImpl.timestampFormatter.format(new Date());
+            indexName = (live) ? "live_" + indexName : "working_" + indexName;


Issue found: Prefer StringBuilder (non-synchronized) or StringBuffer (synchronized) over += for concatenating strings

dev-dotcms · 2020-06-16T23:29:18Z

+            recordsToIndex = APILocator.getReindexQueueAPI().recordsInQueue();
+            Logger.info(DotCMSInitDb.class, "Records left to index : " + recordsToIndex);
+            // ten minutes
+            if(++counter>30000) {


Issue found: Avoid assignments in operands

dev-dotcms · 2020-06-16T23:29:19Z

+
+        // sorting folder identifiers by parent path in order to pass parent check
+        Collections.sort(folderIdents, new Comparator<Identifier>() {
+            public int compare(Identifier o1, Identifier o2) {


Issue found: Avoid variables with short names like o1

dev-dotcms · 2020-06-16T23:29:20Z

+
+                    DotConnect dc = new DotConnect();
+                    StringBuffer sb = new StringBuffer("insert into portlet values (?,?,?,?,");
+                    if (dcResults.get("narrow").equalsIgnoreCase("f") || dcResults.get("narrow").equalsIgnoreCase("false")


Issue found: The String literal "narrow" appears 4 times in this file; the first occurrence is on line 560

dev-dotcms · 2020-06-16T23:29:21Z

+                counter = true;
+            } else if (_className.equals("Image")) {
+                image = true;
+            } else if (_className.equals("Portlet")) {


Issue found: Position literals first in String comparisons

dev-dotcms · 2020-06-16T23:29:22Z

+        if (!this.validateZipFile()) {
+            throw new DotRuntimeException("starter data invalid");
+        }
+        final File backTemporalFile = new File(getBackupTempFilePath());


Issue found: Overridable method 'getBackupTempFilePath' called during object construction

dev-dotcms · 2020-06-16T23:29:23Z

+     */
+    protected boolean hasReindexRecords() throws DotDataException {
+        DotConnect dc = new DotConnect();
+        String sql = (DbConnectionFactory.isMsSql()) 


Issue found: Useless parentheses.

dev-dotcms · 2020-06-16T23:29:24Z

+
+    }
+    @CloseDBIfOpened
+	private static void loadStarterSite() throws Exception{


Issue found: A method/constructor should not explicitly throw java.lang.Exception

dev-dotcms · 2020-06-16T23:29:25Z

+            if (filter != null) {
+                for (Object obj : all)
+                    if (filter.includeIt(obj))
+                        l.add(obj);


Issue found: This statement should have braces

dev-dotcms · 2020-06-16T23:29:26Z


+	private final String pgCreate  = "CREATE TABLE db_version (db_version integer NOT NULL, date_update timestamp with time zone NOT NULL, CONSTRAINT db_version_pkey PRIMARY KEY (db_version));";
+	private final String myCreate  = "CREATE TABLE `db_version` (`db_version` INTEGER UNSIGNED NOT NULL,`date_update` DATETIME NOT NULL, PRIMARY KEY (`db_version`))";
+	private final String oraCreate = "CREATE TABLE \"DB_VERSION\" ( \"DB_VERSION\" INTEGER NOT NULL , \"DATE_UPDATE\" TIMESTAMP NOT NULL, PRIMARY KEY (\"DB_VERSION\") )";


Issue found: This final field could be made static

dev-dotcms · 2020-06-16T23:29:27Z

+            Logger.info(this, "Importing:\t" + _className);
+
+
+            List l = new ArrayList();


Issue found: Avoid variables with short names like l

dev-dotcms · 2020-06-16T23:29:28Z

+     * The path where tmp files are stored. This gets wiped alot
+     */
+    private String backupTempFilePath = ConfigUtils.getBackupPath() + File.separator + "temp";
+    private ArrayList<String> classesWithIdentity = new ArrayList<String>();


Issue found: Private field 'classesWithIdentity' could be made final; it is only initialized in the declaration or constructor.

dev-dotcms · 2020-06-16T23:29:29Z

+                    //by an NCLOB field. In the case of dot_containers table, the second field, CODE, is an NCLOB field. Because of this,
+                    //ordering is done only on the first field for the tables, which is INODE
+                    if(com.dotmarketing.beans.Tree.class.equals(clazz)){
+                        _dh.setQuery("from " + clazz.getName() + " order by parent, child, relation_type");


Issue found: The String literal "from " appears 6 times in this file; the first occurrence is on line 144

dev-dotcms · 2020-06-16T23:29:30Z

+import com.dotcms.content.elasticsearch.business.ContentletIndexAPIImpl;
+import com.dotcms.content.elasticsearch.business.IndiciesInfo;
+import com.dotmarketing.business.APILocator;
+import com.dotmarketing.common.db.DotConnect;


Issue found: Classes implementing Serializable should set a serialVersionUID

#18576 spring cleanup of starter import

f11b236

wezell mentioned this pull request Jun 3, 2020

Unable to import starter with XML 1.1 characters in it #18576

Closed

wezell added 3 commits June 3, 2020 11:44

#18576 fixing relationshipfields

ddb2f7c

#18576 testing round trip generation/import with customer data

f789489

#18576 merging master

56c6fc4

jdotcms reviewed Jun 3, 2020

View reviewed changes

freddyucv approved these changes Jun 4, 2020

View reviewed changes

dsilvam approved these changes Jun 5, 2020

View reviewed changes

Merge remote-tracking branch 'origin/master' into issue-18576-starter…

3c0a396

…-xml

nollymar reviewed Jun 8, 2020

View reviewed changes

Comment thread dotCMS/src/main/java/com/dotmarketing/db/DotCMSInitDb.java

nollymar reviewed Jun 8, 2020

View reviewed changes

wezell added 17 commits June 9, 2020 13:41

#18576 allow starter indexing to finish

009455d

#18576 starter imports cleanly

730d288

#18576 messages

9347cf6

#18576 now returning index stats

414c617

#18576 updating resource

b0d34f4

#18576 add timeout on delete

b6b04c3

#18576 async javascript refresh

571b321

#18576 merging indexresource and esindexresource

5054e9c

#18576 fixing log error

9f76a61

Merge remote-tracking branch 'origin/master' into issue-18576-starter…

b8e8972

…-xml

#18576 fixing bad data

64572f8

#18576 large customer reindex completed

ebe2461

#18576 display options for building indexes

6734a11

#18576 removing unneeded commit

0810fa8

#18576 get indexes for tests

b7b5526

Merge remote-tracking branch 'origin/master' into issue-18576-starter…

1f67450

…-xml

#18576 postman tests

095d6f5

dev-dotcms reviewed Jun 16, 2020

View reviewed changes

nollymar approved these changes Jun 17, 2020

View reviewed changes

nollymar merged commit 91b7953 into master Jun 17, 2020

delete-merged-branch Bot deleted the issue-18576-starter-xml branch June 17, 2020 17:49

This was referenced Jul 6, 2026

OpenSearch: esindex DELETE reports success without checking the result — migration fan-out swallows primary delete failures that previously surfaced as errors #36430

Open

fix(os-index): surface primary index-delete failures instead of silent 200 #36432

Open




		Try.run(() -> loadStarterSite()).getOrElseThrow(e->new DotRuntimeException(e));


		public void executeStartUpTasks() throws DotDataException {

		Logger.debug(this.getClass(), "Running Startup Tasks");


		private final String createTableSQL() {

		return (DbConnectionFactory.isPostgres())

		Logger.info(this, "Importing:\t" + _className);


		List l = new ArrayList();

Uh oh!

Conversation

wezell commented Jun 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

wezell commented Jun 3, 2020 •

edited

Loading