[Reindex] : Perform Site check before Folder check

[jcastro-dotcms]

Reported during customer upgrade.
Reproducible in all versions of dotCMS.

During the upgrade process for a customer -- coming from dotCMS 4.3.3 -- the full ES Reindex returned errors with several contents. For example:

"An error occurred when mapping properties of Contentlet with ID '6d0ccc3a-cb2d-4470-a5a5-2216f90ff23f': Parent folder '/' in Site '88d50eff-98c9-4659-88a7-c71db9118b5e' was not found via API. Please check that the specified value points to a valid folder."

After taking a deeper look at the actual data with @manuel-miranda , we figured out that the ID 88d50eff-98c9-4659-88a7-c71db9118b5e is NOT a Site, but a simple contentlet. This is wrong as Contentlets can only live under System Host, a Site, or a folder. We noticed tat we are validating this scenario, but it's being called after validating the Contentlet's folder. Such a validation should happen after the Site has been validated.

We haven't been able to determine how the customer ended up with Contentlets pointing to an invalid Site.

To Reproduce
Steps to reproduce the behavior:

1. Create a simple Content Type with two fields: Site or Folder, and Text.
2. Create two sample contents of such a type pointing to SYSTEM HOST.
3. Go to the database, in the identifier table, and change the host_inode column to any Identifier that IS NOT A VALID SITE.
4. Kick off a full reindex, and you should get an error like this:

"An error occurred when mapping properties of Contentlet with ID '{CONTENTLET-ID}: Parent folder '/' in Site 'SITE-ID' was not found via API. Please check that the specified value points to a valid folder."

Expected behavior
The ES Reindex should indicate a problem with an invalid Site, and NOT an invalid folder.

Additional context
Misleading error messages take a toll on troubleshooting times for Support tickets.

[nollymar]

Passed internal QA:

dotcms_1         | 21:32:51.667  ERROR business.ContentletIndexAPIImpl - Can't get a mapping for contentlet with id_lang:bc1c3a20613a266d34e6877245ab3a3c_1 Content data: {owner=dotcms.org.1, identifier=bc1c3a20613a266d34e6877245ab3a3c, nullProperties=[wfExpireDate, wfPublishDate, wfActionAssign, description, wfActionId, wfActionComments, showOnMenu, filterKey, whereToSend, wfNeverExpire, _path_to_move, iWantTo, wfPublishTime, wfExpireTime], modDate=2021-08-16 16:30:24.789, fileName=Screen Shot 2021-08-16 at 4.12.38 PM.png, languageId=1, title=Screen Shot 2021-08-16 at 4.12.38 PM.png, inode=fb51653c-566a-412f-890e-a171b8cf378d, fileAsset=/data/shared/assets/f/b/fb51653c-566a-412f-890e-a171b8cf378d/fileAsset/Screen Shot 2021-08-16 at 4.12.38 PM.png, metaData={}, folder=SYSTEM_FOLDER, disabledWYSIWYG=[], sortOrder=0, modUser=dotcms.org.1, host=12345, stInode=33888b6f-7a8e-4069-b1b6-5c1aa9d0a48d}
dotcms_1         | com.dotcms.content.business.DotMappingException: An error occurred when mapping properties of Contentlet with ID 'bc1c3a20613a266d34e6877245ab3a3c':: Identifier 'bc1c3a20613a266d34e6877245ab3a3c' is pointing to a Site that is not valid: '12345'. Please manually change this record to point to a valid Site, or delete it altogether.
dotcms_1         | 	at com.dotcms.content.elasticsearch.business.ESMappingAPIImpl.toMap_aroundBody0(ESMappingAPIImpl.java:490) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotcms.content.elasticsearch.business.ESMappingAPIImpl$AjcClosure1.run(ESMappingAPIImpl.java:1) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjrt-1.9.2.jar:?]
dotcms_1         | 	at com.dotcms.aspects.aspectj.AspectJDelegateMethodInvocation.proceed(AspectJDelegateMethodInvocation.java:42) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotcms.aspects.interceptors.CloseDBIfOpenedMethodInterceptor.invoke(CloseDBIfOpenedMethodInterceptor.java:29) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotcms.aspects.aspectj.CloseDBIfOpenedAspect.invoke(CloseDBIfOpenedAspect.java:41) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotcms.content.elasticsearch.business.ESMappingAPIImpl.toMap(ESMappingAPIImpl.java:320) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotcms.content.elasticsearch.business.ContentletIndexAPIImpl.addBulkRequest(ContentletIndexAPIImpl.java:770) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotcms.content.elasticsearch.business.ContentletIndexAPIImpl.appendBulkRequest_aroundBody14(ContentletIndexAPIImpl.java:711) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotcms.content.elasticsearch.business.ContentletIndexAPIImpl$AjcClosure15.run(ContentletIndexAPIImpl.java:1) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjrt-1.9.2.jar:?]
dotcms_1         | 	at com.dotcms.aspects.aspectj.AspectJDelegateMethodInvocation.proceed(AspectJDelegateMethodInvocation.java:42) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotcms.aspects.interceptors.CloseDBIfOpenedMethodInterceptor.invoke(CloseDBIfOpenedMethodInterceptor.java:29) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotcms.aspects.aspectj.CloseDBIfOpenedAspect.invoke(CloseDBIfOpenedAspect.java:41) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotcms.content.elasticsearch.business.ContentletIndexAPIImpl.appendBulkRequest(ContentletIndexAPIImpl.java:688) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotcms.content.elasticsearch.business.ContentletIndexAPIImpl.appendToBulkProcessor(ContentletIndexAPIImpl.java:726) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotcms.content.elasticsearch.business.ContentletIndexAPIImpl.appendToBulkProcessor(ContentletIndexAPIImpl.java:661) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotmarketing.common.reindex.ReindexThread.runReindexLoop(ReindexThread.java:219) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at com.dotmarketing.common.reindex.ReindexThread.lambda$1(ReindexThread.java:155) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	at java.lang.Thread.run(Thread.java:829) ~[?:?]
dotcms_1         | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
dotcms_1         | 	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
dotcms_1         | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
dotcms_1         | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
dotcms_1         | 	at java.lang.Thread.run(Thread.java:829) [?:?]
dotcms_1         | Caused by: com.dotmarketing.exception.DotDataException: Identifier 'bc1c3a20613a266d34e6877245ab3a3c' is pointing to a Site that is not valid: '12345'. Please manually change this record to point to a valid Site, or delete it altogether.
dotcms_1         | 	at com.dotcms.content.elasticsearch.business.ESMappingAPIImpl.toMap_aroundBody0(ESMappingAPIImpl.java:354) ~[dotcms_21.09_999999.jar:?]
dotcms_1         | 	... 24 more

[bryanboza]

Fixed, tested with the provided steps and now we have the same log that @nollymar put in her comment, tested on release-21.09 // Postgres // FF

[jcastro-dotcms]

NOTE: The previous commit: 15f0186 refers to the dotCMS 5.3.8.7 LTS release instead.