Archive Later functionality (Lombiq Technologies: OCORE-72) #10935

hishamco · 2021-12-30T15:07:55Z

hishamco · 2021-12-30T15:09:19Z

I will try to add unit tests if requires. Refactoring the common APIs with PublishLater module could goes into another PR

hishamco · 2021-12-30T16:25:55Z

Strange, that VS tell that the cast is unnecessary, but the build fails here

deanmarcussen

Looks good, but we are still waiting for the background job infrastructure to support this correctly.

Fortunately I am in the middle of working on it, so this should wait until it is complete, as it will change and improve how small jobs like this can be scheduled.

hishamco · 2021-12-31T10:43:13Z

Looks good, but we are still waiting for the background job infrastructure to support this correctly.

I already started another PR in my local machine, I tried to add OC.ContentManagement.Scheduling which is quite useful to schdule anything related to content items, but again this might need to rely on Scheduling Infrastructure APIs which may introduce some new types. Again may be there's similarity with BackgroundTask or it could be move to the new infrastructure

@deanmarcussen could you please outline what you are planning to do to avoid duplicate the work, regarding this PR we could merge then make both ArchiveLater & PublishLater build on top of the new infrastructure

Piedone · 2022-01-05T13:08:37Z

If you haven't done it already, please also check if the button styling is OK if Publish Later is also enabled (or if there are any clashes by chance).

hishamco · 2022-01-05T15:07:30Z

please also check if the button styling is OK if Publish Later is also enabled (or if there are any clashes by chance).

Seems I saw something similar when I introduced a content scheduling APIs, I will double check if this happen if both modules enables. Thanks for this reminder

hishamco · 2022-01-08T08:26:51Z

@deanmarcussen any update on the work that you done in the infrastructure APIs, it will be good to share WIP PR if its possible

hishamco · 2022-01-08T09:29:11Z

If you haven't done it already, please also check if the button styling is OK if Publish Later is also enabled (or if there are any clashes by chance).

Perhaps is a browser cache issue

deanmarcussen · 2022-01-08T10:18:37Z

@deanmarcussen any update on the work that you done in the infrastructure APIs, it will be good to share WIP PR if its possible

It was demoed on Tuesday. You can see the video when it is on youtube.

hishamco · 2022-01-08T10:45:30Z

It was demoed on Tuesday

Oh really, I will check the Lombiq channel if the video is posted

Piedone · 2022-08-21T21:53:19Z

Since Dean is not replying under #5755, let's merge this. For that, please resolve the merge conflict, Hisham, then I'll review.

jtkech · 2022-08-22T05:35:03Z

@Piedone @hishamco

Just for info some quick thoughts (so need to be checked) that may also concern the PublishLater.

Sorry for the long comment as usual ;) but most of them are only descriptions.

Maybe in place of doing a joined .Query<ContentItem, ArchiveLaterPartIndex>(), we could do a query directly on the index .QueryIndex<ArchiveLaterPartIndex>(), so that most of the time we do a lighter query. We would need to add the ContentItemId to the index so that, and only if some items need to be archived, we can query them.

I saw that you set the Utc to null in an handler both for PublishLater and ArchiveLater, but I think it is better to use the Published and Latest in the query delegates. For example it fails when Unpublishing a non latest version, the handlers are not called if already published or not published. Hmm, and what happens if we soft delete an item still marked as to be published/archived later.

Updated: Okay, normally on soft delete the index provider remove it from the index table.

So maybe filter the query on Latest and not Published for publishLater, and Latest and Published for ArchiveLater. The logic needs to be checked but the goal is to prevent some use cases where items would always match forever, and also to have a better bounded query. And if we use a direct .QueryIndex() we would need to also add Published and Latest to the index (see below), as I'm now used to do.

Finally about the IDX_ArchiveLaterPartIndex_... used to improve a query (joined or not), it is optimized when the db find all fields it needs in this IDX_ non clustered index, e.g. the DocumentId used in a joined query, otherwise it is forced to use clustered data.

But if we use a QueryIndex() it will returns all the field of the Index including its Id field. That's why I'm now used to add this Id field to an IDX_... non clustered index when we plan to use a direct QueryIndex() on the related Index table (hmm maybe always good to do for this small int field).

For example

    // All fields are selected if not using a joined query but 'QueryIndex()' directly,
    // in that case to only use non clustured indexes, they need to include all fields.
    // So we now also add 'Id'.

    // Non clustered index for performance only.
    SchemaBuilder.AlterIndexTable<SomeIndex>(table => table
        .CreateIndex(
            "IDX_ReleaseIndex_DocumentId",
            "Id",
            nameof(ContentItemIndex.DocumentId),
            nameof(SomeIndex.ContentItemId),
           ...
           ..
            nameof(SomeIndex.Published),
            nameof(SomeIndex.Latest))

.

Piedone · 2022-08-24T20:00:58Z

Hisham, could you complete this, together with JT's remarks, please?

hishamco · 2022-08-25T17:37:34Z

Sure, hope I find time tonight to read @jtkech comments and apply the requested changes

hishamco · 2022-08-25T17:46:27Z

BTW I did a prototype for content scheduling APIs that may I share it later

hishamco · 2022-08-26T16:18:53Z

@jtkech I think there's no need for the ArchiveLaterPartHandler because of the filtering that you suggested, am I right?

One more thing @Piedone do we need to show Scheduled to be archived on ... message after that item has been archived? I notice similar behavior in OC.PublishLater module

Piedone · 2022-08-27T11:55:36Z

Do you mean a notification after saving? Yeah, that looks useful.

hishamco · 2022-08-27T13:41:27Z

I didn't mean the notification, there's a label show up even if the item has been archived, I'm not sure if it's act as indicator to the admins, but still I don't see any value if the item is archived

Piedone · 2022-08-29T07:10:40Z

Please show a screenshot of what you mean because I don't understand.

hishamco · 2022-08-31T21:02:15Z

src/OrchardCore.Modules/OrchardCore.ArchiveLater/Views/ArchiveLaterPart.SummaryAdmin.cshtml

+@if (Model.ScheduledArchiveUtc.HasValue)
+{
+    <div>
+        <span class="hint">@T["Scheduled to be archived on {0}", (object)(await DisplayAsync(await New.DateTime(Utc: Model.ScheduledArchiveUtc)))]</span>


@Piedone I mean this hint

Do you mean that we shouldn't display this once ScheduledArchiveUtc passed? That's a good idea, yes, in that case we should hide this.

Exactly. I'm still waiting for Jean to prove the latest changes, then I can hide the hint

So if we still set the date to null when the item is archived, nothing to do here

But it depends on what we may want, see my last comment in the main thread.

jtkech · 2022-09-01T22:59:55Z

@hishamco @Piedone

Sorry didn't have so much time but after a quick look most of things look good to me.

I think we should still set ScheduledArchiveUtc to null but it may depend on what we want.

If the bg task unpublish the item and then the user re-publish it manually, do we want that the item will be then unpublished again by the bg task "immediately". If not we should still set the date to null. Here I think we should set the date to null, unless we really want to force the user to edit the item to update the date to be able to re-publish an item that has been archived.
If the user unpublish the item manually before the bg task, and then re-publish it, do we want that the process has been interrupted by the first action (just like an anticipation of the bg task). Hmm maybe less important, here there are pros and cons, so as you want.

If we should set the date to null in the both above cases we should just keep the handler, if only for the first case one solution would be to set the date to null in the bg task itself (easy to do).

@Piedone

What you think about doing the same tweaks but for PublishLater, in a separate PR I think, the more sensitive part would be a migration step, normally no problem to add columns but for the IDX_... non clustered index, as I remember we would need to drop it an then re-add it with all new included columns.

Piedone · 2022-09-02T11:49:40Z

I don't really see much appeal in making things more complex for some edge cases. I don't really understand how the index you mention relates to all this.

jtkech · 2022-09-02T15:21:51Z

@Piedone Sorry if I was not clear, here a summary

Here do we need to call again the handler to set the archive date to null, or just set it to null in the bg task itself, or not at all, it may depend on the above 1. and 2. in my previous comment, I let you choose.

Then do we need to do the same kind of tweaks for PublishLater, e.g. add Latest and Published to the index and use them, which would imply to add a migration step.

Piedone · 2022-09-02T15:48:00Z

Ah, I see. Yes, we should then go back to the Publish Later modules indeed.

hishamco · 2022-09-02T19:20:44Z

What you think about doing the same tweaks but for PublishLater, in a separate PR I think

I already created another PR after I made the update on this, please check #12276

jtkech · 2022-09-02T21:05:29Z

@Piedone @hishamco

Just for the fun another use case, if we use both PublishLater and ArchiveLater, if we don't set the schedule date to null, an item could be Published then Unpublished then Published ... every minute.

So looks good to reset both states that triggered the bg Task action, including the schedule date, for example in the bg Task itself, I will add a review comment.

src/OrchardCore.Modules/OrchardCore.ArchiveLater/Handlers/ArchiveLaterPartHandler.cs

src/OrchardCore.Modules/OrchardCore.ArchiveLater/Indexes/ArchiveLaterPartIndex.cs

src/OrchardCore.Modules/OrchardCore.ArchiveLater/Migrations.cs

src/OrchardCore.Modules/OrchardCore.ArchiveLater/Services/ScheduledArchivingBackgroundTask.cs

src/OrchardCore.Modules/OrchardCore.ArchiveLater/Startup.cs

hishamco · 2022-09-03T15:39:13Z

Just for the fun another use case, if we use both PublishLater and ArchiveLater, if we don't set the schedule date to null, an item could be Published then Unpublished then Published ... every minute.

I'm not sure how it will be a problematic, coz we already filter the items in the service, let me think

jtkech · 2022-09-03T19:52:48Z

@hishamco

Just for info

Yes this is confusing, we have Index Tables that we used for joined queries and that are regular SQL tables, then each index table, can have itself inner indexes, as you can see e.g. under Sql Mangement Studio if you expand the index table under its dedicated indexes "folder".

These inner indexes are not required but we add non clustered ones for performance only, those that are named IDX_Something, non clustered meaning that it can "cache" data outside the database clusters, so that for a given query it may prevent to hit again the database. But a given non clustered index may be quite useless if most of the queries request a column that is not included in this index (or another one).

That's why we have CreateMapIndexTable(), then AlterIndexTable() e.g. to .AddColumn() but also to create an inner index with .CreateIndex("IDX_Something).

hishamco · 2022-09-11T17:13:31Z

@Piedone can we file an issue descriping the background job infrastructure features list?

Piedone · 2022-09-11T18:50:09Z

Please do.

hishamco · 2022-09-11T20:02:10Z

I think we need @jtkech help on this, to mention what we should offer in such infrastructure? How should they different from background tasks we have

BTW I remembered I started a prototype long time back JUST to simplify the scheduling for both Archive & Publish Later modules

hishamco · 2022-09-11T20:13:12Z

May be we already have a rich info here

Adds ArchiveLater module

24c5623

Add DateTime? casting

92ac687

deanmarcussen requested changes Dec 31, 2021

View reviewed changes

Fix migration return value

1e8509a

sebastienros added the needs triage label May 12, 2022

Piedone mentioned this pull request Jul 18, 2022

Scheduled task infrastructure #5755

Open

hishamco added 4 commits August 25, 2022 23:23

Merge remote-tracking branch 'origin/main' into hishamco/archive-later

fdbf2b3

Address feedback

3734d5b

Use file-scoped namespace

b0cafe7

Sort usings

06e62c2

hishamco mentioned this pull request Aug 26, 2022

Use QueryIndex() for better performance in OC.PublishLater #12276

Merged

hishamco requested a review from jtkech August 31, 2022 20:58

hishamco commented Aug 31, 2022

View reviewed changes

jtkech reviewed Sep 2, 2022

View reviewed changes

Address feedback

df53326

jtkech approved these changes Sep 3, 2022

View reviewed changes

Piedone merged commit 83f7028 into main Sep 3, 2022

Piedone deleted the hishamco/archive-later branch September 3, 2022 20:17

Archive Later functionality (Lombiq Technologies: OCORE-72) #10935

Archive Later functionality (Lombiq Technologies: OCORE-72) #10935

Conversation

hishamco commented Dec 30, 2021 • edited by Piedone

hishamco commented Dec 30, 2021

hishamco commented Dec 30, 2021

deanmarcussen left a comment

Choose a reason for hiding this comment

hishamco commented Dec 31, 2021

Piedone commented Jan 5, 2022

hishamco commented Jan 5, 2022

hishamco commented Jan 8, 2022

hishamco commented Jan 8, 2022

deanmarcussen commented Jan 8, 2022

hishamco commented Jan 8, 2022

Piedone commented Aug 21, 2022

jtkech commented Aug 22, 2022 • edited

Piedone commented Aug 24, 2022

hishamco commented Aug 25, 2022

hishamco commented Aug 25, 2022

hishamco commented Aug 26, 2022

Piedone commented Aug 27, 2022

hishamco commented Aug 27, 2022

Piedone commented Aug 29, 2022

hishamco Aug 31, 2022

Choose a reason for hiding this comment

Piedone Sep 1, 2022

Choose a reason for hiding this comment

hishamco Sep 1, 2022 • edited

Choose a reason for hiding this comment

jtkech Sep 1, 2022

Choose a reason for hiding this comment

jtkech commented Sep 1, 2022

Piedone commented Sep 2, 2022

jtkech commented Sep 2, 2022

Piedone commented Sep 2, 2022

hishamco commented Sep 2, 2022 • edited

jtkech commented Sep 2, 2022 • edited

hishamco commented Sep 3, 2022

jtkech commented Sep 3, 2022

hishamco commented Sep 11, 2022

Piedone commented Sep 11, 2022

hishamco commented Sep 11, 2022

hishamco commented Sep 11, 2022

hishamco commented Dec 30, 2021 •

edited by Piedone

jtkech commented Aug 22, 2022 •

edited

hishamco Sep 1, 2022 •

edited

hishamco commented Sep 2, 2022 •

edited

jtkech commented Sep 2, 2022 •

edited