_ensureIndex on more fields

[mitar]

Should we have indexes also on:

collection._ensureIndex
  priority: 1
  retryUntil: 1
  after: 1

getWork sorts by those fields.

I have not measured anything, this is just by observing queries.

[vsivsi]

Probably yes... I'm still living in a mindset where indexing is a developer decision, but I guess I already violated that some time ago. I don't have much time between now and the end of the year, so this kind of thing (not really a bug) will probably need to wait.

[mitar]

Should we add this then just to the documentation? To the section where we are mentioning indexes?

[rhyslbw]

+1 for just adding them in. Is there anytime these wouldn't want to be indexed?

[vsivsi]

@mitar If you want to implement this, I'm fine with it.

[mitar]

Fixed in dev branch.

[rhyslbw]

👍 FYI @vsiverson and @mitar I'm about to look into implementing Job Collection as a work queue for management tasks in Space.eventSourcing, such as snapshotting and projection rebuilds, so I'll be around if you need to bounce any ideas etc

[aldeed]

@mitar @vsivsi It seems like a single key index on status is also needed. This is currently listed as the slowest query on our Compose:

Using package version 1.2.2

[mitar]

Where does this query comes from?

[aldeed]

Oh good point, looks like it's from some code we have that checks for stuck jobs periodically. So I guess it's up to you whether you want the package to set up the index. I believe pkgs should ensure indexes for the queries they do, but whether you want to anticipate other common queries, that's personal preference I guess.

[mitar]

Yea, I think this should be done by your code then. I think this is clear that if you are doing some other queries it is on you to create indexes.

[mitar]

I see that we are also doing queries on status for finding stale jobs. But I do create our own indexes.

[mitar]

Hm, but those are no extra indexes, besides what is provided by this package.

But also indexes depends on the number of jobs you have. For not many jobs it might not be so needed.

[aldeed]

I don't think the type-status combined index helps when you're querying on status only

[mitar]

I don't think the type-status combined index helps when you're querying on status only

That is my understanding as well.

[vsivsi]

I agree that the package should only create indexes for queries it generates internally. Maintaining indexes isn't free, so generating indexes that you don't know will be needed is wasteful.

[vsivsi]

Mitar's additional indexes will be released in 1.3.0 on Atmosphere today.