After adding a new field, how to get the default value into existing documents?

[joeytwiddle]

TLDR: Loopback uses the default for a field when a document is created with that field missing.

But wouldn't it be useful to also fill out that field when reading and writing documents?

(E.g. if a new field is added, but documents in the DB do not have a value for that field.)

If not, then what is the recommended approach for adding a new field, with a default, to an existing DB? A migration step?

---

I have added a new field to my model:

  @property({
    type: 'boolean',
    default: true,
  })
  isFoo: boolean;

Now, when I fetch an old existing document from the collection, I want to see it has the property isFoo: true. How can we achieve that?

---

Plan A: I fetch pre-existing instances of the model from MongoDB, but this new property is not present on the objects.

I was hoping to see isFoo: true on the objects I had fetched.

---

Plan B: I fetched a document, and saved it back:

const document = await myModelRepository.findOne({});
await myModelRepository.update(document!);

I was hoping doing this would set the default, storing isFoo: true in the DB.

But I actually saw isFoo: null stored in the DB. 😱 That seems like a violation of contract to me! (Given that Loopback's validation would reject such a document if I sent it to the API.)

---

Plan C: I tried to manually update the documents in the DB, using the defaults from new Model() but that object has no fields set at all, so I cannot see what the default value is supposed to be.

---

Plan D: This is what I have needed to do in practice (and also what derdeka mentions below). I run a migration script that checks for any documents with that field unset, and updates the value to true.

(This is basically Plan C except in Plan C I found I was unable to get the default value out of the model, so I have to hard-code the true in my migration script.)

This has a few disadvantages, not least of which needing to repeat the value true in a second place in the codebase. (Violates SSOT)

Another difficulty is finding these documents. For small collections, I simply do a find({}) and then loop them. But for larger collections, I found the query { field: { exists: false } } doesn't find the documents. For string fields, I found I had to do { field: { nlike: '.' } }. (I am currently working with @loopback/repository@2.8.0)

---

This was easy with Mongoose: Whenever documents were fetched, or written, any missing fields were populated with the default values.[1] That seems to me like desirable behaviour, which would be good in Loopback.

But as far as I can tell, Loopback 4 is only populating default values when I create a new document.

Is that the expected behaviour?

I built a fresh example project to ensure it wasn't just a problem with my project.

---

What am I asking for?

• Discuss whether populating default values should be a feature when reading and writing entities, instead of only when creating. (This is what mongoose does, and it I have found it quite useful.)

• Alternatively, please recommend how to achieve the population of new fields when needed (e.g. as a migration step).

• Document the default property on the models page, when it is applied, and how to migrate old documents. (The current documentation explains what it is, but not what it does.)

---

[1] Mongoose will skip population of defaults with sparse: true or validate: false options, useful when dealing with large numbers of records.

[derdeka]

@joeytwiddle I have no idea how it works with mongodb, but in mysql and postgres I use database level defaults which are added when the migration process runs. After that they automatically return the default value.

[achrinza]

This could potentially be a use-case for #487

[stale]

This issue has been marked stale because it has not seen activity within six months. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository. This issue will be closed within 30 days of being stale.

[joeytwiddle]

@derdeka MongoDB has no feature that I know of to set default fields.

Therefore I think the options available are:

1. Have loopback fill default values when reading documents (and maybe when writing documents too).

2. Loopback could expose a function that will check and update all documents in a collection. Like step 1. but it is manually triggered, and checks all documents in one call.

   This function could then be called by the developer occasionally (e.g. when a migration is performed).

3. The developer could manually add a migration script to update the DB in a DB-specific way.

   (E.g. for MongoDB, the script would update all relevant documents in the DB to reflect the new defaults, as in step 2. But for MySQL, it could just update the database-level defaults.)

4. Just like 3, but the migration script is automatically generated by Loopback. (I'm not sure whether @derdeka is using 3. or 4.)

Option 1: Might be simpler to implement, and could potentially be reused for all different types of database (even MySQL if desired).

However it would add an overhead when reading large batches of documents. (Mongoose works around this by offering .lean() or {lean: true} which the developer can add to a query to skip the validation step, when performance is a concern.)

Option 2: This could also work on any type of DB. But by only doing the work when requested, it would not affect runtime performance. However, it might take a long time to run this process over huge datasets. (Option 1 reduces migration downtime, in exchange for a slightly slower runtime.)

Options 3 and 4: These could be nice and efficient on MySQL, but would essentially fall back to option 2 for MongoDB.

[stale]

This issue has been marked stale because it has not seen activity within six months. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository. This issue will be closed within 30 days of being stale.

[stale]

This issue has been closed due to continued inactivity. Thank you for your understanding. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository.