Collection2 and meteor collection design practice?

[testbird]

Hi there,
from reading about meteor I picked up a prelimnary list of ideas concerning collection design. I think it would be interesting to discuss them with regard to collection2 and relation handling:

1. The database collections are primarily modelled for/after different pub/sub queries and to provide atomic documents, not as a normalized data object store.
2. Publications can contain documents from different collections.
3. The templates and partials model the (dom) objects that can join data from different (subscribed) collections, and are updated automatically when data changes.
4. The "less" css styles (with class inheritance) define the appearance (dom view).
5. Javascript event hooks are added to templates to modify data, which then triggers the computations of invalidated templates and code in Deps.autorun().

• Thus do a careful analysis of the most commonly used queries in your application.
• Design collections for optimized server/client data transfer.
• Collections should reflect/support the performance of the types of queries made.
• It can be ok to consolidate and merge multiple tables into one collection.
• Think in terms of documents with different schemas that may be in the same collection, rather than table schemas.
• In particular, instead of having to rely on a query that joins data from different tables, documents with different schemas may be fetched in one sweep from a single common collection.

However, separate collections can be handled easier when it comes to meteor publication/subscriptions.

And the templates (the dom model) can still join different queries together when rendered. As the data on the client is already reduced to a smaller subset, performance should be reasonably fast even with minimongo not yet supporting indexes.

• References (relation links) are used to reduce duplication/repetition.
• A reference may point to another document in the same or another collection.

[testbird]

As an idea, if the meteor appoach differs from traditional orm mapping: May it be reasonable to define multiple schemas for a C2 collection, and let C2 maintain an additional field in each document that states its schema?

[testbird]

New to noSQL the denormalization (data duplication) is a little uneasy to me.

Is there maybe a useful pattern with collection2 (hooks to use?) to automatically update the other duplicates when one is updated?

[aldeed]

When there are collection changes necessary as a result of other collection changes, I've found it easiest to set up observe on the server collections and make the secondary changes in the observe callbacks. The only drawback really is that rollbacks aren't supported like they would be with a transaction.

[testbird]

Ah, thank you for the pointer! That looks more meteoric and to work for insert/update/delete. I had only thought of autoForm hooks or collection-hooks so far.

[testbird]

Found this example now (thanks to your search terms): https://stackoverflow.com/questions/12464451/in-meteor-how-to-update-one-db-field-when-another-field-changes Is that like what you meant?

Would you think calling OtherCollection updates from autoValue functions would be viable alternative, to avoid the observe inefficienty and reactivity conflicts (like this one https://stackoverflow.com/questions/14250259/why-does-meteor-undo-changes-to-collections-nested-in-an-observer-method?rq=1), and to keep everything defined in the schema?

[aldeed]

Yes, that's essentially what I meant.

The issue you reference is just a client-side thing, I believe. I usually define the observe on the server only, so it's never a simulation. I also don't believe observe callbacks are all that inefficient, as long as you write them wisely.

I have a gut feeling against using autoValue for side effects like this. One major drawback is that you don't know for sure that the write operation will succeed when you're in an autoValue function. With observe or observeChanges, by contrast, you know for sure that the database was already updated so it's safe to do corresponding writes in other collections.

[testbird]

I see, yeah same applies to deny functions. Of course you raise the question, what is a wise observe function? :-) Like the example that runs on every add and change, but acts only on field changes to avoid ping pong beween two collections that update each other.

[testbird]

package isolate-value?

[aldeed]

isolate-value seems to be mostly for client side templates. I'm not sure if it will even be needed after Meteor UI is out.

I think wisdom for observe is things like using observeChanges if you can, using added instead of addedAt if you can, simple checks at the top of the functions to stop execution as soon as you can deem that no further changes are necessary.

[testbird]

Ok, the new Meteor UI hasn't been on my beginner's radar yet. I found the observe things you mentioned are coverd in the docs, so that is good.

Would you think the command queue method pattern (example from answer to 2. link above) could be a viable basis for transactions/rollback/undo in the relations handling etc.?

[aldeed]

You certainly could use custom methods as the answer in that link suggests. That will give you some limited transaction/rollback ability, but only if you write all of the logic yourself. I feel like rollbacks would be possible in server observes, too, but I can't recall if I've done it. You'd just have to write carefully to avoid infinite loops.

In general, these are the exact questions we're hoping to solve with the C2 relationships development.

You can test out Meteor UI by running your app from the shark branch. Info: https://github.com/meteor/meteor/wiki/New-Template-Engine-Preview

[testbird]

Wow, you're really going for the complete solution with relations.

I have thought about how to implement hierarchies/tags with c2 collections, and my current idea would be to use a separate collection to stores hierarchies with documents for arbitrary "namespaces" (e.g. user-tags, categories, user-types).

C2 fields of tag/hierachy documents:
namespace, [parentId], totalRefCount, [CollectionName: RefCount], materializedPath(autoValue)

Extra C2 fields on the data collection:
[namespace: hierarchyId], [namespace: materializedPath]

Finally, a collection-hook/observe to update the matializedPaths in the data collection when the hierachy changes, and to update the refCounts in the hierarchy collection on ref changes.

Does that sound reasonable?
Why were you not using collection-hooks (saying observe is easiest)? Without own experience they would look like the right tool to me at first sight.