way to publish a transformed version of a cursor

[Tarang]

I was experimenting around with published data to help me with another issue #820

I have the following in my publish function

Meteor.users._transform = function(doc) {
    if(doc.services.facebook) {
                console.log("transforming");
        delete doc.services.facebook['accessToken']
    }
    return doc;
}

So the publish data doesn't seem to account transform (with both _transform and using the method defined in the docs), It does get called I can see it logging "transforming" so if this is intentional in some way there is a wasteful process occurring somewhere

I think I get why it does this, 'transform` might be used more for adding stuff like functionality which may need the same transform function to be passed down to the client so its passed as vanilla

I know the code is ok, because it does work on the client but I don't want the access token to reach the client.

Also is there a better pattern than the above, other methods do use the accessToken but setting transform this way makes it global

[cmather]

@Tarangp, To make sure I understand the expectation - if you pass a transform function in the constructor of your collection, you're expecting the publish function to apply the transformation before the data is sent over the wire? Could you not limit the fields returned in a regular Mongo query inside the publish function?

[Tarang]

I tried to limit the fields with the #820 but mongo won't let me so I thought I'd use this instead but It won't leave me with any options. I understand why publish can't send down a transformed collection it defies alot of logic, unless the transform is also on the client. But I tried everything pretty much. Editing the _collection before its wired down to to avail. There aren't many options left besides restructuring the user's data and having it update on every login.

[cmather]

Hey @Tarangp, Just commented on other thread since it seems like that's the root issue. Hopefully we can figure out a way for you to get this done. #820

[Tarang]

I think this needs a better solution as people might want to send down a transformed collection. Someone on SO today also wanted something, which is tricky because its hard to attach a collection to the Meteor.users collection without using _transform. On the server accessing the full document deck makes it very versatile

Related question:
http://stackoverflow.com/questions/15583290/get-a-collection-and-add-a-value-to-the-response

[jperl]

Agreed, consider this scenario: I want to send computed fields to the client based on private data on fields I do not want to send to the client.

[jperl]

In case someone else runs across this scenario. What I ended up doing (at least for the meantime) is to compute the field in EJSON.toJSONValue so it gets stored as part of the document.

[lalomartins]

Making transform work when publishing a cursor would solve a lot of use cases of more complex publishes.

In this case I don't want to publish NewsItems to the client at all:

Meteor.publish 'useritems', ->
    UserItems.find
        user: @userId
    ,
        transform: (doc) ->
            doc.item = NewsItems.findOne doc.item

[makertum]

Yes it would! Not only for removing fields, also for adding: I.e. for sending security credentials to the client that are generated with a private key on the server. Sometimes you might want to have them expiring quickly, so storing them in the db would make your db contain lots of expired credentials sooner or later. Then you have to clean it up. Better to generate them on publish!

[jadus]

An other example to show that this feature definitely needed :
I have a collection of geolocalized events. And before to publish them to the user I have to add some temporary fields to each event depending on the user's location (distance by car between user and event for example). These fields are calculated via the use of the mapquest web api. What I'd like is to transform the collection before publishing it. Is there an other way around while this feature is not implemented ?

[makertum]

Right now the best way is to create a new "read-only" collection, publish it to the client, store your additional fields in this new collection by using docs with the same _id, then use observe() and/or setInterval() to keep it in sync with the original collection.

[jadus]

Well I don't understand how that helps... Do you have an example ? Or can you give more details ? I don't understand how and why you store in a new collection fields that depends on the user's current location...

[makertum]

Hey Jadus, for your application i suggest on the server you do sth like:

var userExtraFields = new Meteor.Collection("UserExtraFields");

Deps.autorun(function() {
    var myCursor = Meteor.users.find({}, {fields: "profile.location"});
    myCursor.observeChanges({
        changed: function(id, user) {
            userExtraFields.update({_id: id}, {$set: {extraField: sthLocationBased(user.profile.location)}});
        }
    });
});

Meteor.publish("UserExtraFields", function() {
    return userExtraFields.find({_id: this.userId});
});

function sthLocationBased(_location){
    // do something with the location data, i.e. reverse the string
    return _location.split("").reverse().join("");
}

on the client just subscribe to "UserExtraFields" and there you go!

[dandv]

Adding fields containing server-side dynamic computations is a big general use case for sending down transformed collections. Another is add-on fields tacked on by a package, e.g. a custom display order without touching the user's original collection.

I hope this feature gets implemented soon! CC @Slava

@jperl: can you expand on how you worked around this limitation with EJSON?

[jperl]

@dandv I apologize it has been a while since I did this sort of thing -- and I think I found a better way since then.

If I remember correctly, I built a EJSON type that was like a wrapper with a computed property specifically for the client. I published that type instead of the raw collection object. Does that help? What's your specific use case?

[dandv]

@jperl: the use case is meteor-autocomplete server-side, where the server would sort documents from a collection in a custom way, and send a limited number of them to the client. Transformed results would be an easy way to tack that ordering onto the records.

[Tarang]

@dandv a way to send down transformed results is manually using this.added (http://docs.meteor.com/#publish_added).

You have to manage when the document changes with an observer but you get the equivalent of transformed documents.

[ghost]

Return an object into a data context via iron-router, or transform the cursor..but don't forget to reset it! Pretty sure the first way will work, not sure about second..plan to try both tomorrow. :)

[glasser]

We do not use GitHub to track feature requests (other than for Blaze). The core team's current roadmap is at https://roadmap.meteor.com/. Discussions about features that users desire are great topics for the meteor-talk mailing list, where the community can help come up with solutions that don't require core changes.

transform is our temporary hack until we have time to do a full pass on joins, models, and schemas, which are all on our roadmap.

[dandv]

@glasser: now that GitHub is used to track feature requests, how can this one be revived?

SO questions requesting this: 1, 2, 3, 4.

[mitar]

Hm, I created a nice pluggable API for this in meteor-middleware. So instead of just providing a transform, you can stack them one on another. This allows for code reuse, permissions checks (like removing or aggregating fields based on permissions), etc.

[mitar]

BTW, I have my own stack for all of this: schemas, migrations, joins, transforms. :-) I think Meteor already provides enough stuff, except for two things:

• way to officially re-register the same collection
• way to have observe/observeChanges without initial set of documents being processed

[IstoraMandiri]

@mitar,

For "way to have observe/observeChanges without initial set of documents being processed", does the _suppress_initial: true option solve your problem? Edit: looks like it only works on the client.

[mitar]

_suppress_initial is only for MiniMongo, not for MongoDB on the server.

[glasser]

Sure, I'll reopen (though it's great to see the community packages that help towards this).

There's some trickiness here --- observeChanges is carefully designed to map directly on to DDP, and putting an extra transform layer in the middle means you need to have more levels of caching or re-querying (not that Meteor doesn't have many of them already).

[hwillson]

Hi all - as mentioned in #821 (comment), Meteor already provides the necessary hooks to implement most of this (with a few small exceptions). Given that there are community packages available to help with this, and that there hasn't been any traction on this issue in over 2 years, I'll close it off as part of the on-going old issue cleanup process. If anyone disagrees with this decision, and would like to see the specific parts of this issue implemented that are missing (as mentioned in #821 (comment)), please consider opening a new issue to track those requirements specifically. Thanks!

[mitar]

For "way to have observe/observeChanges without initial set of documents being processed" there is already #3694.

For re-registering collections I do not remember if there is anything.