Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Add a Data Guide (or guides) #4014

justinbmeyer opened this issue Mar 7, 2018 · 9 comments

Proposal: Add a Data Guide (or guides) #4014

justinbmeyer opened this issue Mar 7, 2018 · 9 comments


Copy link

justinbmeyer commented Mar 7, 2018

TLDR: A guide to teach people how to use CanJS's data layer (can-connect, can-query-logic, etc). This will be created after can-query-logic is finished.

This was discussed on a recent live stream (28:37) and previously here (32:34) and here (22:43).


  • Adding in the comments the use cases you've encountered that you would like to see covered.
  • What type of service layer are you connecting to? What does it look like to query a list of items? Does your server handle relationships?

Sub Proposals


CanJS has a lot of powerful tools around data modeling - real-time, caching, "instance awareness".
But these tools are underutilized for probably two reasons:

  1. Lack of awareness of how they work, and what they do.
  2. Configuration difficulty. Our tools require a high degree of conformity to be used easily. CanJS expects your data to look a certain way, your services to work a certain way, etc. While these tools do make customization possible, it's hard to figure out.


I propose a series of guides for working with CanJS's data layer:


  • An overview of building a data-model layer against a "perfect" data service. The data-layer will be added piece-by-piece up to including everything in superMap. The goal will be to teach people the concepts and pieces in an ideal environment.

Then a bunch of micro guides on dealing with less than ideal environment:

  • Alternate query params
  • Alternate REST services (urls are wrong, data is wrong)

Then some micro guides on specific areas

  • Relationships
  • Sessions


Rough Content


The purpose of CanJS's data modeling packages is to make it
easy to connect your application to a backend data source.

While fetch, [can-ajax], and XMLHTTPRequest can be used directly in a CanJS application,
using CanJS's data modeling tools can solve some difficult problems with little configuration:

  • Provide a standard interface for retrieving, creating, updating, and deleting data.
  • Convert raw data from the server to typed data, with methods and special property behaviors.
  • Caching
  • Real time updates of instances and lists
  • Prevent multiple instances of a given id or multiple lists of a given set from being created.
  • Handle relationships between data.

When to use

  • You want some of the benefits above.
  • when you can have lots of consistency on the server

Core Concepts

  • Types
  • Queries
  • Data Interface
  • Instance Interface
  • Behaviors


Todo is a map-type, comprised of other types: Boolean, Number etc.

Serialized vs unserialized.


Represent sets of data. Can make use of the underlying type information.

How can {gt: 5} be made to generically work with String and Number types?

Data Interface

CRUD connection. All data involved is "serialized".

Instance Interface

Methods added to a type so you can CRUD remote representations of it.


Add or overwrite data and instance interface methods.

Use Cases


caching strategies

connecting to services for "normalized" sql-like databases (Bitballs type apps)

connecting to services for document-like databases (Bitcentive type apps)

how to handle the session

Singleton sessions like Bitcentive

  • Session.current - see if there is a current session. Observable.
  • new Session().save() - Create a new session.
  • Session.current.destroy() - Logout from anywhere.

how to handle relationships

    id: this.gameId,
    withRelated: ["stats",

Make sure that the list of stats will know who it belongs to:

stats: {
		Type: Stat.List,
		set: function(stats){
			if (stats) {
				stats[Stat.connection.listQueryProp] = {filter: {gameId: }};

			return stats;

Refs should make this easier.

  • There's a jsonapi version of this.
@chasenlehara chasenlehara changed the title Proposal: Data Guide Proposal: Add a Data Guide Mar 9, 2018
@justinbmeyer justinbmeyer changed the title Proposal: Add a Data Guide Proposal: Add a Data Guide (or guides) Mar 9, 2018
Copy link

mikemitchel commented Nov 28, 2018

I voted for this mostly for this reason

CanJS has a lot of powerful tools around data modeling - real-time, caching, "instance awareness".
But these tools are underutilized for probably two reasons:

Lack of awareness of how they work, and what they do.

Copy link

A couple of things I’ve seen in APIs that we should probably address:

  1. An endpoint that returns different data types, e.g. one API call that returns { data: { projects: [ ... ], todos: [ ... ] } } (how to make this call, how to manage caching, how to model the data, etc.)

  2. The opposite of 1, where there are different endpoints for slightly different types of data (e.g. /api/completed and /api/incomplete endpoints that return todos that have slightly different properties); how should you get all todos, how to manage caching, how to model data, etc.

Copy link

I think it makes sense to show how to work with Request Headers in CanJS (I assume this requires can-connect) and maybe other low level request manipulation.

Real life case is "Token based" authentication. For example JWT authentication is very popular nowdays, and in order to implement it, you have to add Bearer token to request header, and build flow around it. Check its expiration, try to get new token if expired, re-try original request with new token attached.

Internally we implemented this by behaviors, but since it looks like trivial scenario it makes sense to show how to do something like this.

Copy link
Contributor Author

@Lighttree, great suggestion! Anyway you can share some of the code for that behavior?

Copy link
Contributor Author

justinbmeyer commented Nov 30, 2018

One thing I've been thinking about, is how to include the session information with the query ...

  • especially when the session impacts the query
-> unauthorized 

/api/todos JWT
A -> unauthorized 
B -> [ todos for that user ] SIMILAR TO /api/todos?filter[createdBy]=5

   {name: "should not be in your list", createdBy: 6, assignedTo: 5} -> would get added to a list!!
  • What should be passed with the query?
    Todo.getList({token: XYZ, filter:{}})
    • token is not going to be on your data response ....
  • What could be read from some global state?
      return requestPromise

Copy link

I'll share more info on Monday since I don't have access to code right now.

Main issues that we faced during implementation is that we weren't able to access 'request headers' and "before send" logic of original request. So additionaly to behavior that modify request we had to use custom ajax wrapper to met requirements.

It might be that today there is better way to do this. Anyway I'll return with more technical details on Monday :)

Copy link

I had a really hard time trying to use can-connect that last time I had a look at it. I ended up rolling a rather simple interaction mechanism. This along with my shuttle-access for identity & access control gets me to where I need to be.

Any mechanism I use should be flexible enough to allow one to add any behaviour (if required) for various "events" within the interaction pipeline. If I need to add request headers then it should be a rather simple affair. In most cases dropping down to the relevant level does actually help but it would be nice to keep it in the can-connect component for the 80% cases.

For security I simply add a header like so:

$.ajaxPrefilter(function (options, originalOptions) {
    options.beforeSend = function (xhr) {
        if (access.token) {
            xhr.setRequestHeader('access-sessiontoken', access.token);

        if (originalOptions.beforeSend) {

I don't think session management should be baked in. It should, however, be possible to add some implementation that works through a common interface. These things can be tricky so if it isn't going to help much then it could also be excluded.

For me the main focus should be the data interaction. How do I go about retrieving the data into the relevant DefineMap or DefineList structures? I also do not want to set up a massive can-connect configuration for each endpoint. There should be something that covers most cases and options that could override some behaviour; in some cases a totally different can-connect may be required but that should not happen all-too-often.

I would still suggest that each of the interaction pipelines have a predefined set of steps (events) it passes through and that one could attach an observer to any of those events to enrich the state. This roughly follows a pipes-and-filters mechanism.

Only base-bones functionality that is truly common should be baked-in and any other mechanisms or behaviours should be added explicitly.

Copy link
Contributor Author


  • .log()
  • See proto-chain

Copy link

Hi, sorry it took so long, to return with some details..

Basically our solution includes 2 parts: custom behavior and "customizable" jQuery ajax.

Its role to append token to every request where this behavior used. If token is expired it should renew it by sending another request to endpoint that generates them. (tokens are short-living and saved in sessionStorage to use them for request during life-time)

After this token generated it should append new token to request and re-try original request. If it succeed - ok. If generation of token fails - we redirect user to login.

import connect from 'can-connect';
import $ from 'jquery';

// This is endpoint that generates new tokens...
import JWTAuthToken from './JWTAuthToken.model';

const methods = ['getListData', 'getData', 'createData', 'updateData', 'destroyData'];

let setHeaders = function (xhr) {
    if (this) {
        xhr.setRequestHeader('Authorization', `Bearer ${this.token}`);

const getToken = function (isRefreshNeeded) {
    let isJwtTokenExpired;
    let storedData = {};

    isJwtTokenExpired = ((new Date()).getTime() - /* token expiration date */) > 0;

    if (isRefreshNeeded || isJwtTokenExpired) {

    if (!sessionStorage.getItem('token')) {
        return JWTAuthToken.get({}).then((data) => {
            let time = (new Date()).getTime();
            sessionStorage.setItem('token', data.token);
            return data;

    storedData.token = sessionStorage.getItem('token');

    return $.Deferred().resolve(storedData);

const JWTBehavior = connect.behavior('JWTBehavior', (baseConnection) => {
    const behavior = {};

    for (let i = 0; i < methods.length; i++) {
        behavior[methods[i]] = function (params) {
            let prom = $.Deferred();
            getToken().then((data) => {
                let self = this;
                let ajaxConf = {
                    tryCount: 0,
                    retryLimit: 1
                ajaxConf.beforeSend = setHeaders.bind(data);

                ajaxConf.success = function (successData) {

                ajaxConf.error = function (xhr) {
                    if (this.tryCount <= this.retryLimit && xhr.status === 401) {
                        getToken(true).then((secondResponse) => {
                            this.beforeSend = setHeaders.bind(secondResponse);
                        }).catch(/* do something */);
                    } else {
                // This is possible due to custom ajax provided.

                baseConnection[methods[i]].call(this, params);
            }).catch(/* do something */);
            return prom;

    return behavior;

export default JWTBehavior;

After you can use such behavior in models:

import DefineMap from 'can-define/map/map';
import DefineList from 'can-define/list/list';
import connect from 'can-connect';
import url from 'can-connect/data/url/url';
import constructor from 'can-connect/constructor/constructor';
import canMap from 'can-connect/can/map/map';
import ajaxConfigurable from 'canjs-auth/ajaxConfigurable';
import JWTAuthBehavior from 'canjs-auth/JWTAuth';

const Entity = DefineMap.extend({ seal: false }, {
  // fields

Entity.List = DefineList.extend({
    '#': Entity

Entity.connection = connect([
], {
    url: 'secured/endpoint/entity',
    ajax: ajaxConfigurable(),
    Map: Entity,
    List: Entity.List

export default Entity;

This is done for Can@3 and maybe for Can@5 it can be done way more better.

@justinbmeyer justinbmeyer added this to Review in progress in Data Guide May 5, 2019
@justinbmeyer justinbmeyer moved this from Review in progress to In progress in Data Guide May 5, 2019
@justinbmeyer justinbmeyer moved this from In progress to To Do in Data Guide May 5, 2019
@nlundquist nlundquist moved this from To Do to Done in Data Guide Jun 25, 2019
@nlundquist nlundquist moved this from Done to Review in progress in Data Guide Jun 25, 2019
Data Guide automation moved this from Review in progress to Done Jul 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Data Guide

No branches or pull requests

6 participants