Time-of-day scale?

[curran]

Hello,

Firstly, thank you for this great work on the new D3 modules.

I'm wondering, would it be possible to create a time scale that represents time-of-day, rather than absolute time? Here's one example visualization that uses such a scale (conceptually at least) to show tweet density by time-of-day (by @ericfischer):

Here's the data behind this visualization.

In this visualization, it looks like a linear scale and corresponding axis represents time-of-day based on hours. This seems like a reasonable approach, just coerce the values to numbers. Another approach might be to parse the values into Date objects with an arbitrary year, month, and day filled in, and format the axis ticks to only show the time of day. I'm wondering if you have had any other ideas for implementing such scales and axes for [interval]-of-[interval] in a more general way.

Similarly, it seems like day-of-week and month-of-year data need to be represented using ordinal scales on strings ("Monday", "Tuesday", etc.), but this seems like another place where the d3-time and d3-scale modules could somehow handle these cases with a more explicit representation of time intervals. This leads to a kind of combinatorial explosion with all the possibilities of nested intervals, of which these are several:

• minute-of-hour
• minute-of-day
• hour-of-day
• day-of-week
• day-of-month
• day-of-year
• month-of-year

Here's another related visualization case that shows hour-of-day vs. day-of-week - Day / Hour Heatmap.

Perhaps this is beyond the scope of your goals, but I'd be curious what you think about these cases.

[mbostock]

Yes, the way I typically solve this is I just pick an arbitrary day (typically January 1, 1900 due to the default behavior of d3-time-format), and then map all of my dates to that day. The simplest way of doing that (in D3 3.x) is something like this:

var parseTime = d3.time.format.utc("%H:%M").parse;
var midnight = parseTime("00:00"); // "Mon, 01 Jan 1900 00:00:00 GMT"

Then, make a time scale for the day:

var x = d3.time.scale.utc()
    .domain([midnight, d3.time.day.utc.offset(midnight, 1)])
    .range([0, width]);

The result:

I’m not sure it’d be possible for a scale to do this for you automatically, since it’s more of a data-processing step to pick a consistent representation for time-of-day (given that JavaScript only provides a natural representation for absolute time). I suppose you could standardize those arbitrary intervals, though, and then maybe do some modulus arithmetic so that any date passed to the scale outside the standard interval was converted. At any rate, the current solution doesn’t feel terrible.

There’s an interesting (and common!) problem with this particular use case, which is that by aggregating the data in UTC, you blur the local pattern slightly since the local time zone offset changed over the course of data collection. That can only be fixed by changing how the raw data was aggregated, though, since at this point the counts have already been assigned to the wrong bins.

[curran]

Thank you for this explanation and example. It makes total sense. Indeed, using JavaScript Dates to represent relative time does not feel terrible. I'm learning lots of things from studying this example, like passing negative values to tickSize, and how to measure text. Also, that's a great point about using local times before aggregation to show the pattern more clearly.

I noticed that your normalization does not account for the aggregation over 521 Twitter users. The Y axis shows the number of tweets per hour summed over that whole group, but if you divide by 521 when assigning d.rate, then the Y axis would be the average tweets per hour per user.

As a side note, the Vega-Lite project has relative time units in its roadmap.

Closing the issue, as I think the fundamental answer here is "it’s more of a data-processing step to pick a consistent representation for time-of-day (given that JavaScript only provides a natural representation for absolute time)". Thanks again.

[mbostock]

I noticed that your normalization does not account for the aggregation over 521 Twitter users.

Right. I’m plotting the rate of Eric’s feed (i.e., the number of tweets he needs to read every hour to not miss out). You could divide by the number of followers to estimate their average tweet rate (i.e., how often one of his followers typically tweets), but I doubt that this rate follows a normal distribution so I don’t think the average would tell you much. It’s more likely it follows a power law, with some users tweeting far more frequently than others and being responsible for a disproportionate part of his feed.

[curran]

Haha what a great article on "Fear of Missing out"! I never heard that term before.

I see what you mean. Interesting insight that tweet rates probably follow a power law. Thanks for clarifying.

[e-n-f]

@mbostock Yes, not too far from a power law, although the people at the top aren't as prolific as you'd expect, and there are a bunch of people credited with one tweet that I've never even heard of, so I don't know what they're doing in my API responses.

There's probably a better way to model it but it's not jumping out at me.

[mbostock]

Neat, Eric. Thanks for sharing!

[curran]

Nice! I wonder if those one-tweet people are from those "Promoted" tweets that Twitter injects into everyone's feed.