Skip to content

@imathews imathews released this Apr 29, 2019

This release bundles major data cleaning improvements to the data uploader alongside numerous quality-of-life features and bugfixes in the project tool

🚀 New Features

  • Ability to copy transforms between projects
  • Ability to filter on a union variable without keeping it (#138)
  • Ability to duplicate new variables within a transform
  • Support for data cleaning pipelines as part of data ingest: retype variables, clean dates, drop records, recode values, and drop variables as you bring your data in from (potentially) multiple data sources
  • Integrations for data ingest: import data from Google Cloud Storage, BigQuery, Google Drive, Box, AWS s3, and Microsoft OneDrive
  • Added ability to resize the minimap in the project tool
  • Support import / export of value lists from the project tool

⚠️ Notable changes

  • Literals can no longer be referenced from within a join clause in a transform. This was done to streamline the most common use cases (matching on variable(s)) and to remove a common user error when a literal was provided instead of the intended variable. You can still filter records from a given table by providing the relevant clause in the "Filter rows" section

🐛 Bug Fixes

  • Make sure to fully validate all dates (e.g., 2000-02-31 is invalid — Feb has at most 29 days — and will be parsed as a string)
  • Fixed transform cancelling behavior
  • Fixed an issue where the left/right scrollbar on the project minimap was hidden
  • Fixed a significant figures issue when displaying negative summary statistic values
  • Fixed an issue when sorting projects by size on the workspace + admin panel
  • Fixed an issue in computing version deltas when variables are removed on the first version
  • Fixed an issue when trying to union more than two tables (#164)
  • Fixed a SQL generation issue when ordering on a variable that was not in the "Keep" column
  • Fixed a bug that could occur when escaping values in some string methods within the transform
  • Fixed an issue when new variables couldn't be referenced from within a join
  • Mark join as invalid if using <, >, etc. with full/left/right joins, as this generates invalid sql
  • Fixed a bug where whitespace characters supplied as a constant generated invalid sql
  • Fixed an issue where revert transform wouldn't properly reset values that were previously null
  • When uploading a file, if a variable is null throughout, don't set the type to 'string'

UI

  • Improved SQL formatting within transforms
  • Several improvements to the transform variable selector, including:
    • Right-click functionality to view summary statistics, rename, and retype
    • Improvements in displaying working with union variables
    • Automatic renaming when keeping variables with the same name from multiple tables (#139)
  • Right click functionality on filters and new variables within a transform
  • Redesigned value list interface within projects
  • Streamlined comments within a project
  • Improvements to viewing and editing the description on a transform
  • New monospace font
  • Edits made throughout the data upload / versioning process no longer need to be "applied"
  • The transform shift left / right icon is only visible when the transform is currently active, to avoid unintentional clicks (you can also shift any transform by right clicking on it)
  • Several improvements to dropdowns and input fields to better support keyboard-based navigation and quick value entry
  • Improved transform invalid messages
  • More prominent display of the cost to run a transform
  • Bundle limit and order clauses in the transform — this helps to discourage ordering on the full table, which can lead to ResourceExceeded errors during compute and doesn't have a meaningful impact on the results. In the rare case that a user wants to order the entire table, they can still pass a suitably large value for "limit"
  • Fixed some issues with interactive tooltips that made it difficult to hover onto the tooltip. Additionally, fixed some issues where non-interactive tooltips would block content unnecessarily.
  • Make sure code editor fills the screen; don't scroll to the bottom when opening the code viewer
  • Make sure whitespace in literals displays as a character
  • Minor changes to the "drop duplicates" UI to make clear which variables effect this clause
  • Display default type associated with value lists
  • Remove spaces in SAS/Stata files exported from Redivis to improve the scripting experience

🏠 Internal

  • Performance improvements to the uploader when simultaneously uploading multiple files
Assets 2

@imathews imathews released this Mar 11, 2019

This release includes an overhauled authentication system, updates to organization membership, improvements to the table export process, and the beta availability of our public API

🚀 New Features

  • You can now log in to Redivis through over 2,500 academic institutions around the world! Redivis will authenticate directly through your institution, allowing you to authoritatively communicate your affiliation(s) when applying for organizations' data on Redivis
    • Redivis is now an InCommon service provider, a U.S. based federation for inter-institutional authentication; InCommon is in turn part of eduGAIN, a global federation of academic institutions. More universities are regularly being added; if you cannot yet log in to Redivis through your academic email address, contact us.
  • The Redivis API is now publicly available as a beta product. Navigate to your workspace/settings to generate scoped access tokens, and utilize the documentation at https://apidocs.redivis.com to begin your exploration. New endpoints are already in the works, and don't hesitate to reach out with further feedback and requests.
  • Improved information modals within the project tool — easily view the source dataset(s) of any table, understand your access restrictions, and find entry points to download data, metadata, and query the node through the API.
  • Improved data export interface that provides clearer messaging around the export process as well as any export restrictions that may be currently applied.
  • Integration with ORCID, allowing you to easily and authoritatively link your ORCID iD to your Redivis account.
  • Update your profile with your primary discipline(s) to help collaborators and administrators better understand your work.
  • Additional filtering options for admins, largely around working with collections and permission groups.

⚠️ Notable changes

  • When you join an organization, you must provide the login (e.g., one of you@stanford.edu , you@gmail.com) by which administrators of that organization knows you. When working with that organization's data, you must have recently authenticated with this login; if you haven't, you may lose access to some of the organization's restricted data until you authenticate or provide new login credentials.

🐛 Bug Fixes

  • Fixes to the profile image editing interface
  • Fixed jumpTo shortcut (ctrl+K) on windows machines
  • Allow for newline characters in cells when importing non text-delimited file types (e.g., SAS, Stata)

UI

  • Improved messaging when you don't have access to nodes within a project, alongside clear entry points to resolve your access.
  • Improved sign-in flow to make it more intuitive and faster when you're reauthenticating on the same computer
  • Streamlined membership form UI

🏠 Internal

  • Improved error messaging around various API, login endpoints
Assets 2

@imathews imathews released this Feb 21, 2019

New homepage! And supplemental pages! Isn't it pretty?

🚀 New Features

  • New homepage at redivis.com, plus supplemental pages (features, about, security, etc...). We're maybe using the term "feature" loosely here, but we hope this gives our community a foundation in who we are, what we're up to, and where we're going
  • New "jump to" functionality — quickly navigate between pages, projects, datasets and organizations within Redivis. Click on the "Jump to" input in the top right, or press cmd/ctrl + K to navigate like a pro.
  • Improved search functionality to pull deeper from datasets' metadata — including variable value labels!
  • New status page at https://status.redivis.com — monitor for any service disruptions and follow our steps to remediation, though hopefully this page will stay mostly in the green.

🐛 Bug Fixes

  • Numerous bugfixes and improvements to metadata editing (especially when assigning value labels). No more excuses to leave those metadata unpopulated :)
  • Fixed an issue where some cell previous weren't displaying the first record in a table
  • Addressed an edge case where output type was undetermined when using constants in transforms
  • Fixed an off-by-one issue in searching datasets by timeframe

UI

  • Updates to the profile menu dropdown (what you see when clicking your profile image at top right
  • Cleaned up and standardized page footers
  • Standardized and smoothed some display issues with input fields across Redivis. More input UI love coming soon...
Assets 2

@imathews imathews released this Jan 18, 2019 · 1 commit to master since this release

This release includes major improvements to data uploading and versioning, including user-uploaded datasets. We've also various bugfixes and UI tweaks

🚀 New Features

  • User uploaded datasets! You can now upload your own data to Redivis and use them in your projects.
    • Keep datasets private, share with your colleagues, or make them public to the world
    • Utilize data cleaning and ETL tools to get your data up to shape
    • Leverage Redivis's robust data versioning infrastructure to store a comprehensive change log and version history of your datasets
    • Initially limited to 1GB per dataset / 10 datasets per user, though we plan to lift these limits in the future. (contact us if your use case requires an increase to these limits)
    • Closes #149
  • Full featured data versioning system
    • Store a complete version history of your data. Redivis only stores the diffs between versions, allowing for efficient storage and display of clear deltas between versions
    • Version both data and variable metadata
    • Easily bring together and merge data from multiple source data files
    • Closes #150
  • Added ability to set the PI on a study
  • Added ability to filter by Public datasets

⚠️ Notable changes

  • Require user input to select permission group before creating a dataset (previously defaulted to "Public")

🐛 Bug Fixes

  • Allow filtering by global metadata on admin panel datasets tab
  • Fixed an issue where the requirements list for an organization member weren't sorted properly
  • Fixed an issue with form date fields with some international locales
  • Alphabetized some non-sorted lists in the admin panel (choose collection, permission group, etc.)
  • Fixed some broken links to help docs
  • Improved some problems where multiple searches on a page would interact with each other in surprising ways

UI

  • Improved projects display and search on the workspace
  • Improved datasets display and search on the workspace
  • Minor styling tweaks to the organization administrative panel
  • Use html links across the site to improve accessibility
  • Changed responsive behavior on small screens to favor scrolling over inconsistent layouts
  • Various pixels were pushed

🏠 Internal

  • Improved permissioning calculations in the project tool when determining the source dataset(s) for every table and transform
Assets 2

@imathews imathews released this Oct 3, 2018 · 1 commit to master since this release

This release represents a major improvement to data discovery and metadata management on Redivis. Our searching infrastructure has been completely overhauled, allowing users to easily navigate and compare myriad datasets.

Numerous pages and interfaces have been completely redesigned to streamline the data discovery & access flow and present the most relevant information cleanly and clearly

🚀 New Features

  • New dataset search!!! -> Try it live at https://redivis.com/StanfordPHS/datasets
    • Searches now operate across dataset names, documentation, tags, variables and variable metadata
  • Other searches have been overhauled across the board:
    • On the workspace, you can filter your projects and datasets by numerous relevant parameters
    • On the administrative panel, various organization resources are filterable and inter-linked
  • Substantially improved administrative tooling for bulk metadata management
    • Linked documentations allow for the application of documentation information across multiple datasets
    • Linked metadata allows for applying variable labels, descriptions, and value labels across datasets
  • Improved markdown editor, allowing admins to embed images and other rich content in various documentations and the organization "about" section

🔄 Notable changes

  • "Dataset groups" have been renamed to "Collections" to better reflect their purpose of organizing semantically related datasets
  • Datasets can now be part of multiple Collections, not just one

UI

  • Redesigned organization home page
  • Redesigned dataset page
  • Redesigned access flow for getting approval to work with datasets
  • Redesigned projects list on workspace
  • Redesigned saved tables list on workspace
  • Redesigned datasets list on workspace
  • Redesigned "Add dataset(s) to project" modal

🏠 Internal

  • Added a bunch of indexes to make all of these advanced searches even snappier than they were before
  • Overhauled the styling infrastructure to help make the UI pixel-perfect
  • Rewrote and substantially improved our access and permissioning infrastructure, allowing for complex searches that respect the access level of the user making the query
Assets 2

@imathews imathews released this Sep 14, 2018 · 1 commit to master since this release

This release implements some highly requested features around table exports and raw SQL queries. We have some further exciting updates planned for the weeks to come, stay tuned...

🚀 New Features

  • New table export experience! Export those data into your favorite format(s), download their metadata, download their SQL, and generally, just have fun.
  • You can now save any table in your project as a snapshot for future reference / export
  • Any saved tables that exceed export limits will be marked as pending, and administrators will have the ability to approve or reject these saved tables for export on a case-by-case basis
  • A full-featured, raw SQL interface in the project tool! For those of you who couldn't wait to get your hands dirty writing SQL... probably time to stock up on soap 🤓
    • Convert any transform from GUI mode to SQL mode
    • Interchange SQL and GUI transforms throughout your project
    • Live project updates as you join in other tables through SQL
    • Reference any lists throughout your project via SQL parameter expressions
    • Read the docs ->
    • NB: This is a Beta feature. Please notify us of any issues at this repo or via support@redivis.com

🐛 Bug Fixes

  • We think we've gotten to the bottom of a couple bugs that caused the project tool to stop updating if your internet got disconnected. Probably a few more issues to sort out here, but remember, if anything seems funky, just reload the page and you should be back up and running.
  • Resolved lack of UI feedback when a table export is being generated

🏠 Internal

  • Some internal data restructuring that should improve error messages and stability of the project tool
Assets 2

@imathews imathews released this Jul 20, 2018 · 11 commits to master since this release

This release contains some exciting new functionality around multi-user collaboration in Redivis. Researchers can now collaborate on a project in real time, viewing each others' work and communicating through a node-based commenting system

screen shot 2018-07-20 at 12 43 44 pm

🚀 New Features

  • Work with your peers and collaborate in real time!
    • You can now comment on any node in projects that you have "write" access to. Ask questions, mark to-do's, and even send to outside experts for review / clarification
    • View each other's location on the minimap, seeing where they're currently working
    • See all actions and updates triggered by others in real time
  • Many new transform methods
    • Implemented several additional mathematical, temporal, and string-based methods.
    • Implemented over a dozen new partition methods, focused on statistical aggregation and ordered ranking within a partition

🐛 Bug Fixes

  • Several additional querying bugs have been ironed out — many thanks to those who reported issues that they encountered!

🏠 Internal

  • Performance improvements when working with transforms that reference many variables
Assets 2

@imathews imathews released this Jul 3, 2018 · 11 commits to master since this release

This release represents a major update to how researchers query data on Redivis. The transform interface has been rewritten from the ground up, and now implements some of our most requested features, while substantially improving stability and general usability.

Additionally, we have overhauled access controls within projects, allowing for researchers to share both read and write access with their collaborators, as well as the larger community. Data access is now computed at each individual node within a project, meaning that a project can be opened even if the user doesn't have full access to all the data within it.

Please consult our help documentation at help.redivis.com for more information. Stay tuned for some exciting additional updates around project collaboration and a new table export interface in the coming weeks, and as always, keep the feedback, feature requests, and bug reports coming!

⚠️Notable changes

In spite of these substantial changes, no action is required on your part, and all existing projects will produce the same output as they did before this update; they have been automatically migrated as necessary

  • Joins have been removed as their own node. Instead, join and transform functionality has been merged within one node
  • The variable types decimal and text have been renamed to float and string to better reflect the underlying storage mechanism and reduce confusion
  • The output type on CASE statements is now implicitly computed. Case statements that previously overrode the default type output have been migrated by wrapping them within a cast statement to preserve exactly the behavior in existing projects.

🚀 New Features

  • Multiple joins are now supported within one transform — bring together several tables as you query them!
  • More complex join clauses, with nested logic and support for dynamic expression, are now supported
  • Variable types can be changed inline
    • No more creating a new variable just to recast an old one
  • Improved inline edited state to showcase what you've edited since the node was last run
  • Implemented the ability to revert an edited node to its state in the last successful run
  • Support for multiple left-hand variables (e.g., Diag*)
  • Improved code view with better syntax highlighting
  • Ability to export the code generated within a transform
  • Support for deeper nesting in logical queries with parenthetical statements
  • Implemented partitioned queries, allowing for you to generate new aggregate variables without dropping any columns or rows
  • Adds support for the LIMIT clause
  • Adds support for ordering the output table
  • Substantially improved invalid state / error messaging
  • Support for LIKE / NOT LIKE operators
  • Improved inline documentation of new variable methods
  • The frequency distribution in the stats modal can now be searched by value labels (when present)
  • Write access to projects can be shared! Work with your collaborators on the same project, all in real time.
  • Read-only access to projects can be shared via shareable link, or by making the project public to the world (viewers will still need appropriate access to the underlying data to view the project's content)
  • Access to the data within a project is now computed at the node level, meaning that you can open a project even if you don't have access to all of its data
    • In fact, you may have access to some of the data — allowing you to work with and view a sub-branch, even if you can't see everything

🐛 Bug Fixes

  • The underlying architecture of the transform has been completely rewritten, and is fundamentally far more stable and performant. Numerous bugs and odd behaviors have been eliminated
  • In the case of a bug, the page no longer fully crashes, but instead provides error messaging and a clear path towards reloading the page

UI

  • Did we mention the redesigned transform? Take it for a spin, let us know what you think!
  • Various pixel pushing and polish

🏠 Internal

  • Substantial improvements have been made to the performance in editing a transform. It should all feel much snappier.
Assets 2

@imathews imathews released this Jun 20, 2018 · 11 commits to master since this release

This is a smaller release that focuses on the user login experiences, as well as a redesigned homepage

🚀 New Features

  • A new, slick, simple homepage to help users learn more about Redivis and navigate to their organization(s) of interest
  • Users can now log in with a google account or an institutional email account. Currently, only @stanford.edu accounts are supported
  • New login flow gives users control before a new account is created, allowing them to choose a username, profile image, etc.

🐛 Bug Fixes

  • Fixed an issue that could cause variable indexes in sample tables to become mismatched
  • If a user tries to login with a non-existing account, they'll be prompted to create an account, rather than receiving an error

UI

🙄 Known issues

  • Currently only one email login is supported, though in the coming days we will be releasing a patch to support multiple logins to the same account
Assets 2

@imathews imathews released this Jun 13, 2018 · 11 commits to master since this release

A quick release to support enhanced metadata extraction from common file formats

🚀 New Features

  • Support for full metadata extraction (variable labels and value labels) from Stata (.dta) and SPSS (.sav) files

🐛 Bug Fixes

  • Fixed a few issues that could occur when restricting dataset location to the EU region

UI

  • Improved the default sorting behavior on various tables

🙄 Known issues

  • SAS metadata uploads are not yet uploaded — SAS stores its metadata in a separate file (.sas7bcat), presenting some unique challenges. However, SAS data uploads (sas7bdat) are fully supported
Assets 2
You can’t perform that action at this time.