New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The storage API is difficult to use in Node.js #500

Open
jcnelson opened this Issue Jul 24, 2018 · 10 comments

Comments

7 participants
@jcnelson
Copy link
Member

jcnelson commented Jul 24, 2018

The storage API (getFile, putFile) depends on the following browser-isms to work correctly:

  • location.search must exist as a global
  • window must exist as a global
  • window.location.origin must be set
  • window.location.search must be set to ?authResponse=...
    • The developer must create a fake but well-formed authResponse JWT for the above
  • window.localStorage must exist, and must implement the semantics of the LocalStorage API

This makes using Gaia in Node.js programs somewhat painful and unintuitive. I have had to add the following to my top-level index.js get it to work at all (and even then, I'm not 100% sure this is sufficient):

function makeFakeGaiaSessionToken() {
  const appPrivateKey = '583d0cf5572735a95bf2d6de10952131fedbe9c72b272d5081e11e7f77e576c6'
  const ownerPrivateKey = '24004db06ef6d26cdd2b0fa30b332a1b10fa0ba2b07e63505ffc2a9ed7df22b4'
  const transitPrivateKey = 'f33fb466154023aba2003c17158985aa6603db68db0f1afc0fcf1d641ea6c2cb'
  const transitPublicKey = '0496345da77fb5e06757b9c4fd656bf830a3b293f245a6cc2f11f8334ebb690f1' + '9582124f4b07172eb61187afba4514828f866a8a223e0d5c539b2e38a59ab8bb3'

  window.localStorage.setItem('blockstack-transit-private-key', transitPrivateKey)

  const authResponse = blockstack.makeAuthResponse(
    ownerPrivateKey,
    {type: '@Person', accounts: []},
    null,
    {},
    null,
    appPrivateKey,
    undefined,
    transitPublicKey,
    null);

  return authResponse;
}

const localStorageRAM = {};

global['window'] = {
  location: {
    origin: 'localhost',
    search: `?authResponse=${makeFakeGaiaSessionToken()}`
  },
  localStorage: {
    getItem: function(itemName) {
      return localStorageRAM[itemName];
    },
    setItem: function(itemName, itemValue) {
      localStorageRAM[itemName] = itemValue;
    },
    removeItem: function(itemName) {
      delete localStorageRAM[itemName];
    }
  }
};
global['localStorage'] = global['window'].localStorage

Moreover, this only works with the upcoming blockstack.js 18.x.x (I haven't gotten it to work with 17.x.x).

Any thoughts or recommendations on what we can do here? @larrysalibra @kantai @yknl

@jcnelson jcnelson self-assigned this Jul 24, 2018

@jcnelson

This comment has been minimized.

Copy link
Member

jcnelson commented Jul 24, 2018

Also, to use getFile(), I first have to call handlePendingSignIn() and call getFile() in the then() branch. The reason for all of this is that handlePendingSignIn() both depends on the availability of window.localStorage and location.search, which it then uses to set the local Gaia hub connection (via getOrSetLocalGaiaHubConnection()).

@kantai

This comment has been minimized.

Copy link
Member

kantai commented Jul 24, 2018

Multi-player reads are fairly straight-forward. There is one offending line at:

https://github.com/blockstack/blockstack.js/blob/develop/src/storage/index.js#L315

The following code works:

window = { location: { origin: false } }
bsk.getFile('statuses.json', { 
    decrypt: false, verify: false, username: 'collectivism.id',
     app: 'https://app.travelstack.club' }).then(console.log)

If we fix that line, then multi-player reads will work (buffer support still needs the cross-fetch update from #492) without setting the fake window global.

putFile and single-player getFile --- I'm not sure if they make sense outside of the context of the browser, which is where blockstack.js mostly assumes client sessions should live.

This seems like a great argument in favor of moving a lot of the gaia functionality to a gaia.js library, which blockstack.js would just include and call --- blockstack.js is mostly a webapp SDK, whereas gaia.js would be callable from node.

@jcnelson

This comment has been minimized.

Copy link
Member

jcnelson commented Jul 24, 2018

The following code works:

Ah, I didn't realize I needed to explicitly set verify: false and decrypt: false. Makes sense given that there's no app key set in localstorage.

putFile and single-player getFile --- I'm not sure if they make sense outside of the context of the browser, which is where blockstack.js mostly assumes client sessions should live.

Node.js apps that need to either upload data or download and decrypt/verify data still seem to need my above hacks to work (the CLI needs them, for example).

This seems like a great argument in favor of moving a lot of the gaia functionality to a gaia.js library, which blockstack.js would just include and call --- blockstack.js is mostly a webapp SDK, whereas gaia.js would be callable from node.

Definitely!

@bodymindarts

This comment has been minimized.

Copy link
Contributor

bodymindarts commented Jul 24, 2018

Since making blockstack.js work in node.js requires decoupling from browser primitives, it would be great if the web-worker context would also be taken into account. I've used blockstack.js in web-workers, but with similar hacks on the top level context as are required for node.js.

@x5engine

This comment has been minimized.

Copy link

x5engine commented Jul 24, 2018

@jcnelson @bodymindarts why not add a test in blockstackjs that checks if window and other web browser globals are defined otherwise it won't throw an error and assume it is a server-side thing?

Example

 const defaults = {
    decrypt: true,
    verify: false,
    username: null,
    app: window ? window.location.origin : "",//or whatever it should be
    zoneFileLookupURL: null
  }

Instead of hacking around it ?

@larrysalibra

This comment has been minimized.

Copy link
Member

larrysalibra commented Jul 25, 2018

I'm also seeing a similar problem with our encrypt/decryptContent methods. Will put this on the list to address for our next release. Specifically, if you don't specify a private key it gives you an error about window not existing instead of something useful.

@sirpy

This comment has been minimized.

Copy link

sirpy commented Aug 13, 2018

FYI loadUserData is also used in storage and uses window

@AC-FTW

This comment has been minimized.

Copy link

AC-FTW commented Aug 18, 2018

@larrysalibra, Stealthy @prabhaav is working on a project that could really benefit from this functionality. @jcnelson mentioned you were looking into this.

We're specifically interested in doing the following all within node:

  • Blockstack Auth
  • GAIA read and write
  • GAIA multiplayer read (kinda easy as @kantai mentioned--we're already spoofing this in iOS)

We're shooting for the end of this month and are happy to guinea pig your changes if possible.

@larrysalibra

This comment has been minimized.

Copy link
Member

larrysalibra commented Aug 20, 2018

We're shooting for the end of this month and are happy to guinea pig your changes if possible.

@AC-FTW That would be great! I'll hopefully have updates for you soon!

@jcnelson

This comment has been minimized.

Copy link
Member

jcnelson commented Sep 5, 2018

Just adding to this list of known difficulties (happy to move to another issue if you think it's too different): there is a growing use-case for allowing users to generate and pass in authentication tokens and addresses to the putFile() and getFile() calls. For example, if user A is storing data to user B's Gaia hub, then user A needs either an authorization token signed by user B's private key, or an authorization token with an embedded association token signed by user B. User A would need to obtain this information out-of-band (i.e. via the app), and then supply it in these calls. Was thinking this could be done by expanding the options argument to include a Gaia hub config, and was thinking we could add/expose the methods for generating authorization tokens (and later, association tokens).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment