Create a more prototyping-friendly development experience

[jmuchovej]

👋 I've been rather frustrated with the import/delete cycle I've needed to go through while prototyping studies – especially when this loop is caused by needing to check that JATOS functionality is working as intended.

I understand why this might not be a priority to develop, but I'm willing to contribute to this, if needed.

Essentially, as I'm developing experiments with @bjoluc/jspsych-builder, to test out JATOS-specific functionality, like data storage/upload, I usually find myself in a cycle of build -> import -> test -> delete if buggy.

It would be nice to circumvent this most of the process during development, but I can understand why that might be undesirable from a reproducibility standpoint.

I can imagine a few solutions to this, but I'm not sure which makes the most sense both in terms of JATOS' goal(s) and development time/effort. I'm also not sure of the design schema of JATOS at the moment, which I imagine could constrain the ways towards this.

1. On a live JATOS server a study has the ability to be overwritten by upload (or treated as vN.M.
2. A local JATOS server (read as: locally hosted and distinct from a production) acts as both an API endpoint and a database – emulating the functionality of an ongoing study without more production-oriented features. (I'm new to JATOS, so I haven't fully grokked what knobs are more production oriented.)

[kristian-lange-tw]

Sorry for the late reply. I had to think a bit about it. I understand that in the way you are using JATOS it appears as not user-friendly. The reason is JATOS wasn't build to be used in the way you described: builder generates a jzip -> import -> test -> repeat until done. The development cycle we had in mind was more like: create a new study or import one from the example studies -> edit files in the study assets folder directly -> test -> repeat until done. There is no import necessary (except the initial if one starts with an existing study jzip).

But I see the point of using a builder like jspsych-builder (or lab.js or OSWeb). They make it much easier to create the actual experiment - something that JATOS does not help you with. Now I'm thinking of a way to make it easier for those builders to interface with JATOS, so one does not have to import the study repeatedly. So far I came up with two ideas that I want to sketch here:

1. Improve JATOS' API
   JATOS has already an API but it's only used internally so far. Mostly because it needs a session cookie that is generated after one logs in. So first one has to login (or simulate a login with the proper requests), grab the session cookie and then one can use all endpoints that JATOS' GUI is already using just by putting the session cookie in a header with every request. An overview over the endpoints (without any documentation) can be found here. This grabbing of a session cookie is cumbersome and opposes the purpose of a builder to make things easier for the enduser. Tokens can help here, similar to what GitHub is doing with those personal access tokens. Every user of JATOS could generate access tokens that can be used instead of the username/password combination. The token then must be somehow handed to the builder (e.g. in some config) and from then on the builder can interface directly with JATOS and e.g. upload jzips directly or create/delete studies. Authentication happens with each request passing on the token in some HTTP header. No session cookie and no prior login would be necessary.

2. Use Git, GitHub or GitLab
   This is actually something I'm thinking about for some time now. Git is already used by some JATOS users/admins to add history and versioning to their study asset files (basically put the study asset folder or the whole study asset root folder under git). Since it's just a folder in the file system JATOS doesn't care about the additional git. But I thought about to integrate Git and GitHub (GitLab or Bitbucket is probably possible too, but I'm not a frequent user of them) into JATOS. A study assets folder could be put under git control and pushed to a GitHub repository. This can be done by a JATOS or by a builder. From there other JATOS (even remote server installations) can pull it. Of course the builder and each JATOS need to be configured with the repo URL and a GitHub access token. If you want to use a builder the work flow would be: use the builder to create a study -> builder pushes to GitHub -> JATOS pulls from GitHub -> testing -> repeat if a problem occurs. The Git/GitHub approach has some advantages: git is well-defined, it can be used everywhere and it adds versioning to any study project. But if we don't want to use any proprietary GitHub API we have to stick with Git alone and this might be difficult to implement in JATOS and the builders (actually I'm not sure - never did it).

So far my thoughts. This builder-JATOS interfacing has to be well-thought through. Do you think they could work out? Which one would you prefer? Maybe there is another variant I haven't thought about? Thinking of the users: are they easy enough to use? How was lunch today? I'd like to start a discussion here.

[jmuchovej]

Disclaimer: I've never used lab.js, nor OSWeb, so I can't speak to their development cycle/process.

Re: point 1

Tokens can help here, similar to what GitHub is doing with those personal access tokens. Every user of JATOS could generate access tokens that can be used instead of the username/password combination.

Something like this seems ideal, at first glance.

... e.g. upload jzips directly or create/delete studies.

jspsych-builder, during development, runs similarly to most Static Site Generators – watching for changes in particular files and recompiling the site as needed (though usually only recompiling only what's necessary). I'm not sure that full-on building/exporting is necessary, per se, to test JATOS interaction.

e.g., in my typical workflow, I don't actually care that the data is saved, just that I can interact with JATOS in all the expected ways. (e.g., I can retrieve *JsonInputs and the like.)

i.e., I don't need the assets folder/whatever else because I have a local development server running on my computer.

The token then must be somehow handed to the builder (e.g. in some config) and from then on the builder can interface directly with JATOS ...

Given it's a PAT, I think the default mode should be either an environment variable or storage in something like ~/.netrc which is unlikely to be tracked in git repository, thus sidestepping credential leaks.

---

Re: point 2

So, re:VCS – I like the idea of auto-deploys from such a system – but what you described sounds like the same process I've been frustrated with in a different form. 😅

---

I'm generally agreed on the idea that interfacing should be well thought out.

I'm not sure about lab.js or OSWeb, but I do know that jspsych-builder, functionally, results in a single JATOS component.

I think an ideal (though potentially [unnecessarily] complex) route would something that allows for an API much like JSPsych does in ES modules (e.g., consider the following).

// This could short-circuit and essentially search for an existing global
//   jatos variable (or something like `process.env.MODE == "production"`
const jatosJS = initJATOS({
  token: `<your-PAT>`,  // honestly this should be grabbed from `process.env.JATOS_TOKEN` or similar
  url: `<jatos-installation-url>`,
  ...options
})

I believe an interface like this would actually allow JATOS to be used by jspsych-builder, as well as other static-site generators (like next.js, nuxt.js, etc.). I'm not sure about lab.js nor OSWeb.

---

Maybe this begs another question of – are the JATOS API and GUI fully disentangled? If so, is the JATOS API something that is/could be fully accessible through JavaScript? (It seems that your preferred language is Scala, so I'm not sure where Scala begins/ends for JATOS. 😅)

If the API and GUI are fully disentangled, then I'm pretty sure that I'd be comfortable defining studies in JavaScript. Given early days and the most alpha-like state this would be in, I think treating JATOS in a development and production environment as drastically different is probably okay?

Sorry for the longwinded thoughts. I'm not exactly sure how to approach this, but I guess, succinctly, I'd like to be able to either define a study in JATOS that is a test-study that allows me to interact with JATOS as I would in production (though advancing to other components is irrelevant to me, I think), or fully-defining a study in JavaScript. In either case, having something like a Docker container that could be spun-up to act as the appropriate endpoint seems like a good solution – since I think the folks opting for local development using tools like npm, webpack, and the like aren't going to shy away from a more advanced development process.

(I definitely understand that your ideal modus operandi of JATOS being "anyone could do this", but I'm not sure if "anyone" would actually need this development capacity – since this seems to be something that would mostly "plague" folks developing experiments outside of the intended JATOS way.)

[bjoluc]

Re 1: That's a great idea! jsPsych-builder-wise, this would enable something like a deploy command (or even a deploy flag to the build command). All a builder would need then are the public JATOS URL and a token 🎉

@jmuchovej I think that's all I'd like to support in jspsych-builder then: It's helpful for development and production. For most users, I guess, development work concentrates on the actual jsPsych experiment, not so much the JATOS integration. Hence, I woudn't like to bake a special JATOS mode into the run command. If you're testing your JATOS integration, just use nodemon to build and push to JATOS on source changes – a few seconds slower than some sophisticated run mode, but equivalent to your manual "delete, build, import" cycle and fully automated!

Re 2:

A study assets folder could be put under git control and pushed to a GitHub repository. This can be done by a JATOS or by a builder.

Right, although I personally wouldn't like to check my (possibly non-final) build outputs into source control just to get them into JATOS. But for folks versioning their JATOS study assets with git, this would be perfect!

I love the idea of improving/documenting the API and adding auth token support. This could be used to approach your second point too by providing some script to update remote studies according to local files (reading the JATOS API URL and auth token from env variabels). Without tightly coupling JATOS with the platforms themselves then, you could provide examples of invoking the script "on push" for all kinds of CI providers (GH Actions, GitLab CI, Bitbucket Pipes, ...), or even publish a "deploy-to-jatos" GitHub action that brings the script and runs it. One solution to rule them all and make builders happy 🥳

Thinking of the users: are they easy enough to use?

The API itself certainly not. But the "Use Git, GitHub or GitLab" approach – which could boil down to throwing a CI config and two secrets at GitHub/GitLab/Bitbucket – for sure!

How was lunch today?

Great, thanks! 😁

Some promising resources regarding API docs:
https://github.com/cloud-annotations/docusaurus-openapi
https://swagger.io/blog/api-development/playing-with-swagger-using-swagger-and-swagger-ui/
https://swagger.io/docs/specification/about/

[kristian-lange-tw]

Sounds like improving the JATOS API and adding auth tokens is the way to go. I like this approach 😄. And I agree with @bjoluc, a tight coupling with GitHub is probably not necessary - one can do this with a script or GitHub Actions or something similar.

And thanks @bjoluc for the link to docusaurus-openapi, I didn't know about this one yet. We use Docusaurus already for the JATOS docs, so this should be the obvious tool to use.

And @jmuchovej about your question: "are the JATOS API and GUI fully disentangled?". It is an API with many HTTP endpoints (inspired by REST but not fully REST). So, yes it's fully disentangled, in the meaning that you can use it from wherever you want, JS running in browser or server-side, Python, Java (any other language) or tools like Postman. Only the backend is written in Java and Scala.

So my plan is to start working on the JATOS API improvements after the 3.7.4 release (which should come out in a week or two). But I can't yet say how long this will take. The token stuff is new to me and since this is about authentication we have to do this thoroughly. And then API documentation. And I even think about if we have to change / add some endpoints to be usable outside the GUI. I guess the time frame for this whole improvement is 6 to 12 weeks. Sorry, can't be more precise. But it would be nice if you could be testers and give feedback as soon as there is something like an API with auth tokens. JATOS has a Slack and this would be a better place for discussions. What do you think?

[bjoluc]

@kristian-lange-tw Sounds great! Happy to test things out and draft an integration in jspsych-builder. I didn't find a public Slack invite link, so feel free to invite me via mail@bjoluc.de 🙂 Thanks for working on this!

[jonathon-love]

Now I'm thinking of a way to make it easier for those builders to interface with JATOS, so one does not have to import the study repeatedly.

oh, i just found this thread, and i'd like to propose an alternative. here's my proof of concept:

#258 (comment)

jonathon

[jdekarske]

If it's any help, here's my pipeline which is pretty painless:

JATOS in a docker container

  jatos:
    image: jatos/jatos:3.7.4
    user: daemon
    restart: unless-stopped
    volumes:
      - $PWD/../jatos/database:/opt/docker/database:rw
      - $PWD/../jatos/logs:/opt/docker/logs:rw
      - $PWD/../jatos/study_assets_root:/opt/docker/study_assets_root:rw

deploy script (in package.json)

scp -r ./dist/* $USER@$REMOTE:$VOLUMEPATH/study_assets_root/$STUDYNAME

After making changes, I simply run npm run deploy and test on JATOS.
The only gotcha is that you need to create the study on JATOS first.