New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Permissions flow between environments #672

Closed
storm1er opened this Issue Feb 19, 2018 · 18 comments

Comments

Projects
None yet
4 participants
@storm1er

storm1er commented Feb 19, 2018

Node.js version:
v8.9.4
npm version:
5.6.0
Strapi version:
3.0.0-alpha.10.2
Operating system:
Ubuntu 17.10

Do you want to request a feature or report a bug?
Question

What is the current behavior?
Before alpha.10.1, I was able to easily upgrade from one env to another, especially when editing user-permission. But as you said, last upgrade moved all the config from file to database.
This ended up to create "one-shot-update" scripts, executed when deploying.
The point of 10- versions was that strapi handle this itself through files.

What is the expected behavior?
Maybe find some hybrid solution ?

  • Option 1 : Keep configuration files, but can be overrided through ui
    • Should work in most case, but how do we handle file override an already overrided configuration ?
  • Option 2 : Block user-permission edition to env, based on config
    • So now, the configuration will be project based :
      • you choose to edit user-permission only from dev : file based
      • you choose to edit user-permission wherever you want : database config
    • It's probably the best answer but dev consuming (on strapi side)
  • Option 3 : give us the possibility to have some "script on deploy" stuff that strapi could handle (using this ?)
@Aurelsicoko

This comment has been minimized.

Contributor

Aurelsicoko commented Feb 21, 2018

There are three possible solutions:

1 - Manually migrate the data from the table users-permissions_permissions development to production as you will have to do using others frameworks.
2 - Add a way to export permissions and import them in the production environment. But it's not optimal because you have to start the server (at this time, the permissions will be set with default configuration), then import the permissions. So you will have a lapse of time (few seconds) without the right permissions.
3 - The latest options (3) seems the best to me. It makes sense to me.

@storm1er

This comment has been minimized.

storm1er commented Feb 22, 2018

@Aurelsicoko I agree with the latest option

@storm1er

This comment has been minimized.

storm1er commented Feb 24, 2018

@lauriejim Can you transform this into "Feature request" ? ^^ And it's kinda important as arround me i'm not the only guy who is loosing their mind : "How to do this without destroy everything" x)

Most of us directly edit permission through mlab ^^'

@Aurelsicoko

This comment has been minimized.

Contributor

Aurelsicoko commented Mar 19, 2018

I would love to see someone try to make some work on this part. We need to improve it but we are too busy to start this work right now.

@storm1er

This comment has been minimized.

storm1er commented Mar 20, 2018

Well ... to be fair, I think that moving from file to db was a bad move ...
I mean, moving ENTIRELY from file to db ... We should keep file, and on execution of npm start (eq. on each restart) the server reload what was saved from file.

Drupal is doing this and it's quite intuitive ... I'll look for more details if I have time.

@Aurelsicoko

This comment has been minimized.

Contributor

Aurelsicoko commented Mar 21, 2018

@storm1er However, if you update the permissions, you need to restart the server if they are stored in a file. That's the main reason it was better for us to move the permissions in the database.

@storm1er

This comment has been minimized.

storm1er commented Mar 23, 2018

TL;DR:

"Import/Export permissions" button, and an optional "Import permission on server restart" checkbox would do the job :)


@Aurelsicoko said

However, if you update the permissions, you need to restart the server if they are stored in a file. That's the main reason it was better for us to move the permissions in the database.

No you don't need to : Settings are stored in database, and when needed, you "click upload permission" to recreate / regenerate permissions files.

In our case, when restarting the server, it wouldn't care about what was in database, it recreate permission through files (to build the database settings).

Drupal 8 example

More details on this workflow example :
Drupal 8 - Configuration management
Drupal 8 - Managing your sites configuration

By default, the "active" configuration is stored in the database ("config" table). This is for performance and security reasons. This is the complete configuration for the entire site at that moment. Configuration can be exported and imported as YAML files, either in its entirety, or a single piece of configuration, using Drush and/or Drupal Console config commands or the Configuration Manager. (See below for more details.)

  • You can develop every detail of you website on your develop environment
    • So you can sync every environment easily !
  • You can deploy permission fix instead of manipulating strapi B.O. for every modifications
    • Think about business pipeline : dev/staging/qa/client/prod/... how many time will you lose ?
    • And edit database directly is always a bad idea, having something natively is always a better solution because you know (as a framework) what happens.
  • You can still "quickfix" in prod when you missed something (through B.O.)
    • But yes, you have to recreate/regenerate your permission's file to keep them upon next deliveries
    • But no, you don't need to restart you server to apply what you edit through B.O.

Why I think it was a bad move

What was great in strapi before that was the fact that EVERYTHING was versioned, whatever you were doing, just looking at the code was self-explanatory... Now we must have some deploy scripts to be sure that permissions are correctly set on every environment of our pipeline.

I supposed that not everyone will agree with this, but working with mates on the same project where you can't git permission (specially on API framework) feels really weird.

Sorry about writing mistakes, I don't often write so much english.

@Aurelsicoko

This comment has been minimized.

Contributor

Aurelsicoko commented Mar 26, 2018

Thank you so much for your research on this topic. Let me resume with my words the potential new workflow:

  • Store the "active" permissions configurations in the database (as we already do).
  • Be able to import/export the permissions to a JSON file from the BO.
  • This exported file could be placed in ./plugins/users-permissions/config/permissions.json and manually edited. The framework will use that file to update the "active" permissions in the database during the server start or where the administrator clicks on the button "Recreate permissions" in the BO.

It makes sense to me. However, the risk is very high to forgot to export the permissions when you will do a "quick-fix" in production... We should display a warning in the BO to inform the developers.

Edited: We could also rethink the permissions.json file to split it into multiple files. It was too complicated to edit the file manually.

@storm1er

This comment has been minimized.

storm1er commented Mar 26, 2018

@Aurelsicoko That's the idea !

About "However, the risk is very high to forgot to export the permissions"
You're right, but except in little project, most of the time it's highly unrecommended/prohibited to edit permission in prod, because we all know what happens when an intern have to edit something in prod 😇

Also, a BIG 👍 about display a warning in the BO telling " ⚠️ permissions has been edited manually blablabla" & about splitting permissions.json 😺

@lauriejim lauriejim changed the title from How to export database based config from dev to staging to prod ? to Export database based config from dev to staging to prod ? Apr 8, 2018

@lauriejim lauriejim changed the title from Export database based config from dev to staging to prod ? to Permissions flow between environments Apr 9, 2018

@Aurelsicoko

This comment has been minimized.

Contributor

Aurelsicoko commented Apr 16, 2018

@storm1er I'm actively working on this topic. After hours of reflexion and studies of how the others projects like Drupal are handling this, here my thoughts:

  • We should be able to export the configurations using the UI and the CLI.
  • A plugin should be able to handle its own export/import strategy (ex: Users & Permissions plugin needs to handle the permissions migration by itself).
  • We should be able to rollback an import.

Proposal

CLI

  • strapi config:export to export every configurations (core & plugins).
  • strapi config:import to import every configurations (core & plugins).
  • strapi config:export --core --users-permissions to export specific configurations (core & users).
  • strapi config:import --core to import specific configurations (core only in this example)
  • strapi config:rollback to cancel the latest import and execute the previous one if available.

File structure proposal

I think it's the most difficult part. We need to keep it simple but also make it flexible.

- /config
  - /sync
    - /import_1523893204
      - core-store.json
      - /users-permissions
        - application.json
        - content-manager.json
        - content-type-builder.json
        - settings-manager.json
        - upload.json
        - users-permissions.json

I suggest we should add a sync folder where we will put all the data to import. Every import will be named like this import_<timestamp>. In this folder, the core-store.json contains the data which are stored in the core_store collection in MongoDB. The users-permissions folder contains the permissions split by category.

Why I have named it import_<timestamp> instead of config_<timestamp>?
I think it could also be the perfect place to put our files to seed the database with test data. So, import is more generic and fits better to this future usage.

The format of the core-store.json file could be like that to easily compare value between environment.

{
  "plugin_users-permissions_grant": {
    "tag": "",
    "value":  {}
  },
  "plugin_users-permissions_email": {
    "tag": "",
    "value": {}
  },
  "plugin_users-permissions_advanced": {
    "tag": "",
    "value":  {}
  },
  "plugin_upload_provider": {
    "tag": "",
    "environments": {
      "development": {
        "value":  {}
      },
      "production": {
        "value":  {}
      }
    }
  }
}

The format of the files in the users-permissions folder could be like that:

{ 
  "post.create": {
    "roles": {
      "admin": {
        "enabled": false,
        "policy": ""
      },
      "public": {
        "enabled": false,
        "policy": ""
      }
    }
  },
  "post.destroy": {
    "roles": {
      "admin": {
        "enabled": false,
        "policy": ""
      },
      "public": {
        "enabled": false,
        "policy": ""
      }
    }
  },
  "post.find": {
    "roles": {
      "admin": {
        "enabled": false,
        "policy": ""
      },
      "public": {
        "enabled": false,
        "policy": ""
      }
    }
  },
  "post.findone": {
    "roles": {
      "admin": {
        "enabled": false,
        "policy": ""
      },
      "public": {
        "enabled": false,
        "policy": ""
      }
    }
  },
  "post.update": {
    "roles": {
      "admin": {
        "enabled": false,
        "policy": ""
      },
      "public": {
        "enabled": false,
        "policy": ""
      }
    }
  }
}

In both file structure proposal, I've always focused my mind to make them easily readable for a human. These files could be manually edited so we have to make them understandable and readable.

How a plugin will handle its own migration strategy?

I suggest that we should introduce two new files export.js and import.js in the ./plugins/**/config/functions folder. The framework will detect the plugin which have a migration strategy. It will make them eligible to be displayed in the UI through the Configurations link.

What do you think about this proposal? I'll try to post some screenshots of the UI in the coming days to show a little bit more how the UI will look like...

@storm1er

This comment has been minimized.

storm1er commented Apr 18, 2018

Hi,

💯 ! Great work, my mind was near that idea but I couldn't find a way to make it "universal" for every strapi's components.

The only comment I have is: files are human-readable, which is perfect but,

  • Why not use YAML files instead ? (not really important, but if the goal is to be human readable, it's an idea)
  • Why using unix timestamp instead of date ? (kind of important, especially for rollback feature)
@Aurelsicoko

This comment has been minimized.

Contributor

Aurelsicoko commented Apr 18, 2018

We don't have any YAML file in the project. We are always using JSON files, it's easier to manipulate in JavaScript even if there are YAML to JSON converters.

Great idea to use a more readable date format 👍


As promised, a preview screenshot of the UI. We are also thinking about renaming the Settings Manager plugin to Configurations and make it accessible through the Configurations link in the General section in the left menu. It would make more sense, and we could integrate the Migration part into it.

screen shot 2018-04-18 at 11 16 17

@lauriejim lauriejim added this to the 3.0.0-alpha.12.3 milestone Apr 19, 2018

@lauriejim lauriejim removed this from the 3.0.0-alpha.12.3 milestone May 1, 2018

@storm1er

This comment has been minimized.

storm1er commented May 3, 2018

@lauriejim I see this has been deleted from 3.0.0-alpha.12.3 milestone, can we have some follow up on this ? Thanks :)

@Aurelsicoko

This comment has been minimized.

Contributor

Aurelsicoko commented May 7, 2018

@storm1er We're still focusing our efforts on the stability. Some of our resources aren't available as expected so we made the choice to delay the development of this feature.

@storm1er

This comment has been minimized.

storm1er commented Jul 10, 2018

@Aurelsicoko Any news on the milestone targetted for this feature ? =)
Great jobs for the lasts release !

@Aurelsicoko

This comment has been minimized.

Contributor

Aurelsicoko commented Jul 20, 2018

Not at all... However, we merged a PR (#1514) which will certainly improve the transition between environment.

@derrickmehaffy

This comment has been minimized.

Contributor

derrickmehaffy commented Nov 9, 2018

@Aurelsicoko

This comment has been minimized.

Contributor

Aurelsicoko commented Nov 11, 2018

You're right! As this issue is very important, I want to remember that everyone can still comment on it or ping us if needed. We're closing the feature request issues to keep the repository as clean as possible, and we're moving all the features requests on Product Board.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment