-
-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal for allowing multiple schemas in a single instance #1106
Comments
I'm building a schema based multi-tenancy app and for the most part PostgREST ver 0.4.4.0 works for me. The ability to set My schema structure:
Tenant schemas are exact copies from base. DB functions are created only once in base and use search_path to access appropriate tenant schemas. Default DB I'm presently doing this by setting {
"message": "Could not find foreign keys between these entities, No relation found between mytable_a and mytable_b"
} If I set I'm happy to contribute in anyway. |
the hack mentioned by @rsubr somehow affects the |
Hi, I am using docker with external PostgreSQL and need multiple endpoints at the same machine (so same PostgreSQL and databases).... So, no matter if it is implemented as "one PostgREST instance" or "multiple PostgREST instances". The focus of a user like me is "multiple endpoints" and the best solution (faster and good reuse of resources). So, I am perhaps backing to #1090 (that was redirected to this issue) ... The scenario is:
So, to obtain SQL's table
What is the best solution to obtain this scenario? |
Multiple instances with docker would be better handled with For mapping the instances to different endpoints in the same domain we recommend Nginx, we're missing an example in the docs for this, but should be something like: location /endpoint1/ {
proxy_pass http://localhost:3001/; # Reverse proxy to a PostgREST instance
}
location /endpoint2/ {
proxy_pass http://localhost:3002/; # Reverse proxy to a PostgREST instance
}
... |
Support for dynamically selecting the schema at runtime would be fantastic. Would open up a whole new suite of applications that can be built with PostgREST. |
Hello everyone, my team requires this functionality as well. Initially we tried using the db-extra-search-path configuration parameter. The issue with that is that if 2 different schemata contain a table with the same name it would query the table in the schema that is mentioned first in the config. So I went ahead and looked into how to implement the possibility without breaking any existing functions(or so I hope). What I ended up doing is giving the possibility to specify a schema in the request URI given that the configured user has the proper permissions. Today I created a fork https://github.com/mahmoudkassem/postgrest/commit/6dd5bf1d80eb33adf7c71d161395da64a8463179, I'm aware that this just the first step, for instance the Swagger Doc generation needs to reflect that extension as well(so instead of using just one DbStructure Instance it would require a Map Schema DbStructure). Is there some other part of the application that this extension would need to affect? Initially I had about a week to enable the functionality(which was mostly used to get familiar with the code), but now things cooled down and I have the capacity to further work on this, but I require support on the conceptual level and I can't grasp all the ramifications this change would have for the existing functionalities. With your support we can hopefully have this feature properly implemented. |
@MahmoudKassem Great idea! I think your The main thing we need to ensure is that our sandbox(safe) approach is maintained toward the exposed schemata . For this we shouldn't think about The schema shouldn't be dynamic, because you risk exposing unsafe database objects, like the ones in If We also need to make sure is that there's no chance that something like For ensuring this, we have been very careful until now, we took a Fortunately, now the OpenAPI output takes a schema as a parameter. So, if we restrict your solution to schema based multitenancy it shouldn't be a problem if we don't detect relationships to other schema database objects. The findRelation(finds relationships between two tables) function also takes a schema parameter. For the above concerns, I think It'd be a matter of passing the schema from the url to those functions plus the whitelist, and of course adding a new test suite for making sure all of this works. DbStructure would be a different beast. I haven't yet fully grasped its whole complexity for doing mutli schema. But I think we can work on this together gradually one PR at a time, I'm also interested on adding this feature for this release. Let me know what you think and if you have any question! |
@steve-chavez all the above is good except the "one PR at a time", i don't see how you can do that, it's a single (big) feature that either works correctly in all cases (and does not compromise previous security) or it does not. If in the middle of the work (after a few PRs) some corner case is discovered that makes this unfeasible or too complicated, it would be hard to revers things. For specifying the "dynamic" schema, a header should be used. This will not break the current url format and not introduce ambiguity. Ppl that want it in a URL can use nginx to do the "translation" [edit] removed the part about fully qualifying which you already mentioned in the original post |
Thanks for your input/insight. https://github.com/mahmoudkassem/postgrest/commit/6415217b42f6af5d517d6aa880bff02c1716cc6f I've extended the db-schema parameter to be a list of schemas separated by comma. Whereas the first value of the list is the default. The passing of the schema is now via header instead of the url as suggested. The passed value has to be a member of the list otherwise the default schema is used. So now only explicitly added schemas to db-schema can be used. What is still to do is the check on the query level. Any input/suggestions whether this is going into the right direction are most welcome. |
Perhaps something as using a URI syntax that is not a PosgreSQL syntax, as the character |
@MahmoudKassem I think erring in case of a wrong schema instead of picking a default would be better. It seems easy to mistype a number in Regarding the header for the schema, I don't think there's a standard for this(unless we go with versioning on the media type My vote would be for versioning on the url. I think it wouldn't imply a breaking change. Unless in some case the user has named his schema Having the schema in the url makes the API easier to explore. |
As I comment, there is no mistype when using a schema separator mark like |
https://github.com/mahmoudkassem/postgrest/commit/750698bf05f07349b787c74ce32412a07502020f The easier discovery + that it doesn't break backwards compatibility(for the most part) convinced me, so I've added the dynamically specified schema back to the url and now instead of falling back to the default schema it will result in an 404 not found if the schema is not specified in db-schema. I find the /schema/table approach more intuitive, maybe that's the influence from the time I spend working with node.js. @ppKrauss could you please elaborate on how /schema/table causes Postgresql specific issues? I'm not too familiar with Postgresqls' syntax. I'm currently in the process of looking into how the request is transformed into a query to implement the schema check on the query level. |
Hi @MahmoudKassem , it is a syntax suggestion, an alternative to your It is valid as URI (no conflict) and valid to express with "no ambiguity risk", to confuse the PostgreSQL schema names and table names, or to confuse with other endpoint's REST syntax, like API version. Rules:
NOTE. You can suppose "rule 3. |
To solve ambiguity, how about if when
That would only be the case when @ppKrauss #1421 could be solved in another way(check my latest comment). Since these schemas are independent, I think the slash |
@steve-chavez I like this /schema-name/rpc/......... |
Hi, this issue is confuse for me, I not see utility on versioning for my applications, but I need access to multiple schemas at PostgREST endpoint... Anyone who has understood, please explain to me, or copy/paste the final consensual definitions:
|
I've edited my previous comment. I'll refrain from discussing migration strategies here, this could be done later in the docs. @MahmoudKassem What do you think of that url syntax? I think with that we can also reject an unknown schema at the http level and we wouldn't need extra work at the sql level. |
Thanks @steve-chavez for edits on explanations. Commenting it:
This endpoint-syntax proposal breaks compatibility of classic PostgREST, where the endpoint path
Hum... Maybe you want to say "only for one default schema"... Or are you also affirming that it is not a proposal for multiple schemas? Note about versioning demandsThere are two independent main motivations for PostgREST endpoint versioning, not only one:
Ideal is 3 independent proposals for issue discusstion: database versioning, PostgREST versioning and multiple schema. Your proposal merges multiple schema demand with database versioning demand. |
@ppKrauss You're right about the shortcomings of the url versioning. I think we can agree that api versioning has no right way. So the best we can do is to provide flexibility and let the user do the url versioning if he wants. The only way to do this is to switch schemas with a header. I've never been a fan of adding the version in the media type since it makes the api harder to explore, but one of the articles you shared proposes the The only problem is that that header is not that "semantic" for schema based mutlitenancy. I've seen two other headers that seemed more fit for this: Accept-Profile and Accept-Schema. But I'm not sure if we would be abusing them by just providing a string value like All in all, I think the safest way would be to go with a header(whichever we choose). We could then provide an nginx snippet for users that'd like the url versioning. |
I think pREST can do it. But I haven't tried it yet... |
I think we can go with the @MahmoudKassem What do you think? |
@steve-chavez I agree on community conses being more important than semantic coherence, so I've modified the schema passing to use the Accept-Version header instead of the URL https://github.com/mahmoudkassem/postgrest/commit/be330be9e74fa695a8ba278db39f15e3061c7183. |
@MahmoudKassem Design-wise I think this is done! All what's left would be adding tests. You could add a postgrest/test/fixtures/schema.sql Lines 15 to 20 in 663faa1
Then add a new spec in This spec would take a Lines 95 to 96 in 663faa1
In
You can open a PR while you do this if you'd like, we can keep working on the tests there. |
@steve-chavez I opened the PR #1436 and will add the tests now. Thanks to your detailed outline writing the tests should be straightforward I will probably require support with maintaining the sandbox on the query level though. This is not implemented yet ofc this would also need to have tests as well. |
Solved in #1450! Basically, the schema to be used can be selected through the The headers are based on this nascent spec: Content Negotiation by Profile. Edit: This is only possible if |
Hello team, first of all thank you for the feature. I'm trying to make use of multiple schemas in a single instance, but I have a limitation of not being able to know the Edit: I found the answer within the documentation itself. I had missed it earlier. Please ignore this comment. |
This is needed both for api versioning and schema based multitenancy, currently it can be done with many PostgREST instances but it's unwieldy as there's considerable ops overhead, I'll make my proposal from the api versioning perspective.
Specifying the API version
Traditionally this has been done by having a prefix url path like
/v1/projects
, other options include a header that specifies the version or a date, like Stripe does, or a mime type that appends the version(application/vnd.com.v2+json
).All of this approaches are global and if we want to make them work with just SQL it would imply having to copy all the unmodified views from
v1
schema tov2
, ideally only the views that contain breaking changes would be inv2
.We can reuse PostgreSQL dot notation in the url to version the resources, this also has the advantage of being able to define embeds and specify the target resource version:
(Notice that in the filter
v2.clients
is just referred asclients
, the schema can only be specified at the url path and in theselect
part, this will ensure that we don't break our current filter syntax)Maintaning backwards compatibility
I think that REST API evolution can be handled in a similar way to regular software packages evolution,
we can use schema names that follow SemVer, or maybe just a MAJOR MINOR convention(like
v1_2
, underscore used to avoid conflicts with.
) to represent additions/breaking changes, comments on columns/views/schemas can notify deprecations and maybe release date.This combined with the schemas in the
search_path
, specified throughdb-schema
as previously discussed, would allow us to maintain backwards compatibility:Dropping support can be achieved by removing the schema from the
search_path
and maybe doing a redirect with a 3xx to the new url, this part would have to be done in the proxy layer(maybe later when adding #1088 this could be done in thepre-request
function).Security considerations
That we can specify the schema on the url will not mean that a client can access any schema like
pg_catalog
, we will only allow querying the schemas specified in thedb-schema
, internally we'll explicitly qualify all the SQL queries as we have done until now with a single schema.The text was updated successfully, but these errors were encountered: