Download/Export API

[ifokeev]

Describe the bug
api-gateway doesn't allow to pass limit over 50000 https://github.com/cube-js/cube.js/blob/1a8260522a234ebaabb9360f58e9b62095fd87f7/packages/cubejs-api-gateway/index.js#L133 and that's strange. I have no ability to use offset, but I want to increase the limit.

Expected behavior
There should be no limit in api-gateway but may be in playground

---

Updated by @igorlukanin. See partial solution: #251 (comment)

[paveltiunov]

@ifokeev Thanks for posting this! There's hard limit for security purpose. Could you please elaborate a little bit on your use case?

[ifokeev]

There's hard limit for security purpose

This hard limit should be inside playground and not in API. User knows better what SQL he needs while using standalone API. If it's so important so there may be something like skipRestrictions flag

Could you please elaborate a little bit on your use case?

I just need to query more than 50000 of rows. I use it to predict dimensions, not measures. Now CubeJS limiting my opportunity to do this.

[paveltiunov]

@ifokeev I see. Most of cube.js API are exposed to some sort of untrusted environments directly. Without this check it's very easy to exploit it for DoS attack.

As it was discussed previously I'd suggest to provide separate API for export and downloading results. What do you think?

[ifokeev]

@paveltiunov

As it was discussed previously I'd suggest to provide separate API for export and downloading results. What do you think?

May be a good solution. I faced at cubejs-server-core is not extendable by design and I can't change schema validation of cubejs-api-gateway too.
Other solution could be just to allow user-pass schema for queries or disable it at all. For example, I use GraphQL API and don't need validation from api-gateway.

[rickj33]

Most of cube.js API are exposed to some sort of untrusted environments directly. Without this check it's very easy to exploit it for DoS attack.

I dont think it is cube.js responsibility to protect against DoS attacks. Having a default value limiting the results is good, but I should be able override that value to something greater if my requirements are such. If I do then it is my responsibility to protect against DoS attacks. In my case, I am using cube.js in an internal trusted corporate environment where I do not have to worry about DoS attacks.

[paveltiunov]

@rickj33 Hey Rick! I think we can consider adding option to override default limit on server. Could you please elaborate on your use case though? Is it browser that requires to load more results or some other service that just hits cube.js API? How many rows in total do you need to download?

[ifokeev]

@paveltiunov let's go with overriding default validation schema, not only the limit

[rickj33]

That sounds good.

[paveltiunov]

@ifokeev Could you please elaborate?

[ifokeev]

@paveltiunov I mean we need user defined schema here:
https://github.com/cube-js/cube.js/blob/master/packages/cubejs-api-gateway/index.js#L93

Allow user pass his schema or disable it at all

[nikevp]

+1 on api export

[JoshMentzer]

@paveltiunov Here's an use case I have - we've setup our own query 'playground' front end for people to build into an angular dashboard app. When they set a filter, we submit a query for that dimension and same other filters to populate an autocomplete dropdown. Sometimes they select a field that has more than 50k potential items in it; and as such, they don't return all values and end up thinking that the system is 'missing' things. Granted, in some cases we could work around this using paging/offset etc, but we can't count on being able to do that 100% of the time.
I personally would love to be able to override the query limit on queries.