BEP-25: MongoDB query and aggregation

[davie0]

Signed-off-by: David Dashyan mail@davie.li

[elmadj]

Thanks for your comments. Here are some thoughts and questions:

About security/data privacy:
In the case where the blockchain is public, the new proposed features, will allow access to assets
which were already auditable. So we're not adding new ways to see data that is supposed to be
hidden.
In the case where the blockchain is private, it's already possible to do text searches on the
blockchain, so although we giving the user more possibilities to search through and explore the
assets, it seems that possibility is already there.
It is also notable that the new features only allow read-only access.

The plugin-style solution seems perfect to address the generalization to other data stores. It also seems to address any security concerns. We could implement it. Do you have an example of such a setup ?
As I understand it, it would be a configuration that an admin would enable to give users access to the additional endpoints that we are implementing here.

[davie0]

I see! I'll clarify what I meant a bit.

I am aware that we have text search feature anyway. I am more concerned with the fact that we are exposing the mongodb to the public in this BEP. I don't know how this could be exploited. I am not aware of such "nosql" injections but my feeling is that we have to be careful here. It is bft and all that and even in the case that some one will drop the mongodb It will eventually sync. But it can potentially cause problems if such attack is targets the whole network.

I was thinking about adding contrib section to repository I am not yet came to conclusion how to implement It but I'll try experiment with It this week. If you have Ideas I'd love to hear it!

[elmadj]

Here is a proposal for the structure of the contrib folder. I can prototype it quickly to have a quick example so we can iterate.

Goal: integrate the query and aggregation features in way which makes it possible for an administrator to turn the features on and off depending on the use case.
Please refer to the duscussion on the github page : #97

Structure:

The new code will live under a contrib folder in the repository.
The same structure which is used to implement existing endpoints will be replicated in the contrib folder and the following files will be created :

contrib/
  assets/
    query-endpoint/
      web/
        routes.py
      views/
        views.py
      lib.py
      setup.py
    aggregate-endpoint/
      web/
        routes.py
      views/
        views.py
      lib.py
      setup.py

This structure may vary depending on the module, but the idea is to have a general topic folder to group modules per subject ( assets in this case) and then to have one folder per module which will mimic the existing structure under BigChainDB.

Behaviour:

In both modules, setup.py will have a setup function which will load the logic necessary to instantiate the new endpoints along with their supporting logic.
In particular, it will creates the routes definition.

A new configuration will be created with the following format :
{"contrib_modules": ["query-endpoint", "aggregate-endpoint"]}

The server class (in bigchaindb/web/server.py) will look at the contrib_modules config (It is already reading the configuration) and add the new routes along with their underlying logic (logic similar to the add_routes function)

lib.py in each module will have the underlying logic for the operations on assets.

Administrators will just have to list the contrib-modules to be loaded in an environment variables or config json file (both are already supported)

[lana-shanghai]

Hi Madjid, that is very similar to what we discussed with David today, as you said, for the type of contribution such as yours (e.g., queries) there will be a contrib folder where you place the new modules. The question was how to do it so depending on what module we are extending. @zzappie here is another suggestion to organize the contrib process.

[elmadj]

@lana-shanghai We can prototype this with an example to show how it would work with the query-endpoint from the original proposal. It would probably take me a couple of days.
If you have another proposal for the structure, I would be glad to discuss it and see how I could implement an example for the new endpoints.

[davie0]

@elmadj Thanks. Yea -the structure you have described would be a great starting point.

[davie0]

Closing old pull requests