Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for aggregation event hooks #1057

Closed
aarobc opened this issue Sep 3, 2017 · 11 comments
Closed

Add support for aggregation event hooks #1057

aarobc opened this issue Sep 3, 2017 · 11 comments
Milestone

Comments

@aarobc
Copy link

@aarobc aarobc commented Sep 3, 2017

I'm attempting to set up an aggregation endpoint that would select the first point from every day within the location.

An example of a document:

{
    "_id": ObjectID("59ab0a1950d0b5000183f88e"),
    "accuracy": 10,
    "velocity": 0,
    "heading": 0,
    "altitude": 1372,
    "created": ISODate("2017-08-03T23:14:47.000Z"),
    "date": "8/3/2017",
    "location": [
        0,
        0
    ]
}

I set up my settings.py like so:

MONGO_HOST="mongo"
MONGO_PORT=27017
MONGO_DBNAME="history"
ALLOW_UNKNOWN=True
PAGINATION_LIMIT=100
PAGINATION_DEFAULT=100
SCHEMA_ENDPOINT="first"

X_DOMAINS='*'

DOMAIN = {
    'mapped': {
        'allowed_filters' : ['*'],
        'sorting': True,
        'resource_methods': ['GET', 'POST'],


    }
}

first = {
    'datasource': {
        'aggregation': {
            'pipeline': [
               { "$match":{
                    "location": {
                        "$near": {
                            "$geometry": {
                                'type': "Point",
                                'coordinates': ["$lng", "$lat"]
                            },
                            "$maxDistance": "$radius"
                        }
                    }
                }},
                {"$group": {
                    "datee": "$date"
                }}

            ]
        }
    }
}

But testing in the browser retrieves no results. example: http://localhost:5000/first?aggregate={%22$lat%22:0,%22$lng%22:0%22$radius%22:12} in the browser gives:

<resource>
<mapped>
<_id>
<type>objectid</type>
</_id>
</mapped>
</resource>

What am I doing wrong? (Also Doesn't it seem odd you can only have one aggregation for the entire application as per the SCHEMA_ENDPOINT parameter?)

@sergekir
Copy link
Contributor

@sergekir sergekir commented Sep 3, 2017

Looks like #1025

I'm working on this problem

@aarobc
Copy link
Author

@aarobc aarobc commented Sep 4, 2017

Thank you for being so helpful. I've attempted to use the approach you've linked to, but I'm still having the same problem. You can see the simplified and revised approach below:

MONGO_HOST="mongo"
MONGO_PORT=27017
MONGO_DBNAME="history"
ALLOW_UNKNOWN=True
PAGINATION_LIMIT=100
PAGINATION_DEFAULT=100
SCHEMA_ENDPOINT="first"

X_DOMAINS='*'

DOMAIN = {
    'mapped': {
        'allowed_filters' : ['*'],
        'sorting': True,
        'resource_methods': ['GET', 'POST'],
    }
}

first = {
    'datasource': {
        'source': 'mapped',
        'aggregation': {
            'pipeline': [
                {"$match": "$where$"}
            ]
        }
    }
}

the endpoint I'm hitting is:
http://localhost:5000/first?aggregate={"$where":{"altitude":1372}}
url encoded, that looks like:
http://localhost:5000/first?aggregate={%22$where%22:{%22altitude%22:1372}}

At this point I'm just wanting to see that using aggregation can select anything at all from the database, but I'm having the same problem.

Is this a configuration problem? Would you be able to point out where my misunderstanding is, or direct me to more comprehensive examples than are provided in the docs?

@Amedeo91
Copy link

@Amedeo91 Amedeo91 commented Sep 4, 2017

This is how the aggregation work:

def parse_aggregation_stage(d, key, value):
    for st_key, st_value in d.items():
        if isinstance(st_value, dict):
            parse_aggregation_stage(st_value, key, value)
        if key == st_value:
            d[st_key] = value

response = {}
documents = []
req = parse_request(resource)

req_pipeline = copy.deepcopy(pipeline)
if req.aggregation:
    try:
        query = json.loads(req.aggregation)
    except ValueError:
        abort(400, description='Aggregation query could not be parsed.')

    for key, value in query.items():
        if key[0] != '$':
            pass
        for stage in req_pipeline:
            parse_aggregation_stage(stage, key, value)
@aarobc
Copy link
Author

@aarobc aarobc commented Sep 4, 2017

@Amedeo91 That really doesn't clarify anything for me. Would you be able to give a corrected example of aggregation configured correctly using my example?

@Amedeo91
Copy link

@Amedeo91 Amedeo91 commented Sep 5, 2017

That is the thing: I am not sure that Eve is able to parse dictionary to create an entire step of the aggregation pipeline!!

@sergekir
Copy link
Contributor

@sergekir sergekir commented Mar 5, 2018

The solution can be additional Event Hooks: before_aggregation and after_aggregation.

before_aggregation will access the pipeline for aggregation and can dynamically change it.

Is it relevant issue? Need some help?

@nrchandan
Copy link

@nrchandan nrchandan commented Apr 24, 2018

Event hooks for aggregation sounds great. How can I find out if this is in the roadmap?

@nicolaiarocci nicolaiarocci added this to the 0.9 milestone Apr 24, 2018
@nicolaiarocci nicolaiarocci changed the title Aggregation not working Add support for aggregation event hooks Apr 24, 2018
@nicolaiarocci nicolaiarocci modified the milestones: 0.9, 0.8 Apr 24, 2018
@nicolaiarocci
Copy link
Member

@nicolaiarocci nicolaiarocci commented Apr 25, 2018

I just merged this feature to master.

@nrchandan
Copy link

@nrchandan nrchandan commented Apr 25, 2018

Thanks Nicola! Would you recommend using the development version in a new yet-to-be released project? I am tempted to use it just for this feature, but I might have a workaround for the stable version as well.

PS. My untested workaround is to use a pass a match criteria to the aggregate function and require the web application pass the filter criteria (user ID) as an aggregate query parameter. I can use the pre-request event hook to ensure that the user ID provided matches the user ID of the current user. A little dirty, but it should work.

@nicolaiarocci
Copy link
Member

@nicolaiarocci nicolaiarocci commented Apr 26, 2018

A lot of people are using the development branch, I'd say it is (hopefully) stable enough. Also, I plan on releasing v0.8 next month.

@nrchandan
Copy link

@nrchandan nrchandan commented Apr 27, 2018

I switched over to the dev branch. I am able to use the new pre-aggregate event hooks. Looks great, thank you!

I was also trying to understand your commit for this change. It looks very neat. Just two lines of code to enable the aggregate hook mechanism, but good amount of tests and documentation. Very well crafted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.