New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for aggregation event hooks #1057

Closed
aarobc opened this Issue Sep 3, 2017 · 11 comments

Comments

Projects
None yet
5 participants
@aarobc

aarobc commented Sep 3, 2017

I'm attempting to set up an aggregation endpoint that would select the first point from every day within the location.

An example of a document:

{
    "_id": ObjectID("59ab0a1950d0b5000183f88e"),
    "accuracy": 10,
    "velocity": 0,
    "heading": 0,
    "altitude": 1372,
    "created": ISODate("2017-08-03T23:14:47.000Z"),
    "date": "8/3/2017",
    "location": [
        0,
        0
    ]
}

I set up my settings.py like so:

MONGO_HOST="mongo"
MONGO_PORT=27017
MONGO_DBNAME="history"
ALLOW_UNKNOWN=True
PAGINATION_LIMIT=100
PAGINATION_DEFAULT=100
SCHEMA_ENDPOINT="first"

X_DOMAINS='*'

DOMAIN = {
    'mapped': {
        'allowed_filters' : ['*'],
        'sorting': True,
        'resource_methods': ['GET', 'POST'],


    }
}

first = {
    'datasource': {
        'aggregation': {
            'pipeline': [
               { "$match":{
                    "location": {
                        "$near": {
                            "$geometry": {
                                'type': "Point",
                                'coordinates': ["$lng", "$lat"]
                            },
                            "$maxDistance": "$radius"
                        }
                    }
                }},
                {"$group": {
                    "datee": "$date"
                }}

            ]
        }
    }
}

But testing in the browser retrieves no results. example: http://localhost:5000/first?aggregate={%22$lat%22:0,%22$lng%22:0%22$radius%22:12} in the browser gives:

<resource>
<mapped>
<_id>
<type>objectid</type>
</_id>
</mapped>
</resource>

What am I doing wrong? (Also Doesn't it seem odd you can only have one aggregation for the entire application as per the SCHEMA_ENDPOINT parameter?)

@sergekir

This comment has been minimized.

Contributor

sergekir commented Sep 3, 2017

Looks like #1025

I'm working on this problem

@aarobc

This comment has been minimized.

aarobc commented Sep 4, 2017

Thank you for being so helpful. I've attempted to use the approach you've linked to, but I'm still having the same problem. You can see the simplified and revised approach below:

MONGO_HOST="mongo"
MONGO_PORT=27017
MONGO_DBNAME="history"
ALLOW_UNKNOWN=True
PAGINATION_LIMIT=100
PAGINATION_DEFAULT=100
SCHEMA_ENDPOINT="first"

X_DOMAINS='*'

DOMAIN = {
    'mapped': {
        'allowed_filters' : ['*'],
        'sorting': True,
        'resource_methods': ['GET', 'POST'],
    }
}

first = {
    'datasource': {
        'source': 'mapped',
        'aggregation': {
            'pipeline': [
                {"$match": "$where$"}
            ]
        }
    }
}

the endpoint I'm hitting is:
http://localhost:5000/first?aggregate={"$where":{"altitude":1372}}
url encoded, that looks like:
http://localhost:5000/first?aggregate={%22$where%22:{%22altitude%22:1372}}

At this point I'm just wanting to see that using aggregation can select anything at all from the database, but I'm having the same problem.

Is this a configuration problem? Would you be able to point out where my misunderstanding is, or direct me to more comprehensive examples than are provided in the docs?

@Amedeo91

This comment has been minimized.

Amedeo91 commented Sep 4, 2017

This is how the aggregation work:

def parse_aggregation_stage(d, key, value):
    for st_key, st_value in d.items():
        if isinstance(st_value, dict):
            parse_aggregation_stage(st_value, key, value)
        if key == st_value:
            d[st_key] = value

response = {}
documents = []
req = parse_request(resource)

req_pipeline = copy.deepcopy(pipeline)
if req.aggregation:
    try:
        query = json.loads(req.aggregation)
    except ValueError:
        abort(400, description='Aggregation query could not be parsed.')

    for key, value in query.items():
        if key[0] != '$':
            pass
        for stage in req_pipeline:
            parse_aggregation_stage(stage, key, value)
@aarobc

This comment has been minimized.

aarobc commented Sep 4, 2017

@Amedeo91 That really doesn't clarify anything for me. Would you be able to give a corrected example of aggregation configured correctly using my example?

@Amedeo91

This comment has been minimized.

Amedeo91 commented Sep 5, 2017

That is the thing: I am not sure that Eve is able to parse dictionary to create an entire step of the aggregation pipeline!!

@sergekir

This comment has been minimized.

Contributor

sergekir commented Mar 5, 2018

The solution can be additional Event Hooks: before_aggregation and after_aggregation.

before_aggregation will access the pipeline for aggregation and can dynamically change it.

Is it relevant issue? Need some help?

@nrchandan

This comment has been minimized.

nrchandan commented Apr 24, 2018

Event hooks for aggregation sounds great. How can I find out if this is in the roadmap?

@nicolaiarocci nicolaiarocci added this to the 0.9 milestone Apr 24, 2018

@nicolaiarocci nicolaiarocci changed the title from Aggregation not working to Add support for aggregation event hooks Apr 24, 2018

@nicolaiarocci nicolaiarocci modified the milestones: 0.9, 0.8 Apr 24, 2018

@nicolaiarocci

This comment has been minimized.

Member

nicolaiarocci commented Apr 25, 2018

I just merged this feature to master.

@nrchandan

This comment has been minimized.

nrchandan commented Apr 25, 2018

Thanks Nicola! Would you recommend using the development version in a new yet-to-be released project? I am tempted to use it just for this feature, but I might have a workaround for the stable version as well.

PS. My untested workaround is to use a pass a match criteria to the aggregate function and require the web application pass the filter criteria (user ID) as an aggregate query parameter. I can use the pre-request event hook to ensure that the user ID provided matches the user ID of the current user. A little dirty, but it should work.

@nicolaiarocci

This comment has been minimized.

Member

nicolaiarocci commented Apr 26, 2018

A lot of people are using the development branch, I'd say it is (hopefully) stable enough. Also, I plan on releasing v0.8 next month.

@nrchandan

This comment has been minimized.

nrchandan commented Apr 27, 2018

I switched over to the dev branch. I am able to use the new pre-aggregate event hooks. Looks great, thank you!

I was also trying to understand your commit for this change. It looks very neat. Just two lines of code to enable the aggregate hook mechanism, but good amount of tests and documentation. Very well crafted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment