Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix offset parameter for experimental API calls #3159

Closed
wants to merge 8 commits into from
Closed

Fix offset parameter for experimental API calls #3159

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f14e663
Fix offset parameter in API call 'experimental/ciscat/results
Apr 23, 2019
c92fab9
Fix offset parameter for 'experimental/syscollector' API calls.
Apr 23, 2019
7c6d642
Merge branch '3.10' into fix-ciscat-pagination
May 8, 2019
3288ad6
Fix for experimental/packages call.
May 9, 2019
07c3d0e
Merge branch '3.10' into fix-ciscat-pagination
May 23, 2019
1df9db0
Fix offset parameter in distributed api calls.
May 23, 2019
ae1122e
Fix limit parameter in distributed api calls.
May 23, 2019
939481f
Fix limit parameter in distributed api calls.
May 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions framework/wazuh/ciscat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,5 +20,11 @@ def get_results_agent(agent_id, offset=0, limit=common.database_limit, select={}


def get_ciscat_results(offset=0, limit=common.database_limit, select=None, filters={}, search={}, sort={}):
return _get_agent_items(func=get_results_agent, offset=offset, limit=limit, select=select,
filters=filters, search=search, sort=sort, array=True)

result = _get_agent_items(func=get_results_agent, offset=0, limit=limit, select=select,
filters=filters, search=search, sort=sort, array=True)

if offset:
result['items'] = result['items'][offset:]

return result
15 changes: 15 additions & 0 deletions framework/wazuh/cluster/dapi/dapi.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,9 +218,23 @@ async def forward(node_name: Tuple) -> str:
# get the node(s) who has all available information to answer the request.
nodes = self.get_solver_node()
self.input_json['from_cluster'] = True
# apply offset parameter when merging results from nodes
if 'offset' in self.input_json['arguments']:
offset = self.input_json['arguments']['offset'] # save offset
self.input_json['arguments']['offset'] = 0 # set offset=0 before distributing call
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't agree with this solution. I'm not sure it solves the problem. Actually I think it breaks the offset for the rest of endpoints.

We cannot risk the correctness of common endpoints just to fix the experimental ones. We should think another approach and then come back with a better solution.

Copy link
Contributor Author

@druizz90 druizz90 May 28, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forward_request method is called only on distributed_master requests.

We have the following endpoints of distributed_master type:

  • GET /cluster/:node_id/logs
  • GET /rootcheck/:agent_id/cis
  • GET /rootcheck/:agent_id
  • GET /rootcheck/:agent_id/pci
  • GET /sca/:agent_id/checks/:id
  • GET /sca/:agent_id
  • GET /syscollector/:agent_id/packages
  • GET /syscollector/:agent_id/processes
  • GET /syscollector/:agent_id/ports
  • GET /syscollector/:agent_id/netaddr
  • GET /syscollector/:agent_id/netiface
  • GET /syscollector/:agent_id/netproto
  • GET /ciscat/:agent_id/results
  • GET /experimental/syscollector/os
  • GET /experimental/syscollector/hardware
  • GET /experimental/syscollector/packages
  • GET /experimental/syscollector/processes
  • GET /experimental/syscollector/ports
  • GET /experimental/syscollector/netaddr
  • GET /experimental/syscollector/netproto
  • GET /experimental/syscollector/netiface
  • GET /experimental/ciscat/results

Only experimentals calls merge results, so I think I should move the changes inside this conditional in order to avoid secondary effects and fix pagination in experimental endpoints:

wazuh/framework/wazuh/cluster/dapi/dapi.py

Line 229 in 939481f

if len(nodes) > 1:

Thoughts?

# apply limit parameter when merging results from nodes
if 'limit' in self.input_json['arguments']:
limit = self.input_json['arguments']['limit'] # save limit
self.input_json['arguments']['limit'] = common.database_limit # set limit=common.database_limit before distributing call
if len(nodes) > 1:
results = map(json.loads, await asyncio.shield(asyncio.gather(*[forward(node) for node in nodes.items()])))
final_json = {}
# set offset for merging results
if 'offset' in self.input_json['arguments']:
self.input_json['arguments']['offset'] = offset
# set limit for merging results
if 'limit' in self.input_json['arguments']:
self.input_json['arguments']['limit'] = limit
response = json.dumps(self.merge_results(results, final_json))
else:
response = await forward(next(iter(nodes.items())))
Expand Down Expand Up @@ -331,6 +345,7 @@ def merge_results(self, responses, final_json):
self.input_json['arguments']['sort']['fields'],
self.input_json['arguments']['sort']['order'])

final_json['data']['totalItems'] = len(final_json['data']['items']) # update totalItems after merging results
offset, limit = self.input_json['arguments']['offset'], self.input_json['arguments']['limit']
final_json['data']['items'] = final_json['data']['items'][offset:offset+limit]

Expand Down
110 changes: 90 additions & 20 deletions framework/wazuh/syscollector.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,67 +164,137 @@ def get_netiface_agent(agent_id, offset=0, limit=common.database_limit, select={


def _get_agent_items(func, offset, limit, select, filters, search, sort, array=False):
agents, result = Agent.get_agents_overview(select={'fields': ['id']})['items'], []

total = 0

for agent in agents:
items = func(agent_id = agent['id'], select = select, filters = filters, limit = limit, offset = offset, search = search, sort=sort, nested=False)
def get_items(func, agent_id, select, filters, limit, offset, search, sort, nested):
"""
Append items of an agent to 'result' list. It call itself if limit is rebased
"""
items = func(agent_id=agent_id, select=select, filters=filters,
limit=limit, offset=offset, search=search, sort=sort, nested=False)
if items == {}:
continue

total += 1 if not array else items['totalItems']
return
items = [items] if not array else items['items']

for item in items:
if 0 < limit <= len(result):
break
item['agent_id'] = agent['id']
item['agent_id'] = agent_id
result.append(item)

if len(items) == common.database_limit:
offset += common.database_limit
get_items(func, agent_id=agent_id, select=select, filters=filters,
limit=limit, offset=offset, search= search, sort=sort,
nested=False)
# update total items
nonlocal total
total += 1 if not array else len(items)

agents, result = Agent.get_agents_overview(select={'fields': ['id']})['items'], []
total = 0

for agent in agents:
get_items(func=func, agent_id = agent['id'], select = select,
filters = filters, limit = limit, offset = 0,
search = search, sort=sort, nested=False)

if result:
if sort and sort['fields']:
result = sorted(result, key=itemgetter(sort['fields'][0]), reverse=True if sort['order'] == "desc" else False)

fields_to_nest, non_nested = get_fields_to_nest(result[0].keys(), '_')

return {'items': list(map(lambda x: plain_dict_to_nested_dict(x, fields_to_nest, non_nested), result)), 'totalItems': total}


def get_packages(offset=0, limit=common.database_limit, select=None, filters={}, search={}, sort={}):
return _get_agent_items(func=get_packages_agent, offset=offset, limit=limit, select=select,
result = _get_agent_items(func=get_packages_agent, offset=0, limit=common.database_limit, select=select,
filters=filters, search=search, sort=sort, array=True)
if offset:
result['items'] = result['items'][offset:]

if limit:
result['items'] = result['items'][:limit]

return result


def get_os(filters={}, offset=0, limit=common.database_limit, select={}, search={}, sort={}):
return _get_agent_items(func=get_os_agent, offset=offset, limit=limit, select=select,
result = _get_agent_items(func=get_os_agent, offset=0, limit=common.database_limit, select=select,
filters=filters, search=search, sort=sort)
if offset:
result['items'] = result['items'][offset:]

if limit:
result['items'] = result['items'][:limit]

return result


def get_hardware(offset=0, limit=common.database_limit, select=None, sort=None, filters={}, search={}):
return _get_agent_items(func=get_hardware_agent, offset=offset, limit=limit, select=select,
result = _get_agent_items(func=get_hardware_agent, offset=0, limit=common.database_limit, select=select,
filters=filters, search=search, sort=sort)
if offset:
result['items'] = result['items'][offset:]

if limit:
result['items'] = result['items'][:limit]

return result


def get_processes(offset=0, limit=common.database_limit, select=None, sort=None, filters={}, search={}):
return _get_agent_items(func=get_processes_agent, offset=offset, limit=limit, select=select,
result = _get_agent_items(func=get_processes_agent, offset=0, limit=common.database_limit, select=select,
filters=filters, search=search, sort=sort, array=True)
if offset:
result['items'] = result['items'][offset:]

if limit:
result['items'] = result['items'][:limit]

return result


def get_ports(offset=0, limit=common.database_limit, select=None, sort=None, filters={}, search={}):
return _get_agent_items(func=get_ports_agent, offset=offset, limit=limit, select=select,
result =_get_agent_items(func=get_ports_agent, offset=0, limit=common.database_limit, select=select,
filters=filters, search=search, sort=sort, array=True)
if offset:
result['items'] = result['items'][offset:]

if limit:
result['items'] = result['items'][:limit]

return result


def get_netaddr(offset=0, limit=common.database_limit, select=None, sort=None, filters={}, search={}):
return _get_agent_items(func=get_netaddr_agent, offset=offset, limit=limit, select=select,
result = _get_agent_items(func=get_netaddr_agent, offset=0, limit=common.database_limit, select=select,
filters=filters, search=search, sort=sort, array=True)
if offset:
result['items'] = result['items'][offset:]

if limit:
result['items'] = result['items'][:limit]

return result


def get_netproto(offset=0, limit=common.database_limit, select=None, sort=None, filters={}, search={}):
return _get_agent_items(func=get_netproto_agent, offset=offset, limit=limit, select=select,
result = _get_agent_items(func=get_netproto_agent, offset=0, limit=common.database_limit, select=select,
filters=filters, search=search, sort=sort, array=True)
if offset:
result['items'] = result['items'][offset:]

if limit:
result['items'] = result['items'][:limit]

return result


def get_netiface(offset=0, limit=common.database_limit, select=None, sort=None, filters={}, search={}):
return _get_agent_items(func=get_netiface_agent, offset=offset, limit=limit, select=select,
result = _get_agent_items(func=get_netiface_agent, offset=0, limit=common.database_limit, select=select,
filters=filters, search=search, sort=sort, array=True)
if offset:
result['items'] = result['items'][offset:]

if limit:
result['items'] = result['items'][:limit]

return result