New labels format in json #3490

SalvaMilenario · 2019-06-07T12:10:49Z

Related issue
#3429

Description

The format of the labels is changed so that they appear by levels as it is encoded in JSON.

The new format shows in addition to the name of the key and its value tagged as such, it has been added to show if it is a hidden label.

Additionally, the expected behavior when saving two labels with the same key is that only the second one that is entered is saved.

The format with which the labels are displayed is:

# curl -u foo:bar -k -X GET "http://172.16.1.13:55000/agents/001/config/agent/labels?pretty"
{
   "error": 0,
   "data": {
      "labels": [
         {
            "value": "i-001c",
            "key": "id",
            "hidden": "no"
         },
         {
            "value": "default",
            "key": "group",
            "hidden": "no"
         },
         {
            "value": "172.16.1.14",
            "key": "ip",
            "hidden": "no"
         },
         {
            "value": "Sep 18th, 2019",
            "key": "installation",
            "hidden": "yes"
         }
      ]
   }
}

Tests

Compilation without warnings in every supported platform
- Linux
- Windows
Memory tests
- Valgrind report for affected components
- Scan-build
- Address Sanitizer

src/client-agent/config.c

JuantAldea

Please add Valgrind, Scan-build and Address Sanitizer

src/client-agent/config.c

…om/wazuh/wazuh into fix-3429-labels_response_on-demand

SalvaMilenario · 2019-09-13T12:50:44Z

Hi team

This is logs of the valgrind:

==9035== 
==9035== HEAP SUMMARY:
==9035==     in use at exit: 93,264 bytes in 57 blocks
==9035==   total heap usage: 6,308 allocs, 6,251 frees, 3,260,632 bytes allocated
==9035== 
==9035== 272 bytes in 1 blocks are possibly lost in loss record 37 of 48
==9035==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9035==    by 0x4012EE4: allocate_dtv (dl-tls.c:296)
==9035==    by 0x4012EE4: _dl_allocate_tls (dl-tls.c:460)
==9035==    by 0x57E1D92: allocate_stack (allocatestack.c:589)
==9035==    by 0x57E1D92: pthread_create@@GLIBC_2.2.5 (pthread_create.c:500)
==9035==    by 0x465B90: CreateThreadJoinable (pthreads_op.c:47)
==9035==    by 0x465C27: CreateThread (pthreads_op.c:62)
==9035==    by 0x406606: AgentdStart (agentd.c:127)
==9035==    by 0x40909F: main (main.c:179)
==9035== 
==9035== 272 bytes in 1 blocks are possibly lost in loss record 38 of 48
==9035==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9035==    by 0x4012EE4: allocate_dtv (dl-tls.c:296)
==9035==    by 0x4012EE4: _dl_allocate_tls (dl-tls.c:460)
==9035==    by 0x57E1D92: allocate_stack (allocatestack.c:589)
==9035==    by 0x57E1D92: pthread_create@@GLIBC_2.2.5 (pthread_create.c:500)
==9035==    by 0x465B90: CreateThreadJoinable (pthreads_op.c:47)
==9035==    by 0x465C27: CreateThread (pthreads_op.c:62)
==9035==    by 0x40664F: AgentdStart (agentd.c:136)
==9035==    by 0x40909F: main (main.c:179)
==9035== 
==9035== 272 bytes in 1 blocks are possibly lost in loss record 39 of 48
==9035==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9035==    by 0x4012EE4: allocate_dtv (dl-tls.c:296)
==9035==    by 0x4012EE4: _dl_allocate_tls (dl-tls.c:460)
==9035==    by 0x57E1D92: allocate_stack (allocatestack.c:589)
==9035==    by 0x57E1D92: pthread_create@@GLIBC_2.2.5 (pthread_create.c:500)
==9035==    by 0x465B90: CreateThreadJoinable (pthreads_op.c:47)
==9035==    by 0x465C27: CreateThread (pthreads_op.c:62)
==9035==    by 0x40677B: AgentdStart (agentd.c:150)
==9035==    by 0x40909F: main (main.c:179)
==9035== 
==9035== 272 bytes in 1 blocks are possibly lost in loss record 40 of 48
==9035==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9035==    by 0x4012EE4: allocate_dtv (dl-tls.c:296)
==9035==    by 0x4012EE4: _dl_allocate_tls (dl-tls.c:460)
==9035==    by 0x57E1D92: allocate_stack (allocatestack.c:589)
==9035==    by 0x57E1D92: pthread_create@@GLIBC_2.2.5 (pthread_create.c:500)
==9035==    by 0x465B90: CreateThreadJoinable (pthreads_op.c:47)
==9035==    by 0x465C27: CreateThread (pthreads_op.c:62)
==9035==    by 0x4068C8: AgentdStart (agentd.c:188)
==9035==    by 0x40909F: main (main.c:179)
==9035== 
==9035== LEAK SUMMARY:
==9035==    definitely lost: 0 bytes in 0 blocks
==9035==    indirectly lost: 0 bytes in 0 blocks
==9035==      possibly lost: 1,088 bytes in 4 blocks
==9035==    still reachable: 92,176 bytes in 53 blocks
==9035==         suppressed: 0 bytes in 0 blocks
==9035== Reachable blocks (those to which a pointer was found) are not shown.
==9035== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9035== 
==9035== For counts of detected and suppressed errors, rerun with: -v
==9035== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)

The compilation supervised with scan-build not notificated error in the changed files. In addition, address sanitizer not report others memory leak.

JuantAldea

Add tests as discussed.

SalvaMilenario · 2019-09-16T10:22:31Z

The JSON that is generated in the function getLabelsConfig it's the same that the API shows with it.

# curl -u foo:bar -k -X GET "http://172.16.1.13:55000/agents/002/config/agent/labels?pretty"
{
   "error": 0,
   "data": {
      "labels": [
         {
            "label": [
               {
                  "value": "002",
                  "key": "id",
                  "hidden": "no"
               },
               {
                  "value": "default",
                  "key": "group",
                  "hidden": "no"
               },
               {
                  "value": "172.16.1.18",
                  "key": "ip",
                  "hidden": "no"
               },
               {
                  "value": "Sep. 16th, 2019",
                  "key": "installation",
                  "hidden": "yes"
               }
            ]
         }
      ]
   }
}

2019/09/16 10:12:58 ossec-agentd: INFO: {
	"labels":	[{
			"label":	[{
					"value":	"002",
					"key":	"id",
					"hidden":	"no"
				}, {
					"value":	"default",
					"key":	"group",
					"hidden":	"no"
				}, {
					"value":	"172.16.1.18",
					"key":	"ip",
					"hidden":	"no"
				}, {
					"value":	"Sep. 16th, 2019",
					"key":	"installation",
					"hidden":	"yes"
				}]
		}]
}

chemamartinez · 2019-09-17T14:19:52Z

After discussing it with the @wazuh/framework, it doesn't make sense to include the label array since it will be always a unique structure for all the labels of an agent. The correct JSON output should look like:

# curl -u foo:bar -k -X GET "http://172.16.1.13:55000/agents/002/config/agent/labels?pretty"
{
   "error": 0,
   "data": {
      "labels": [
               {
                  "value": "002",
                  "key": "id",
                  "hidden": "no"
               },
               {
                  "value": "default",
                  "key": "group",
                  "hidden": "no"
               },
               {
                  "value": "172.16.1.18",
                  "key": "ip",
                  "hidden": "no"
               },
               {
                  "value": "Sep. 16th, 2019",
                  "key": "installation",
                  "hidden": "yes"
               }
      ]
   }
}

Please @SalvaMilenario apply the changes and test it works fine. Ping @wazuh/frontend as well to check it is a valid format for you to parse this JSON format to XML.

SalvaMilenario · 2019-09-18T09:43:11Z

Hi

I've changed the code with the new JSON format:

# curl -u foo:bar -k -X GET "http://172.16.1.13:55000/agents/001/config/agent/labels?pretty"
{
   "error": 0,
   "data": {
      "labels": [
         {
            "value": "i-001c",
            "key": "id",
            "hidden": "no"
         },
         {
            "value": "default",
            "key": "group",
            "hidden": "no"
         },
         {
            "value": "172.16.1.14",
            "key": "ip",
            "hidden": "no"
         },
         {
            "value": "Sep 18th, 2019",
            "key": "installation",
            "hidden": "yes"
         }
      ]
   }
}

The compilation supervised with scan-build doesn't notification any bugs, and the execution with Valgrind doesn't indicate any memory leaks. Besides, address sanitizer not report other memory leaks.

==5978== 
==5978== HEAP SUMMARY:
==5978==     in use at exit: 93,213 bytes in 55 blocks
==5978==   total heap usage: 41,430 allocs, 41,375 frees, 1,096,927,016 bytes allocated
==5978== 
==5978== 272 bytes in 1 blocks are possibly lost in loss record 37 of 48
==5978==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5978==    by 0x4012EE4: allocate_dtv (dl-tls.c:296)
==5978==    by 0x4012EE4: _dl_allocate_tls (dl-tls.c:460)
==5978==    by 0x57E1D92: allocate_stack (allocatestack.c:589)
==5978==    by 0x57E1D92: pthread_create@@GLIBC_2.2.5 (pthread_create.c:500)
==5978==    by 0x4690B4: CreateThreadJoinable (pthreads_op.c:47)
==5978==    by 0x46914B: CreateThread (pthreads_op.c:62)
==5978==    by 0x406620: AgentdStart (agentd.c:131)
==5978==    by 0x40910D: main (main.c:179)
==5978== 
==5978== 272 bytes in 1 blocks are possibly lost in loss record 38 of 48
==5978==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5978==    by 0x4012EE4: allocate_dtv (dl-tls.c:296)
==5978==    by 0x4012EE4: _dl_allocate_tls (dl-tls.c:460)
==5978==    by 0x57E1D92: allocate_stack (allocatestack.c:589)
==5978==    by 0x57E1D92: pthread_create@@GLIBC_2.2.5 (pthread_create.c:500)
==5978==    by 0x4690B4: CreateThreadJoinable (pthreads_op.c:47)
==5978==    by 0x46914B: CreateThread (pthreads_op.c:62)
==5978==    by 0x406669: AgentdStart (agentd.c:139)
==5978==    by 0x40910D: main (main.c:179)
==5978== 
==5978== 272 bytes in 1 blocks are possibly lost in loss record 39 of 48
==5978==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5978==    by 0x4012EE4: allocate_dtv (dl-tls.c:296)
==5978==    by 0x4012EE4: _dl_allocate_tls (dl-tls.c:460)
==5978==    by 0x57E1D92: allocate_stack (allocatestack.c:589)
==5978==    by 0x57E1D92: pthread_create@@GLIBC_2.2.5 (pthread_create.c:500)
==5978==    by 0x4690B4: CreateThreadJoinable (pthreads_op.c:47)
==5978==    by 0x46914B: CreateThread (pthreads_op.c:62)
==5978==    by 0x406795: AgentdStart (agentd.c:151)
==5978==    by 0x40910D: main (main.c:179)
==5978== 
==5978== 272 bytes in 1 blocks are possibly lost in loss record 40 of 48
==5978==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5978==    by 0x4012EE4: allocate_dtv (dl-tls.c:296)
==5978==    by 0x4012EE4: _dl_allocate_tls (dl-tls.c:460)
==5978==    by 0x57E1D92: allocate_stack (allocatestack.c:589)
==5978==    by 0x57E1D92: pthread_create@@GLIBC_2.2.5 (pthread_create.c:500)
==5978==    by 0x4690B4: CreateThreadJoinable (pthreads_op.c:47)
==5978==    by 0x46914B: CreateThread (pthreads_op.c:62)
==5978==    by 0x4068F6: AgentdStart (agentd.c:189)
==5978==    by 0x40910D: main (main.c:179)
==5978== 
==5978== LEAK SUMMARY:
==5978==    definitely lost: 0 bytes in 0 blocks
==5978==    indirectly lost: 0 bytes in 0 blocks
==5978==      possibly lost: 1,088 bytes in 4 blocks
==5978==    still reachable: 92,125 bytes in 51 blocks
==5978==         suppressed: 0 bytes in 0 blocks
==5978== Reachable blocks (those to which a pointer was found) are not shown.
==5978== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==5978== 
==5978== For counts of detected and suppressed errors, rerun with: -v
==5978== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)

Regards

JcabreraC · 2019-09-25T08:45:24Z

Test

The API call works correctly and shows the new labels configuration format:

# curl -u foo:bar -k -X GET "http://10.0.0.1:55000/agents/014/config/agent/labels?pretty"
{
   "error": 0,
   "data": {
      "labels": [
         {
            "value": "i-052a1838d",
            "key": "aws.instance-id",
            "hidden": "no"
         },
         {
            "value": "sg-1103",
            "key": "aws.sec-group",
            "hidden": "no"
         },
         {
            "value": "172.17.0.0",
            "key": "network.ip",
            "hidden": "no"
         },
         {
            "value": "02:42:ac:11:00:02",
            "key": "network.mac",
            "hidden": "no"
         },
         {
            "value": "January 1st, 2017",
            "key": "installation",
            "hidden": "yes"
         }
      ]
   }
}

This information is also displayed in the kibana app:

chemamartinez · 2019-09-25T10:18:05Z

Blocked by the @wazuh/frontend. The labels configuration should be displayed as follows for the new format:

juankaromo · 2019-09-25T10:22:05Z

Working on it here wazuh/wazuh-dashboard-plugins#1816

JcabreraC · 2019-09-25T15:13:40Z

Test

New JSON format added to the manager. The API call works correctly and shows the new labels configuration format:

# curl -u foo:bar -k -X GET "http://10.0.0.1:55000/manager/config/analysis/labels?pretty"
{
   "error": 0,
   "data": {
      "labels": [
         {
            "value": "i-052a1838c",
            "key": "aws.instance-id",
            "hidden": "no"
         },
         {
            "value": "sg-1103",
            "key": "aws.sec-group",
            "hidden": "no"
         },
         {
            "value": "172.17.0.0",
            "key": "network.ip",
            "hidden": "no"
         },
         {
            "value": "02:42:ac:11:00:02",
            "key": "network.mac",
            "hidden": "no"
         },
         {
            "value": "January 1st, 2017",
            "key": "installation",
            "hidden": "yes"
         }
      ]
   }
}

Tests were added after the review

salvamilenario added 2 commits June 7, 2019 11:33

New label response format for on-demand agents

0de44eb

A level is added in the labels

d79a392

bah07 self-requested a review June 7, 2019 12:47

bah07 self-assigned this Jun 7, 2019

salvamilenario added 2 commits June 10, 2019 12:48

Eliminated the capitals of the labels

c9074c6

Error correction in "system" of the previous commit

6614eac

bah07 changed the base branch from 3.10 to 3.11 June 13, 2019 13:22

bah07 marked this pull request as ready for review June 13, 2019 13:22

Free variable memory label_k

0133f4a

JuantAldea self-requested a review September 9, 2019 15:09

JuantAldea suggested changes Sep 9, 2019

View reviewed changes

Changes requested.

4cb3e15

JuantAldea assigned JuantAldea and unassigned JuantAldea Sep 10, 2019

SalvaMilenario requested a review from JuantAldea September 11, 2019 09:25

JuantAldea suggested changes Sep 11, 2019

View reviewed changes

src/client-agent/config.c Outdated Show resolved Hide resolved

JuantAldea suggested changes Sep 11, 2019

View reviewed changes

src/client-agent/config.c Show resolved Hide resolved

Changes requested.

9a88a8e

SalvaMilenario changed the title ~~Fix 3429 labels response on demand~~ New labels format in json Sep 12, 2019

SalvaMilenario requested a review from JuantAldea September 12, 2019 15:09

salvamilenario added 8 commits September 12, 2019 17:52

New label response format for on-demand agents

47fa593

A level is added in the labels

d05b2b7

Eliminated the capitals of the labels

040ba88

Error correction in "system" of the previous commit

088402d

Free variable memory label_k

a97146a

Changes requested.

e0fe2a1

Changes requested.

0cd64ff

Merge branch 'fix-3429-labels_response_on-demand' of https://github.c…

a0dbb56

…om/wazuh/wazuh into fix-3429-labels_response_on-demand

SalvaMilenario changed the base branch from 3.11 to 3.10 September 12, 2019 16:03

JuantAldea previously requested changes Sep 13, 2019

View reviewed changes

SalvaMilenario requested a review from JuantAldea September 16, 2019 10:22

chemamartinez self-requested a review September 17, 2019 14:17

New JSON format for the labels.

887645e

chemamartinez requested a review from JcabreraC September 19, 2019 10:37

JcabreraC approved these changes Sep 25, 2019

View reviewed changes

chemamartinez approved these changes Sep 25, 2019

View reviewed changes

Added-labels-format-to-manager

61d7c96

chemamartinez changed the base branch from 3.10 to 3.11 September 26, 2019 06:51

chemamartinez merged commit ae91b9c into 3.11 Sep 26, 2019

chemamartinez deleted the fix-3429-labels_response_on-demand branch September 26, 2019 07:01

chemamartinez mentioned this pull request Sep 26, 2019

Adapt how Labels configuration JSON response is shown wazuh/wazuh-dashboard-plugins#1816

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New labels format in json #3490

New labels format in json #3490

SalvaMilenario commented Jun 7, 2019 •

edited

Loading

JuantAldea left a comment

SalvaMilenario commented Sep 13, 2019

JuantAldea left a comment

SalvaMilenario commented Sep 16, 2019

chemamartinez commented Sep 17, 2019

SalvaMilenario commented Sep 18, 2019 •

edited

Loading

JcabreraC commented Sep 25, 2019

chemamartinez commented Sep 25, 2019

juankaromo commented Sep 25, 2019

JcabreraC commented Sep 25, 2019

New labels format in json #3490

New labels format in json #3490

Conversation

SalvaMilenario commented Jun 7, 2019 • edited Loading

Description

Tests

JuantAldea left a comment

Choose a reason for hiding this comment

SalvaMilenario commented Sep 13, 2019

JuantAldea left a comment

Choose a reason for hiding this comment

SalvaMilenario commented Sep 16, 2019

chemamartinez commented Sep 17, 2019

SalvaMilenario commented Sep 18, 2019 • edited Loading

JcabreraC commented Sep 25, 2019

Test

chemamartinez commented Sep 25, 2019

juankaromo commented Sep 25, 2019

JcabreraC commented Sep 25, 2019

Test

SalvaMilenario commented Jun 7, 2019 •

edited

Loading

SalvaMilenario commented Sep 18, 2019 •

edited

Loading