Support 5GTANGO catalogues as storage backend

[mpeuster]

• configure catalogue endpoint with ENV variables using the service's docker file (use localhost as fallback, e.g., local dev env)
• upload NAPD.yaml (Attention: ETS-only packages would not work with this backend!)
• upload VNFDs
• upload NSDs
• upload TESTDs
• upload original package file
• upload other arbitrary files to catalogue (e.g. test binaries)
• upload additional catalogue support mapping (endpoint in catalog?, implement in _build_catalog_metadata)
  • fix integration bug (400) File with {name => upb_logo.png, uuid => 2dd10810-d608-45cb-8712-4491df20fbca} not found in the Catalogue
• return package UUID and send it in callback
• unified upload error handling
• unit tests for storage.py (currently skip_store=True in test_unit_rest.py, re-enable!)
• expose skip_store as CLI and ENV config
• test with all three example packages from tng-schema and misc/

Skip all other files in the first prototype.

---

ENV variable naming:

 CATALOGUE_URL: http://tng-cat:4011/catalogues/api/v2

[mpeuster]

Upload order:

0. napd
1. vnfds (if exists)
2. nsds (if exists)
3. testds (if exists)
4. (any other file)
5. package file (*.tgo) = the ZIP
6. additional file mapping to help catalogue

Format of 6.:

"pd" : {
        "vendor" : "eu.sonata-nfv.package",
        "version" : "0.3.1",
        "name" : "sonata-demo"
    },
    "nsds" : [ 
        {
            "vendor" : "eu.sonata-nfv.service-descriptor",
            "version" : "0.2.1",
            "name" : "sonata-demo"
        }
    ],
    "vnfds" : [ 
        {
            "vendor" : "eu.sonata-nfv",
            "version" : "0.3",
            "name" : "firewall-vnf"
        }, 
        {
            "vendor" : "eu.sonata-nfv",
            "version" : "0.2",
            "name" : "iperf-vnf"
        }, 
        {
            "vendor" : "eu.sonata-nfv",
            "version" : "0.2",
            "name" : "tcpdump-vnf"
        }
    ],
    "deps" : [],
    "son_package_uuid" : "78cdd0e2-e3d4-4789-afc7-56269c202b76"

[mpeuster]

Agreement in call on 2018-04-26:

• v4.0: As described in the comment above. Mostly like SONATA.
• v4.X: Have more generic file storage possibilities.

[mpeuster]

Example for the content of the callback which is send from the packager to the GK to inform it that the package contents are finally stored in the catalog.

{  
    "event_name":"onPackageChangeEvent",
    "package_id":"24c616cf-fe01-4c08-ae44-45d43ae67576",
    "package_location":"http://tng-cat:4011/catalogues/api/v2/tgo-packages/24c616cf-fe01-4c08-ae44-45d43ae67576",
    "package_metadata":<all_other_stuff_tng-sdk-package_knows_about_the_package_ignore_it>,
    "package_process_uuid":"d5cea225-033f-4fc6-816f-4a642461086a",
    "package_process_status": "success"
}

Important: package_id is the UUID returned by the catalog when uploading the *.tng file. NOT the UUID of the package descriptor.

Is this the planned behavior, or should I send the UUID of the package descriptor? @panstav1 and @jbonnet ??

[jbonnet]

Well, @mpeuster, I was counting with the UUID of the package descriptor (which would contain an extra field in its metadata, indicating the package file UUID). Otherwise... how can the descriptor UUID be found?

[mpeuster]

Thats the point and also my view. (see the mail that I am about to send out).

[mpeuster]

Issue is resolved. See mail.

[panstav1]

After the WP3 discussion, below is the updated format of the additional file mapping to help catalogue. The additions are the list of testds trios + a field "files" which includes the file_id, returned from the Catalogue when the individual file is uploaded, and the correspondig filename. @mpeuster and @jbonnet what's your view on this?

{
	"pd" : {
			"vendor" : "eu.sonata-nfv.package",
			"version" : "0.3.1",
			"name" : "sonata-demo"
		},
	"nsds" : [ 
			{
				"vendor" : "eu.sonata-nfv.service-descriptor",
				"version" : "0.2.1",
				"name" : "sonata-demo"
			}
		],
	"vnfds" : [ 
			{
				"vendor" : "eu.sonata-nfv",
				"version" : "0.3",
				"name" : "firewall-vnf"
			}, 
			{
				"vendor" : "eu.sonata-nfv",
				"version" : "0.2",
				"name" : "iperf-vnf"
			}, 
			{
				"vendor" : "eu.sonata-nfv",
				"version" : "0.2",
				"name" : "tcpdump-vnf"
			}
		],
	"deps" : [],
	"testds" : [
				{
					"name" : "simple-ttcn3-test",
					"vendor" : "eu.5gtango.huawei.simple-test-descriptor",
					"version" : "0.1"
				}
				],
	"files" : [ 
					{ 
						"file_uuid" : "71cdd0e1-d7h9-6641-kgm8-7969c202b76"
						"file_name" : "Example.ext"
					}
					],
	"tgo_package_uuid" : "78cdd0e2-e3d4-4789-afc7-56269c202b76"
}

[mpeuster]

Looks ok for me. Just give me the endpoint to send this data, once it is implemented.

Same goes for the endpoint to upload arbitrary files (Example.ext). Once the catalogues support it, I add it to the packager.

[jbonnet]

@panstav1 Just a minor-minor thing: file_name instead of filename? Or fileuuid instead of file_uuid :)

[panstav1]

Updated to file_name :)

[mpeuster]

Ok, almost there. This is an example of what I will send to the mappings endpoint if the WP3 test package is unpacked (just to let you see a typical request):

{  
    "pd":{  
        "vendor":"eu.5gtango",
        "name":"simple-ttcn3-test",
        "version":"0.1"
    },
    "nsds":[  

    ],
    "vnfds":[  

    ],
    "testds":[  
        {  
            "vendor":"eu.5gtango.huawei.simple-test-descriptor",
            "name":"simple-ttcn3-test",
            "version":"0.1"
        }
    ],
    "deps":[  

    ],
    "files":[  
        {  
            "file_uuid":"83322dc5-1af5-462b-bb95-2c7718e8bff7",
            "file_name":"upb_logo.png"
        },
        {  
            "file_uuid":"53372453-acc5-4327-9c32-efb1736f907c",
            "file_name":"LICENSE"
        },
        {  
            "file_uuid":"c380458a-9721-4ac3-a1f5-e80f8043ba56",
            "file_name":"MyExample"
        },
        {  
            "file_uuid":"783a0641-d0ac-4042-8f72-10d129a7574c",
            "file_name":"MyExample.cfg"
        },
        {  
            "file_uuid":"32c0a521-35cc-44f2-8beb-827c72151db1",
            "file_name":"runner.sh"
        }
    ],
    "tgo_package_uuid":"123dafea-1060-4781-b86d-3e84e29c8d4c"
}

[jbonnet]

It starts looking great! 👍
@panstav1 We'll need an example of the response....

[mpeuster]

Ok, run in an error when uploading this mapping, the tng-cat returns:

File with {name => upb_logo.png, uuid => 7ea6cde0-63cd-4054-a002-b64eb8e0ea91} not found in the Catalogue

ecen though a call to curl -H "Content-type:application/x-yaml" http://localhost:4011/api/catalogues/v2/files gives:

- created_at: '2018-05-04T11:41:27.502+00:00'
  grid_fs_id: 5aec46e791521800010000e4
  grid_fs_name: upb_logo.png
  md5: 6e02d6a13e57d4dc114da4a24961f2ae
  signature:
  updated_at: '2018-05-04T11:41:27.502+00:00'
  username:
  uuid: 7ea6cde0-63cd-4054-a002-b64eb8e0ea91

The used mapping is:

{"pd": {"vendor": "eu.5gtango", "name": "simple-ttcn3-test", "version": "0.1"}, "nsds": [], "vnfds": [], "testds": [{"vendor": "eu.5gtango.huawei.simple-test-descriptor", "name": "simple-ttcn3-test", "version": "0.1"}], "deps": [], "files": [{"file_uuid": "7ea6cde0-63cd-4054-a002-b64eb8e0ea91", "file_name": "upb_logo.png"}, {"file_uuid": "0de595d2-af6b-4376-ad07-bce53bb64e1a", "file_name": "LICENSE"}, {"file_uuid": "7122c971-dac4-4f8a-8a0c-ac1828a05c21", "file_name": "MyExample"}, {"file_uuid": "53ead82d-824f-4670-8e11-2c483a163980", "file_name": "MyExample.cfg"}, {"file_uuid": "5023475b-1747-4446-b835-5a025a59c2bb", "file_name": "runner.sh"}], "tgo_package_uuid": "2990f855-18c0-4dc2-8455-f055274abf76"}

Any ideas?

[mpeuster]

Fixed in tng-cat now. This means tng-cat integration is almost done. Needs to be tested a bit more next week.

[panstav1]

Firstly, the tng-cat checks the existence of the tgo_package which is mentioned in the mapping file. Then, The mechanism checks for the existence of all files. Now it is fixed. In case of every file being present in the Catalogue, the response is a 200 code with the message "Updated mappings of <tgo_package_uuid>" In case of a file being absent, the tng-cat return a 400 code with a message implying the {name, vendor, version} trio (for descriptors) or {file_name,file_uuid} (for files). So, suppose a VNF descriptor is missing and being referenced in a mapping file, the return would be a 400 code with a message "VNF Descriptor with {name => , vendor => , version => } not found in the Catalogue". Also, for files, a 400 code with "File with {name => <file_name>, uuid => <file_uuid>} not found in the Catalogue"