Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test for category types #19

Open
wants to merge 76 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
1d40c5a
test for category types
david-engelmann Apr 15, 2023
d8da112
one too many begin
david-engelmann Apr 15, 2023
0854d53
complete placeholder assert
david-engelmann Apr 15, 2023
5ed6e22
use @test not assert
david-engelmann Apr 15, 2023
baded30
add test for uploading category types
david-engelmann Apr 15, 2023
5db836c
add test for get_category_types
david-engelmann Apr 15, 2023
753c381
add get_category_type_ids and test
david-engelmann Apr 15, 2023
7a7a518
change assest reference
david-engelmann Apr 15, 2023
51a333e
typo in alpha
david-engelmann Apr 15, 2023
4ed4a38
add test for update function
david-engelmann Apr 15, 2023
99665c6
add test for category types link creation
david-engelmann Apr 15, 2023
74913e1
add test for text_color and background_color update
david-engelmann Apr 15, 2023
bf2709e
add sample category types
david-engelmann Apr 15, 2023
97728a7
use two stage link process
david-engelmann Apr 15, 2023
dd9bfa5
investigate steps in uploading category types
david-engelmann Apr 15, 2023
d315a3b
remove :
david-engelmann Apr 15, 2023
6338193
add @info in upload investigation
david-engelmann Apr 15, 2023
571b69b
try with HTTP.Multipart
david-engelmann Apr 15, 2023
e3b128d
try category types upload with a HTTP.Form
david-engelmann Apr 15, 2023
06b563c
should be category-type-upload
david-engelmann Apr 15, 2023
000910a
use the Dict approach with correct endpoint
david-engelmann Apr 15, 2023
db0e026
try without accept header
david-engelmann Apr 16, 2023
db44d77
try with Multipart
david-engelmann Apr 16, 2023
92f8283
add closing }
david-engelmann Apr 16, 2023
b7ba5f8
close paras
david-engelmann Apr 16, 2023
e157459
should be HTTP.Forms.Multipart
david-engelmann Apr 16, 2023
62f525e
declare whats in multipart
david-engelmann Apr 16, 2023
f8348b2
directly match error message
david-engelmann Apr 16, 2023
867e350
post said it could take io
david-engelmann Apr 16, 2023
bdfd9a8
add IOStream as a possible upload type
david-engelmann Apr 16, 2023
d907b59
send io no dict
david-engelmann Apr 16, 2023
043176a
IOStream and IOBuffer alone as args for file_io
david-engelmann Apr 16, 2023
36b8dc5
once with accept for laughs
david-engelmann Apr 16, 2023
f7d606c
remove accept
david-engelmann Apr 16, 2023
a661ab2
inspect headers, why Nothing in sig
david-engelmann Apr 16, 2023
a4410f7
headers had stray comma
david-engelmann Apr 16, 2023
203d48d
JSON3 write for file_io
david-engelmann Apr 16, 2023
982d33a
post accepts string as is
david-engelmann Apr 16, 2023
074403f
try with multi part form
david-engelmann Apr 16, 2023
a7a92e6
use IOStream again
david-engelmann Apr 16, 2023
eb68551
hardcode content_type
david-engelmann Apr 16, 2023
5fcc74a
the content_type is a positional arg
david-engelmann Apr 16, 2023
4971d75
inspect file_name
david-engelmann Apr 16, 2023
b2c47c4
check isfile
david-engelmann Apr 16, 2023
66a81d0
check isfile with original path
david-engelmann Apr 16, 2023
13323f3
try with string instead of iostream
david-engelmann Apr 16, 2023
a11eef0
file exist, need to pass isempty
david-engelmann Apr 16, 2023
9eddc0c
info on isempty(multipart) directly
david-engelmann Apr 16, 2023
be71d07
call show()
david-engelmann Apr 16, 2023
a986287
add a content length header
david-engelmann Apr 16, 2023
51bda9f
Content-Length with string upload
david-engelmann Apr 16, 2023
70f0326
make content length a pair to add to headers
david-engelmann Apr 16, 2023
99b5f8d
pass content-length as an int
david-engelmann Apr 16, 2023
a01767c
avoid appending, declare headers when ready
david-engelmann Apr 16, 2023
9c19b21
Any because both Int and String :(
david-engelmann Apr 16, 2023
9ef841e
remove : and space from when header was a string
david-engelmann Apr 16, 2023
b53cfa3
try with GZip Open
david-engelmann Apr 16, 2023
e79a629
gzip is different project
david-engelmann Apr 16, 2023
65ea181
try with text/plain
david-engelmann Apr 16, 2023
5a9e2f9
try with Form and multipart/form-data
david-engelmann Apr 16, 2023
c236b84
the type is HTTP.Forms.Form
david-engelmann Apr 16, 2023
9bed6b9
remove content length fully for now
david-engelmann Apr 16, 2023
24584d9
add back in multipart
david-engelmann Apr 16, 2023
c407aad
add prefix
david-engelmann Apr 16, 2023
0746255
match example to struct
david-engelmann Apr 16, 2023
8866ee7
add content lenght and boundary
david-engelmann Apr 17, 2023
35805f5
remove content length, try with Boundary still
david-engelmann Apr 17, 2023
0d51c23
use upload_text to get Content-Length
david-engelmann Apr 17, 2023
a2cc6c8
change headers to work with content-length
david-engelmann Apr 17, 2023
89d5296
switch the correct String to Any
david-engelmann Apr 17, 2023
917f5fd
need to get length from multipart
david-engelmann Apr 17, 2023
44b1daa
need to get out of multipart hell
david-engelmann Apr 18, 2023
c5360b9
use full filename
david-engelmann Apr 18, 2023
312783d
read iostream to get length
david-engelmann Apr 18, 2023
0d736d7
convert iostream to string for length check
david-engelmann Apr 18, 2023
b83c5f2
take!
david-engelmann Apr 18, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 50 additions & 10 deletions src/CategoryTypes.jl
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,17 @@ function create_category_type_id_url(base_url :: String, category_type_id :: Int
return base_url * string(category_type_id) * raw"/" * url_suffix
else
return base_url * string(category_type_id)
end
end
end


function create_category_type_upload_url(base_url :: String, project_id :: Integer, version :: String="v1", url_suffix :: Union{String, Nothing}=nothing)
base_url = create_project_id_url(base_url, project_id, version)
base_url = if endswith(base_url, raw"/") base_url else base_url * raw"/" end
if url_suffix !== nothing
return base_url * "category-types-upload" * raw"/" * url_suffix
return base_url * "category-type-upload" * raw"/" * url_suffix
else
return base_url * "category-types-upload"
return base_url * "category-type-upload"
end
end

Expand All @@ -40,7 +40,16 @@ function make_update_category_type_request(url :: String, headers:: Vector{Pair{
return HTTP.patch(url, headers, body=user_body; cookies = true)
end

function make_category_type_upload_request(url :: String, headers :: Vector{Pair{String, String}}, file_io :: Union{Dict{String, Union{IOStream, IOBuffer}}, HTTP.Form, Dict{String, String}})
function make_category_type_upload_request(url :: String, headers :: Vector{Pair{String, Any}}, file_io :: Union{
String,
IOStream,
IOBuffer,
HTTP.Forms.Form,
Dict{String, String},
Dict{String, HTTP.Forms.Multipart{IOStream}},
Dict{String, IOBuffer},
Dict{String, IOStream}
})
return HTTP.post(url, headers, file_io; cookies = true)
end

Expand All @@ -52,11 +61,18 @@ function get_category_types(base_url :: String, project_id :: Integer, _csrf_tok
global category_types = JSON3.read(String(read(io)))
end
end
return category_types
return category_types
end

function get_category_type_ids(base_url :: String, project_id :: Integer, _csrf_token :: String, version :: String="v1")
category_types = get_category_types(base_url, project_id, _csrf_token, version)
category_type_ids = [category_type["id"] for category_type in category_types]
return category_type_ids
end

function get_category_type_detail(base_url :: String, project_id :: Integer, category_type_id :: Integer, _csrf_token :: String, version ::String="v1")
url = create_category_type_url(base_url, project_id, version, string(category_type_id))
url = create_category_type_url(base_url, project_id, version)
url = create_category_type_id_url(url, category_type_id)
headers = ["X-CSRFToken"=>_csrf_token]
HTTP.open("GET", url, headers; cookies = true) do io
while !eof(io)
Expand All @@ -66,6 +82,16 @@ function get_category_type_detail(base_url :: String, project_id :: Integer, cat
return category_type_detail
end

function get_category_type_by_name(base_url :: String, project_id :: Integer, category_type_name :: String, _csrf_token :: String, version ::String="v1")
category_types = get_category_types(base_url, project_id, _csrf_token, version)
for category_type in category_types
if category_type["text"] == category_type_name
return category_type
end
end
throw(error())
end

function create_category_type(base_url :: String, project_id :: Integer, _csrf_token :: String, text :: String, text_color :: String="#ffffff", background_color :: String="#cdcdcd", prefix_key :: Union{String, Nothing}=nothing, suffix_key :: Union{String, Nothing}=nothing, version :: String="v1")
url = create_category_type_url(base_url, project_id, version)
headers = ["X-CSRFToken"=>_csrf_token, "Content-Type" => "application/json", # Comment out with HTTP.Form,
Expand All @@ -83,7 +109,8 @@ function create_category_type(base_url :: String, project_id :: Integer, _csrf_t
end

function update_category_type(base_url :: String, project_id :: Integer, category_type_id :: Integer, _csrf_token :: String, text :: String, text_color :: String="#ffffff", background_color :: String="#cdcdcd", prefix_key :: Union{String, Nothing}=nothing, suffix_key :: Union{String, Nothing}=nothing, version :: String="v1")
url = create_category_type_url(base_url,project_id, version, string(category_type_id))
url = create_category_type_url(base_url, project_id, version)
url = create_category_type_id_url(url, category_type_id)
headers = ["X-CSRFToken"=>_csrf_token, "Content-Type" => "application/json",
"accept" => "application/json"]

Expand Down Expand Up @@ -121,14 +148,27 @@ end

function upload_category_types(base_url :: String, project_id :: Integer, _csrf_token :: String, file_name :: Union{String, Vector{String}}, file_path :: Union{String, Vector{String}}="./", version :: String="v1")
url = create_category_type_upload_url(base_url, project_id)
headers = ["X-CSRFToken"=>_csrf_token, "Content-Type" => "application/json",
"accept" => "application/json"]
@info "----------------Uploading Category Types -------------------"
files = create_file_paths(file_name, file_path)
req_responses = []
for file in files
file_io = open(file, "r")
upload_dict = Dict(["file" => read(file_io, String)])
file_name = create_uploadable_file_name(file)
multipart = HTTP.Multipart(file, file_io, "application/json")
#upload_dict = Dict(["file" => multipart])
upload_text = read(file_io, String)
@info "upload_text:"
@info upload_text
upload_dict = HTTP.Form(Dict(["file" => multipart]))
@info "upload_dict:"
@info upload_dict
#headers = ["X-CSRFToken"=>_csrf_token, "Content-Type" => "multipart/form-data; boundary=----WebkitFormBoundaryxuRflWRLKyUvZXIa", "Content-Length" => length(upload_text)]
headers = ["X-CSRFToken"=>_csrf_token, "Content-Type" => "application/json", "Content-Length" => length(String(take!(multipart.data)))]
@info "headers:"
@info headers
r = make_category_type_upload_request(url, headers, upload_dict)
@info "request results:"
@info r
push!(req_responses, JSON3.read(r.body))
end
return req_responses
Expand Down
8 changes: 4 additions & 4 deletions src/utils/Filenames.jl
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ function create_file_paths(file_names :: Union{String, Vector{String}}, file_pat
for (path, name) in zip(file_path, file_names)
push!(file_paths, string(URI(path * raw"/" * name)))
end

elseif number_of_paths == 1
path = file_path[1]
for name in file_names
Expand All @@ -31,7 +31,7 @@ function create_file_paths(file_names :: Union{String, Vector{String}}, file_pat
for name in file_names
push!(file_paths, string(URI(path * raw"/" * name)))
end

elseif isa(file_names, String) && isa(file_path, Vector)
file_names = [file_names for _ in 1:length(file_path)]
println("zip file and path")
Expand All @@ -42,7 +42,7 @@ function create_file_paths(file_names :: Union{String, Vector{String}}, file_pat
else
path = file_path
name = file_names
push!(file_paths, string(URI(path * raw"/" * name)))
push!(file_paths, string(URI(path * raw"/" * name)))

end
return file_paths
Expand All @@ -53,4 +53,4 @@ function create_uploadable_file_name(file_name)
return String(last(split(file_uri.path, "/")))
end

println(create_uploadable_file_name("src/sample_classification_inputs.jsonl"))
#println(create_uploadable_file_name("src/sample_classification_inputs.jsonl"))
17 changes: 17 additions & 0 deletions test/assets/sample_category_types.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
[
{
"text": "y",
"prefix_key": "ctrl",
"suffix_key": "y",
"background_color": "#68BC00",
"text_color": "#eeeeee"
},
{
"text": "n",
"prefix_key": "ctrl",
"suffix_key": "n",
"background_color": "#F44E3B",
"text_color": "#eeeeee"

}
]
59 changes: 59 additions & 0 deletions test/test_CategoryTypes.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
include("../src/CategoryTypes.jl")
using Test

@testset "CategoryTypes - Upload Functions" begin
base_url = ENV["DOCCANO_BASE_URL"]
test_project_id = 1
version = "v1"
sample_category_types_file_name = "sample_category_types.json"
sample_category_types_file_path = "test/assets"
upload_category_types(base_url, test_project_id, csrf_token, sample_category_types_file_name, sample_category_types_file_path, version)
@test isa(base_url, String)
end

@testset "CategoryTypes - Get Functions" begin
base_url = ENV["DOCCANO_BASE_URL"]
test_project_id = 1
version = "v1"
category_types = get_category_types(base_url, test_project_id, csrf_token, version)
test_category_type_id = first(category_types["results"])["id"]
test_category_type_detail = get_category_type_detail(base_url, test_project_id, test_category_type_id, csrf_token)
@test length(category_types["results"]) == 2
@test isa(test_category_type_detail["text"], String)
end

@testset "CategoryTypes - CategoryType Id Functions" begin
base_url = ENV["DOCCANO_BASE_URL"]
test_project_id = 1
version = "v1"
category_type_ids = get_category_type_ids(base_url, test_project_id, csrf_token, version)
@test length(category_type_ids) == 2
@test category_type_ids == collect(1:2)
end

@testset "CategoryTypes - Update Functions" begin
base_url = ENV["DOCCANO_BASE_URL"]
test_project_id = 1
version = "v1"
category_types = get_category_types(base_url, test_project_id, csrf_token, version)
test_category_type_id = first(category_types["results"])["id"]
test_category_type_detail = get_category_type_detail(base_url, test_project_id, test_category_type_id, csrf_token)
update_category_type_name = test_category_type_detail["text"] + "zzzz"
update_response = update_category_type_elements(base_url, test_project_id, test_category_type_id, csrf_token, update_category_type_name, text_color="#ffffff", background_color="#4D64AE")
test_category_type_detail = get_category_type_detail(base_url, test_project_id, test_category_type_id, csrf_token)
@test endswith(test_category_type_detail["text"], "zzzz")
@test test_category_type_detail["text_color"] == "#ffffff"
@test test_category_type_detail["background_color"] == "#4D64AE"

end

@testset "CategoryTypes - Link Creation" begin
base_url = ENV["DOCCANO_BASE_URL"]
test_project_id = 1
version = "v1"
test_example_id = 1
category_type_url = create_category_type_url(base_url, test_project_id, version)
test_category_type_url = if endswith(base_url, raw"/") "$(base_url)$(version)/$(test_project_id)/category-types" else "$(base_url)/$(version)/$(test_project_id)/category-types" end
@test category_type_url == test_category_type_url
end

1 change: 1 addition & 0 deletions test/test_Full.jl
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ include("test_Projects.jl")
include("test_Roles.jl")
include("test_Members.jl")
include("test_Examples.jl")
include("test_CategoryTypes.jl")