Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PR] Adding API that consumes image files #61

Merged
merged 34 commits into from
Jun 14, 2023
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
c4076a6
feat: Adding homepage and README guides. #53
LuchoTurtle Jun 5, 2023
144c9d5
feat: Adding response controller to API requests. #53
LuchoTurtle Jun 5, 2023
52a3a7a
feat: Adding file upload with filetype and size limiting. #53
LuchoTurtle Jun 6, 2023
95a6d9a
feat: Adding README. #53
LuchoTurtle Jun 6, 2023
47a2c6c
feat: Adding public ACL read and URL. #53
LuchoTurtle Jun 6, 2023
a8e0c23
feat: Adding tests. #53
LuchoTurtle Jun 7, 2023
240e30a
feat: Adding CID. #53
LuchoTurtle Jun 7, 2023
aa5e528
feat: Add API info to homepage. #53
LuchoTurtle Jun 7, 2023
4c932ca
Merge branch 'master' into api-#53
LuchoTurtle Jun 7, 2023
fc5b4bb
fix minor typo in api.md
nelsonic Jun 7, 2023
eacf854
fix: Removing dead code. Adding region config. Commenting xpath usage.
LuchoTurtle Jun 12, 2023
2bb3c80
split uploading into a separate function
nelsonic Jun 13, 2023
4ffce22
abstract S3 upload code into separate library lib/app/upload.ex https…
nelsonic Jun 13, 2023
147f6e6
create test/app/upload_test.exs to test the generic upload function h…
nelsonic Jun 13, 2023
374ce3a
substantially simplify api_controller and actually test everything. #53
nelsonic Jun 13, 2023
52b6cbd
add AWS_ACCESS_KEY_ID to .github/workflows/ci.yml fixes #67
nelsonic Jun 14, 2023
c3723ad
attempt to use aws-actions/configure-aws-credentials@v2 to set AWS en…
nelsonic Jun 14, 2023
1070fcb
comment out echo set-env https://github.com/dwyl/imgup/actions/runs/5…
nelsonic Jun 14, 2023
114d917
define role-to-assume in ci.yml #67
nelsonic Jun 14, 2023
bd8bac7
use secrets.CI_AWS_SECRET_ACCESS_KEY #67
nelsonic Jun 14, 2023
741e525
use secrets.CI_AWS_SECRET_ACCESS_KEY #67
nelsonic Jun 14, 2023
2db960f
comment out "aws-actions/configure-aws-credentials@v2" #67
nelsonic Jun 14, 2023
06a6254
move environment variable definition higher in ci.yml #67
nelsonic Jun 14, 2023
e3a2da5
System.get_env("CI_AWS_SECRET_ACCESS_KEY") #67
nelsonic Jun 14, 2023
6bba2cf
replace System.fetch_env!/1 with System.get_env/1 #67
nelsonic Jun 14, 2023
6791bf3
replace System.fetch_env!/1 with System.get_env/1 https://github.com/…
nelsonic Jun 14, 2023
82b55fd
replace System.fetch_env!/1 with System.get_env/1 https://github.com/…
nelsonic Jun 14, 2023
ce7ac23
ensure env in ci.yml #67
nelsonic Jun 14, 2023
5bcafb0
secrets.CI_AWS_SECRET_ACCESS_KEY > secrets.AWS_SECRET_ACCESS_KEY #67
nelsonic Jun 14, 2023
187e33a
remove redundant config in test.exs #67
nelsonic Jun 14, 2023
942134d
fix config typo #67
nelsonic Jun 14, 2023
2f50de8
add instructions for testing with Hoppscotch #53 https://github.com/d…
nelsonic Jun 14, 2023
40e95f4
add upload/1 test to api.md https://github.com/dwyl/imgup/pull/61#dis…
nelsonic Jun 14, 2023
bce4d47
replace System.fetch_env! with System.get_env in README.md for consis…
nelsonic Jun 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,4 @@ npm-debug.log
/assets/node_modules/

.env
.vscode/launch.json
392 changes: 392 additions & 0 deletions api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,392 @@
<div align="center">
nelsonic marked this conversation as resolved.
Show resolved Hide resolved

# Uploading files in `Phoenix` through an API

Learn how to do image uploads from a `Phoenix` API!

</div>

> Before reading this guide,
> please follow the instructions in [`README.md`](./README.md),
> as this guide *continues* the progress made on it.


# 1. Add `/api` scope and pipeline and setting up controller

Let's create our API endpoint.
Open `lib/router.ex`
and uncomment the `pipeline :api` and `scope :api`.
We are going to set our endpoint to create a given image
that is sent via `multipart/form-data`.

```elixir

pipeline :api do
plug :accepts, ["json"]
end

scope "/api", AppWeb do
pipe_through :api

resources "/images", ApiController, only: [:create]
end
```

We need to create our `ApiController`
to serve these requests.
Inside `lib/app_web/controllers`,
create `api_controller.ex`
and paste the following code.

```elixir
defmodule AppWeb.ApiController do
use AppWeb, :controller
alias App.Todo

def create(conn, params) do
render(conn, :create)
end

end
```

We're yet to serve anything,
we'll do this at a later stage.

To render a `json` response,
let's create a simple JSON template.
In the same folder, create `api_json.ex`.

```elixir
defmodule AppWeb.ApiJSON do
def render("create.json", _assigns) do
%{url: "Some URL"}
end

def render(template, _assigns) do
%{errors: %{detail: Phoenix.Controller.status_message_from_template(template)}}
end
end
```

Now, depending on the status of the response,
we will render a sample response
`%{url: "Some URL"}`.
Don't worry,
we'll change this with the public URL
after implementing the feature
that uploads the image file to `S3`.


# 2. Uploading to `S3` bucket

In order to upload the file to an `S3` bucket,
we are going to make use of the
[`ex_aws`](https://github.com/ex-aws/ex_aws) package.
Let's install it by adding the following lines
to the `deps` section in `mix.exs`.

```elixir
{:ex_aws, "~> 2.0"},
{:ex_aws_s3, "~> 2.0"},
{:sweet_xml, "~> 0.7"}
```

Run `mix deps.get` to download the dependencies.

Next, we need to add configuration
of this newly added dependencies
in `config/config.ex`.
Open it and add these lines.

```elixir
config :ex_aws,
access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY"),
region: "eu-west-3"
LuchoTurtle marked this conversation as resolved.
Show resolved Hide resolved
```

This configuration is quite self-explanatory.
We are setting the default region to the `S3` bucket location,
as well as setting our `access_key_id` and `secret_access_key`
from the env variables we were using earlier.

Now let's upload our files to `S3`!
In `lib/app_web/controllers/api_controller.ex`,
change it to the following piece of code.

```elixir
defmodule AppWeb.ApiController do
use AppWeb, :controller


def create(conn, %{"image" => image}) do

# Upload to S3
upload = image.path
|> ExAws.S3.Upload.stream_file()
|> ExAws.S3.upload("imgup-original", image.filename, acl: :public_read)
|> ExAws.request

# Check if upload was successful
case upload do
{:ok, _body} ->
render(conn, :success)

{:error, error} ->

{_error_atom, http_code, body} = error
render(conn |> put_status(http_code), body)
end

end

def create(conn, _params) do
render(conn |> put_status(400), :field_error)
end

end
```

We are pattern matching the request
so the person *has to* send the file
with a field named `image`.
If they don't, a Bad Request response is yielded.

If they do this correctly,
we use `S3.upload` to send the file
to the `imgup-original` bucket we've created previously
with the filename of the image.

Depending on the result of this upload,
we return a success or error response.
For this,
we ought to make some changes to how the `json` response is rendered.
Open `lib/app_web/controllers/api_json.ex`
and change it so it has the following functions.

```elixir
def render("success.json", _assigns) do
%{url: "Some URL"}
end

def render("field_error.json", _assigns) do
%{errors: %{detail: "No \'image'\ field provided."}}
end

def render(template, assigns) do
body = Map.get(assigns, :body, "Internal server error.")
%{errors: %{detail: body}}
end
```

We are adding two clauses:
- the `field_error.json` is invoked
when the pattern matches to the default,
meaning the person passed a field named *not* `image`.
- a default template that uses
the error coming from the `ex_aws` upload,
using its output to show the error details to the person.


# 3. Limiting filetype and size

We want the clients of our API
to only upload fairly lightweight files
and only images.
So let's limit our API's behaviour!

To limit the size,
simple open `lib/app_web/endpoint.ex`
and add the following attribute to the
`plug Plug.Parsers`.

```elixir
plug Plug.Parsers,
parsers: [:urlencoded, :multipart, :json],
pass: ["*/*"],
json_decoder: Phoenix.json_library(),
length: 5_000_000 # Add this new line
```

We are limiting the person to only upload
files up to `5MB`.

Now, let's limit the uploads to only image files!
Luckily for us, this is fairly simple!
The [`Plug.Upload`](https://hexdocs.pm/plug/1.14.0/Plug.Upload.html#types)
that is automatically parsed in our API
(the `image` variable)
already has a field called `content_type`,
which we can use to check if the file is an image.

For this,
open `lib/app_web/controllers/api_controller.ex`
and change the `def create(conn, %{"image" => image})` function
to:

```elixir
def create(conn, %{"image" => image}) do

# Check if file is an image
fileIsAnImage = String.contains?(image.content_type, "image")

if fileIsAnImage do

# Upload to S3
upload = image.path
|> ExAws.S3.Upload.stream_file()
|> ExAws.S3.upload("imgup-original", image.filename, acl: :public_read)
|> ExAws.request

# Check if upload was successful
case upload do
{:ok, _body} ->
render(conn, :success)

{:error, error} ->

{_error_atom, http_code, body} = error
render(conn |> put_status(http_code), body)
end

# If it's not an image, return 400
else
render(conn |> put_status(400), %{body: "File is not an image."})
end

end
```

Now we are checking the filetype of the uploaded file
and providing feedback *back to the person*!


# 4 Returning the URL to the person

On success,
it might be useful for the person using the API
to get the public link where the image is stored.
For this,
we simply need to **parse the XML response from the `S3` upload**.
For this, visit `lib/app_web/controllers/api_controller.ex`
and change the `def create(conn, %{"image" => image})`
to this:

```elixir
def create(conn, %{"image" => image}) do

# Check if file is an image
fileIsAnImage = String.contains?(image.content_type, "image")

if fileIsAnImage do

# Upload to S3
upload = image.path
|> ExAws.S3.Upload.stream_file()
|> ExAws.S3.upload("imgup-original", image.filename, acl: :public_read)
|> ExAws.request

# Check if upload was successful
case upload do
{:ok, body} ->
url = body.body |> xpath(~x"//text()") |> List.to_string()
render(conn, :success, %{url: url})

{:error, error} ->

{_error_atom, http_code, body} = error
render(conn |> put_status(http_code), body)
end

# If it's not an image, return 400
else
render(conn |> put_status(400), %{body: "File is not an image."})
end

end
```

We are using
[`sweet_xml`](https://github.com/kbrw/sweet_xml)
we've imported earlier in our dependencies list.
This will allow us to parse the output from the `S3` upload,
which is in `XML` format.
It has a `<Location>` tag,
which is the URl we are interested in returning to the user.

We then pass this URL to the `render/3` function.
All we need to do is change it
to *return this URL to the person*.

Open `lib/app_web/controllers/api_json.ex`
and change to.

```elixir
def render("success.json", assigns) do
%{url: assigns.url}
end
```

Awesome!
Now if you run `mix phx.server`
and make a `multipart/form-data` request
(we recommend using a tool like
[`Postman`](https://www.postman.com/) or [`Hoppscotch`](https://hoppscotch.io/) - which is open-source!),
nelsonic marked this conversation as resolved.
Show resolved Hide resolved
you should see a public URL after loading an image file!

```json
{
"url": "https://s3.eu-west-3.amazonaws.com/imgup-original/115faa2f5cbe273cfc9fbcffd44b7eab.1000x1000x1.jpg"
}
```

If the person makes an invalid input,
he should see error details.
For example,
if you try to upload another file other than an image:

```json
{
"errors": {
"detail": "File is not an image."
}
}
```

Or an image size that's too large,
you'll get an `413 Request Entity Too Large` error.


# 5 Saving the image with the `CID` of its contents

Similarly to what we've done to the `LiveView`,
we want our images to have its name
**as the `CID` of the image's contents**.

To do this, it's quite simple!
Visit `lib/app_web/controllers/api_controller.ex`
and change the `create` function like so:

```elixir
if fileIsAnImage do

# Create `CID` from file contents
{:ok, file_binary} = File.read(image.path)
file_cid = Cid.cid(file_binary)
file_name = "#{file_cid}.#{Enum.at(MIME.extensions(image.content_type), 0)}"

# Upload to S3
upload = image.path
|> ExAws.S3.Upload.stream_file()
|> ExAws.S3.upload("imgup-original", file_name, acl: :public_read)
|> ExAws.request(get_ex_aws_request_config_override())
```

We are creating a `cid` from the contents of the image.
We then use this `cid` and concatenate with the extension of the
*content type of the image*.
This way, we'll have a cid with the correct format,
e.g. `zb2rhhPShfsYqzqYPG8wxnsb4zNe2HxDrqKRxU6wSWQQWMHsZ.jpg`.


Loading