-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PR] Addressing edge cases #86
Conversation
This should be reviewable now 🆗 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Going to make a couple of adjustments. 🧑💻
Thanks for proactively opening this PR. 👍
On my localhost, ran the following:
Got:
|
Does your
|
It seems the bucket env variables are not being correctly loaded when deploying, according to the logs on [error] ** (RuntimeError) Elixir.ExAws.Operation.ExAws.Operation.S3.perform/2 cannot perform operation on `nil` bucket |
The buckets need to be set on Fly.io: https://fly.io/apps/imgup/secrets The values of the Bucket Environment Variables aren't secret; they appear in the URL ... 🔗 😉
But still getting the error: Couldn't upload files to S3. Open an issue on Github and contact the repo owner. as noted in: #69 (comment) |
With the env vars as required:
Still get: mix c
1) test upload with image keyword (AppWeb.APITest)
test/app_web/api_test.exs:64
Assertion with == failed
code: assert Jason.decode!(response(conn, 200)) == expected
left: %{"compressed_url" => "https://s3.eu-west-3.amazonaws.com//zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png", "url" => "https://s3.eu-west-3.amazonaws.com/imgup-original/zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png"}
right: %{
"compressed_url" => "https://s3.eu-west-3.amazonaws.com/imgup-compressed/zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png",
"url" => "https://s3.eu-west-3.amazonaws.com/imgup-original/zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png"
}
stacktrace:
test/app_web/api_test.exs:74: (test)
....
2) test upload/1 happy path REAL Upload (App.UploadTest)
test/app/upload_test.exs:4
Assertion with == failed
code: assert App.Upload.upload(image) == {:ok, expected_response}
left: {:ok, %{compressed_url: "https://s3.eu-west-3.amazonaws.com//zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png", url: "https://s3.eu-west-3.amazonaws.com/imgup-original/zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png"}}
right: {
:ok,
%{
compressed_url: "https://s3.eu-west-3.amazonaws.com/imgup-compressed/zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png",
url: "https://s3.eu-west-3.amazonaws.com/imgup-original/zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png"
}
}
stacktrace:
test/app/upload_test.exs:18: (test)
3) test upload succeeds (happy path) (AppWeb.APITest)
test/app_web/api_test.exs:51
Assertion with == failed
code: assert Jason.decode!(response(conn, 200)) == expected
left: %{"compressed_url" => "https://s3.eu-west-3.amazonaws.com//zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png", "url" => "https://s3.eu-west-3.amazonaws.com/imgup-original/zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png"}
right: %{
"compressed_url" => "https://s3.eu-west-3.amazonaws.com/imgup-compressed/zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png",
"url" => "https://s3.eu-west-3.amazonaws.com/imgup-original/zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png"
}
stacktrace:
test/app_web/api_test.exs:61: (test)
........
Finished in 1.2 seconds (1.1s async, 0.1s sync)
15 tests, 3 failures
Randomized with seed 88516
----------------
COV FILE LINES RELEVANT MISSED
100.0% lib/app.ex 9 0 0
100.0% lib/app/repo.ex 5 0 0
100.0% lib/app/upload.ex 108 20 0
100.0% lib/app_web/components/layouts.ex 5 0 0
100.0% lib/app_web/controllers/api_controller.e 46 10 0
100.0% lib/app_web/controllers/api_json.ex 20 3 0
100.0% lib/app_web/controllers/page_controller. 25 5 0
100.0% lib/app_web/controllers/page_html.ex 5 0 0
100.0% lib/app_web/endpoint.ex 48 0 0
100.0% lib/app_web/live/imgup_live.ex 93 26 0
100.0% lib/app_web/router.ex 30 4 0
100.0% lib/app_web/s3_upload.ex 127 24 0
[TOTAL] 100.0%
----------------
Generating report...
Saved to: cover/ The URL in the tests is: https://s3.eu-west-3.amazonaws.com//zb2rhXACvyoVCaV1GF5ozeoNCXYdxcKAEWvBTpsnabo3moYwB.png The bucket is being omitted ... 😕 @LuchoTurtle happy to show you this at my desk. |
Ok. We are looking at two separate (but apparently related) issues here.
I'm going to figure out ...
HOMEBREW_PREFIX=/opt/homebrew
HOMEBREW_CELLAR=/opt/homebrew/Cellar
HOMEBREW_REPOSITORY=/opt/homebrew
MANPATH=/opt/homebrew/share/man::
INFOPATH=/opt/homebrew/share/info:
ANDROID_HOME=/Users/n/Library/Android/sdk
AWS_ACCESS_KEY_ID=AWS
AWS_SECRET_ACCESS_KEY=aitookourjobs
AWS_REGION=eu-west-3
AWS_ORIGINAL_BUCKET=imgup-original
AWS_COMPRESSED_BUCKET=imgup-compressed
_=/usr/bin/printenv Checked the spelling on the env vars Ran: Got the following warning: warning: Application.get_env/2 is discouraged in the module body, use Application.compile_env/3 instead
lib/app/upload.ex:7: App.Upload This relates to the following lines: But the Good News is that tests work on mix c
...............
Finished in 1.5 seconds (1.4s async, 0.1s sync)
15 tests, 0 failures
Randomized with seed 591967
----------------
COV FILE LINES RELEVANT MISSED
100.0% lib/app.ex 9 0 0
100.0% lib/app/repo.ex 5 0 0
100.0% lib/app/upload.ex 108 20 0
100.0% lib/app_web/components/layouts.ex 5 0 0
100.0% lib/app_web/controllers/api_controller.e 46 10 0
100.0% lib/app_web/controllers/api_json.ex 20 3 0
100.0% lib/app_web/controllers/page_controller. 25 5 0
100.0% lib/app_web/controllers/page_html.ex 5 0 0
100.0% lib/app_web/endpoint.ex 48 0 0
100.0% lib/app_web/live/imgup_live.ex 93 26 0
100.0% lib/app_web/router.ex 30 4 0
100.0% lib/app_web/s3_upload.ex 127 24 0
[TOTAL] 100.0%
----------------
Generating report...
Saved to: cover/ Now looking into "Part 2" on |
FYI: I've renamed the two AWS_COMPRESSED_BUCKET-> AWS_S3_BUCKET_ORIGINAL
AWS_COMPRESSED_BUCKET -> AWS_S3_BUCKET_COMPRESSED The reason is:
I've updated the variables in the project, CI and Fly.io. flyctl secrets set AWS_S3_BUCKET_ORIGINAL=imgup-original -a imgup
flyctl secrets set AWS_S3_BUCKET_COMPRESSED=imgup-compressed -a imgup Removed the "old" ones to avoid confusion: fly secrets unset AWS_ORIGINAL_BUCKET -a imgup
fly secrets unset AWS_COMPRESSED_BUCKET -a imgup https://fly.io/apps/imgup/secrets FYI: the two bucket names were not in the So added them. ✅ |
```elixir | ||
def upload(image) do | ||
# Create `CID` from file contents so filenames are unique | ||
case File.read(image.path) do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@LuchoTurtle you've shifted the logic without simplifying it.
This is worse than inlining it.
And without any logging this code is basically a black box when debugging.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added comment on this in #86 (comment).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not happy with writing a mock
or code
for an unspecified AWS S3
failure event that we cannot test for.
It's totally fine to have a generic failure event in this exceptionally unlikely event of there being an error with S3
.
Generally speaking, a Software product/app that calls many external APIs in the "backend"
does not expose these details to the client/consumer.
Unless the client already knows those services are being used, e.g. Google Auth
In the case of S3
the image URL
currently has S3
in the address but it definitely won't for ever ...
The last S3
outage was in 2017: https://aws.amazon.com/message/41926/
They are exceedingly rare and if we cannot test the scenario, to all intents and purposes, it doesn't exist.
In the "Mars Rover" analogy: if the uplink back to
NASA HQ
is down
the lander isn't going to send a messages back toNASA HQ
saying the channel is broken because they already know and what channel would they use ... 🤷♂️
If an error occurs with S3
we just return a generic:
"Sorry uploads are not working right now, please try again later."
Or if we want to get fancy, we return a "trace ID" that allows the engineers to immediately find that request in the logs.
See: https://www.w3.org/TR/trace-context/
We can (should) log
the error in our infra so that we know why it happened.
And when we eventually have a "status" page we can write a quick post explaining what the root cause is so people how how we fixed it.
But adding mocks
to the project just so that we can pretend to test for this black swan event is not useful.
We don't yet have a "style guide" for our Elixir
code.
But if we did the with
keyword and <-
(reverse arrow) notation would definitely be on the "No" list.
They make the code terse without any real benefit other than masking complexity of implementation.
with {:ok, {file_cid, file_extension}} <- check_file_binary_and_extension(image),
{:ok, {file_name, upload_response_body}} <-
upload_file_to_s3(file_cid, file_extension, image) do
This is just shifting business logic into one line and requires significant time for the (human) reader to parse.
"Go do all these things and if anything fails return a bunch of errors..."
Then putting a bunch of matches for those errors:
{:error, :failure_read} -> {:error, :failure_read}
{:error, :invalid_extension_and_cid} -> {:error, :invalid_extension_and_cid}
{:error, :invalid_cid} -> {:error, :invalid_cid}
{:error, :invalid_extension} -> {:error, :invalid_extension}
{:error, :upload_fail} -> {:error, :upload_fail}
If you aren't actually doing anything with the error message/type,
then just pass it straight through with a generic "reason":
{:error, reason} -> {:error, reason}
Otherwise you are duplicating logic unnecessarily.
I don't like this code at all. Because all you have done is move the complexity elsewhere.
One of the biggest problems smart people have is they write complex code.
Code that will confuse and overwhelm a beginner.
This is fine in a close source 2-person project.
But not desirable in an Open Source project where we want others to read and contribute to the code.
I don't want to just close this PR and waste both my own time and Lucho's time. ⏳ 💸 🔥 |
For future reference, whenever you are tempted to put the word "and" in a function name, don't. |
@LuchoTurtle I've already removed (commented out) the offending |
…dge-cases-#69 # Conflicts: # lib/app_web/controllers/api_controller.ex # test/app_web/api_test.exs
I've made some changes, namely:
I didn't change "code smells" like
Perhaps my Elixir syntax skills are lacking but I don't know how to make the code more readable and simpler without doing these. Assigning back to you, I've made the changes according to the feedback and commented on those that were affected. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @LuchoTurtle 👌
All good enhancements. ✅
Hopefully we can focus on features requested by people
using the product. 🤞
Cool. Thanks. |
closes #69
This adds pattern matching and returns the person meaningful error messages according to whether there was an error:
path
from theimage
.CID
from the file contents.file extension
.S3
.I had to use
mock
to force therequest
toS3
to fail in tests and to emulate invalid binary data soCID
would also fail, as detailed in #69.