Media type rework #1770
Media type rework #1770
Conversation
} | ||
|
||
implicit val http4sInstancesForMediaRange | ||
: Show[MediaRange] with HttpCodec[MediaRange] with Order[MediaRange] = | ||
new Show[MediaRange] with HttpCodec[MediaRange] with Order[MediaRange] { | ||
override def show(s: MediaRange): String = s.toString | ||
override def show(s: MediaRange): String = | ||
s"${s.mainType}/*${MediaRange.extensionsToString(s)}" |
cquiroz
Apr 5, 2018
Author
Member
I've made show
match whats produced by
renderrather than match
toString. this made it easier to remove
Renderable` from the hierarchy
I've made show
match whats produced by
renderrather than match
toString. this made it easier to remove
Renderable` from the hierarchy
rossabaker
Apr 5, 2018
Member
Interesting. This is something we should be consistent on throughout the model: do we want show
to match the wire representation and be part of the HttpCodec
laws?
Interesting. This is something we should be consistent on throughout the model: do we want show
to match the wire representation and be part of the HttpCodec
laws?
override def toString: String = "MediaType(" + renderString + ')' | ||
} | ||
|
||
object MediaType extends Registry { |
cquiroz
Apr 5, 2018
Author
Member
Bye, bye Registry
. But also registerFileExtension
is gone
Bye, bye Registry
. But also registerFileExtension
is gone
rossabaker
Apr 5, 2018
Member
It won't be missed.
It won't be missed.
app("msword", uncompressible, binary, "doc", "dot", "w6w", "wiz", "word", "wri") | ||
val `application/octet-stream` = app( | ||
/////////////////////////// PREDEFINED MEDIA-TYPE DEFINITION //////////////////////////// | ||
// Copied from the definitions on MimeDB |
cquiroz
Apr 5, 2018
Author
Member
Predefined MediaTypes
are copied verbatim from MimeDB
Predefined MediaTypes
are copied verbatim from MimeDB
rossabaker
Apr 5, 2018
Member
I don't understand what's here vs. in MimeDB.
I don't understand what's here vs. in MimeDB.
cquiroz
Apr 5, 2018
Author
Member
I struggled with this a bit. We had several MediaTypes
pinned here and now those are in MimeDB
. There are a few alternatives:
- We remove these altogether and make callers do e.g.
MimeDB.text.text/plain
- In
MediaType
we create types like text/plain
and link them to MimeDB.text.text/plain
- Or we copy the definitions from
MimeDB
to here (The option used)
Ideally, we'd use one of the first options but it gets a bit complex with application
. There are a few thousand application
mime types, and given the JVM code size limits it needs to be split on the code generation part. e.g. application/xml
is MimeDB.application.application_2.application/xml
. Even worse the numbering is not fixed as the mimedb can change so an update of MimeDB
could break code. In option 1 that's not acceptable as we'd require users to change theyr code. option 2 maybe better as we'd need to update the links when MimeDB
changes anyway.
I'm not heavily invested in option 3. we could switch to option 2 if you think is better
I struggled with this a bit. We had several MediaTypes
pinned here and now those are in MimeDB
. There are a few alternatives:
- We remove these altogether and make callers do e.g.
MimeDB.text.text/plain
- In
MediaType
we create types liketext/plain
and link them toMimeDB.text.text/plain
- Or we copy the definitions from
MimeDB
to here (The option used)
Ideally, we'd use one of the first options but it gets a bit complex with application
. There are a few thousand application
mime types, and given the JVM code size limits it needs to be split on the code generation part. e.g. application/xml
is MimeDB.application.application_2.application/xml
. Even worse the numbering is not fixed as the mimedb can change so an update of MimeDB
could break code. In option 1 that's not acceptable as we'd require users to change theyr code. option 2 maybe better as we'd need to update the links when MimeDB
changes anyway.
I'm not heavily invested in option 3. we could switch to option 2 if you think is better
@@ -0,0 +1,6700 @@ | |||
package org.http4s | |||
|
|||
private[http4s] object MimeDB { |
cquiroz
Apr 5, 2018
Author
Member
MimeDB
is checked in rather than auto generated during the build. This means builds are stable but we could be stale with respect to updates on the upstream db.
There are pros/cons about the alternatives:
Generated on build:
- Pros:
- Always up to date
- We don't need to check in this large file
- Cons:
- Difficult to make the generator using http4s/circe without producing conflicts on the build
- The build could break due to upstream changes
- Cannot build it offline (Though in my experiments the downloaded file was cached)
Checked in:
- Pros:
- Stable builds and we can safely refer to
MimeDB
- This large file would totally kill a
Scala.js
version. If we check this in we can have a MimeDB
for scala.js
which is essentially empty
- Cons:
MimeDB
could get stale
- Eventually somebody could edit this
MimeDB
is checked in rather than auto generated during the build. This means builds are stable but we could be stale with respect to updates on the upstream db.
There are pros/cons about the alternatives:
Generated on build:
- Pros:
- Always up to date
- We don't need to check in this large file
- Cons:
- Difficult to make the generator using http4s/circe without producing conflicts on the build
- The build could break due to upstream changes
- Cannot build it offline (Though in my experiments the downloaded file was cached)
Checked in:
- Pros:
- Stable builds and we can safely refer to
MimeDB
- This large file would totally kill a
Scala.js
version. If we check this in we can have aMimeDB
forscala.js
which is essentially empty
- Cons:
MimeDB
could get stale- Eventually somebody could edit this
rossabaker
Apr 5, 2018
Member
I dislike checking in generated code, but I'm a big fan of repeatable builds. A compromise is a generator pinned to a git hash. But that complicates the build, and makes keeping it up to date a chore, so maybe it's the worst of both worlds.
I agree with your pros and cons and don't have a strong opinion. Would love to hear from others here.
I dislike checking in generated code, but I'm a big fan of repeatable builds. A compromise is a generator pinned to a git hash. But that complicates the build, and makes keeping it up to date a chore, so maybe it's the worst of both worlds.
I agree with your pros and cons and don't have a strong opinion. Would love to hear from others here.
jmcardon
Apr 5, 2018
Member
is this every build, or triggered by a command?
is this every build, or triggered by a command?
cquiroz
Apr 5, 2018
Author
Member
At the moment it is just created with a separate utility and manually added to Http4s
repo and formatted
At the moment it is just created with a separate utility and manually added to Http4s
repo and formatted
} | ||
object application { | ||
lazy val all | ||
: List[MediaType] = Nil ::: application.all ::: application_1.all ::: application_2.all |
cquiroz
Apr 5, 2018
Author
Member
This is slightly convoluted. Given that there are several thousand application/
mime types the classes get too large and we hit the JVM 64kb limit. Splitting them this way lets us workaround that error
This is slightly convoluted. Given that there are several thousand application/
mime types the classes get too large and we hit the JVM 64kb limit. Splitting them this way lets us workaround that error
This is neat. I like the gist of it. |
} | ||
|
||
implicit val http4sInstancesForMediaRange | ||
: Show[MediaRange] with HttpCodec[MediaRange] with Order[MediaRange] = | ||
new Show[MediaRange] with HttpCodec[MediaRange] with Order[MediaRange] { | ||
override def show(s: MediaRange): String = s.toString | ||
override def show(s: MediaRange): String = | ||
s"${s.mainType}/*${MediaRange.extensionsToString(s)}" |
rossabaker
Apr 5, 2018
Member
Interesting. This is something we should be consistent on throughout the model: do we want show
to match the wire representation and be part of the HttpCodec
laws?
Interesting. This is something we should be consistent on throughout the model: do we want show
to match the wire representation and be part of the HttpCodec
laws?
override def toString: String = "MediaType(" + renderString + ')' | ||
} | ||
|
||
object MediaType extends Registry { |
rossabaker
Apr 5, 2018
Member
It won't be missed.
It won't be missed.
app("msword", uncompressible, binary, "doc", "dot", "w6w", "wiz", "word", "wri") | ||
val `application/octet-stream` = app( | ||
/////////////////////////// PREDEFINED MEDIA-TYPE DEFINITION //////////////////////////// | ||
// Copied from the definitions on MimeDB |
rossabaker
Apr 5, 2018
Member
I don't understand what's here vs. in MimeDB.
I don't understand what's here vs. in MimeDB.
@@ -0,0 +1,6700 @@ | |||
package org.http4s | |||
|
|||
private[http4s] object MimeDB { |
rossabaker
Apr 5, 2018
Member
I dislike checking in generated code, but I'm a big fan of repeatable builds. A compromise is a generator pinned to a git hash. But that complicates the build, and makes keeping it up to date a chore, so maybe it's the worst of both worlds.
I agree with your pros and cons and don't have a strong opinion. Would love to hear from others here.
I dislike checking in generated code, but I'm a big fan of repeatable builds. A compromise is a generator pinned to a git hash. But that complicates the build, and makes keeping it up to date a chore, so maybe it's the worst of both worlds.
I agree with your pros and cons and don't have a strong opinion. Would love to hear from others here.
How often does MimeDB change? In general I agree with @rossabaker wrt checking in generated code, but for something like this I can definitely see the point in not rebuilding it on every build. So +1 from me on having an "offline" build and checking in the generated code. |
I count six non-build changes so far this year in the upstream repo. |
That's not much and I'd think only a few new items appear. for the most part MIME types are well known. |
private def uncompressible = false | ||
private def binary = true | ||
private def notBinary = false | ||
lazy val `application/javascript`: MediaType = MimeDB.application.`application/javascript` |
cquiroz
Apr 7, 2018
Author
Member
With the improved structure of MimeDB
I feel ok about linking directly here
With the improved structure of MimeDB
I feel ok about linking directly here
@@ -6,6 +6,30 @@ import org.scalacheck.Prop | |||
import scala.util.Success | |||
|
|||
package object testing { | |||
// Media types used for testing | |||
// Copied from the definitions on MimeDB | |||
val `text/css`: MediaType = MimeDB.text.`text/css` |
cquiroz
Apr 7, 2018
Author
Member
Do you think this is a good place to put these definitions?
Do you think this is a good place to put these definitions?
// Autogenerated file, don't edit | ||
package org.http4s | ||
|
||
private[http4s] object MimeDB { |
cquiroz
Apr 7, 2018
Author
Member
Note I've made MimeDB
private. It is the best for MiMa but could make like harder for clients that want to refer to the mime types. WDYT?
Note I've made MimeDB
private. It is the best for MiMa but could make like harder for clients that want to refer to the mime types. WDYT?
rossabaker
Apr 11, 2018
Member
I guess I'm still finding it a little bit weird that these aren't just mixed into the MediaType
companion. What's the disadvantage to that?
I guess I'm still finding it a little bit weird that these aren't just mixed into the MediaType
companion. What's the disadvantage to that?
cquiroz
Apr 11, 2018
Author
Member
do you mean object MediaType extends MimeDB
?
That would be possible. I'll check the changes needed for that
do you mean object MediaType extends MimeDB
?
That would be possible. I'll check the changes needed for that
rossabaker
Apr 11, 2018
Member
Yes, that's exactly what I mean.
Maybe this is a bad idea, because it will lead to thousands of members in a common type. It might be miserable for people doing autocompletes. But it's where I'd look for them.
Yes, that's exactly what I mean.
Maybe this is a bad idea, because it will lead to thousands of members in a common type. It might be miserable for people doing autocompletes. But it's where I'd look for them.
Latest changes |
val updated = current.updated(lcExt, mediaType) | ||
if (!extensionMap.compareAndSet(current, updated)) registerFileExtension(ext, mediaType) | ||
} | ||
object MediaType extends MimeDB { |
cquiroz
Apr 11, 2018
Author
Member
Now MediaType
extends MimeDB
, thus allowing to write things like MediaType.text.text/plain
Now MediaType
extends MimeDB
, thus allowing to write things like MediaType.text.text/plain
super.register(key, mediaType) | ||
mediaType.fileExtensions.foreach(registerFileExtension(_, mediaType)) | ||
mediaType | ||
def multipartType(subType: String, boundary: Option[String] = None): MediaType = { |
cquiroz
Apr 11, 2018
Author
Member
I had to rename this to multipartType
to avoid a collision with MimeDB.multipart
I had to rename this to multipartType
to avoid a collision with MimeDB.multipart
} | ||
|
||
def forExtension(ext: String): Option[MediaType] = extensionMap.get.get(ext.toLowerCase) | ||
// Curiously text/event-stream isn't included in MimeDB | ||
lazy val `text/event-stream` = new MediaType("text", "event-stream") |
cquiroz
Apr 11, 2018
Author
Member
This is a sore point. Should I make the generator create this manually?
This is a sore point. Should I make the generator create this manually?
I hate to be a pain, but do we want: MediaType.audio.`audio/ogg` or MediaType.audio.ogg I think I like the latter. |
How could we move this forward? |
I think it's ready, but ties into #1797: this is a big change, and we might want to defer big changes until 1.0, but master is a mix of small changes and big changes. Where can we put this and not drive ourselves crazy maintaining three branches? |
Sounds good, let’s hold until the right time is due |
We decided to press forward with master. Merging. |
Great! I'll add the generator is another PR to let anybody update |
This is fairly large PR that refactors
MediaType
making it more compatible with the current style, using an auto-generated list of mime types and including typeclasses forMediaType
.Some notes:
Registry
!MediaTypes
is now auto-generated from MimeDB. The fileMimeDB.scala
contains that list on a form amenable tohttp4s
. This fixes #73MimeDB.scala
is generated with a utility athttps://github.com/cquiroz/mime-http4s-generator
. This should likely be transferred to thehttp4s
organization.MimeDB.scala
. I thought about making it part of the build but givenmime-http4s-generator
usehttp4s-client
andcirce
leading to conflicts. Also, I think it's better to have a stable checked in version rather than risking the build to change when the db is modified.MimeTypes
included in theobject MimeType
to the minimal to allow compilation. This may break some user codeMediaRange
andMediaType
are no longerRenderable
MediaTypes
any longer though you can manually create anything you need. This only could affect serving files for unknown file types