Science Parse parses scientific papers (in PDF form) and returns them in structured form. As of today, it supports these fields:
- Sections (each with heading and body text)
- Bibliography, each with
- Mentions, i.e., places in the paper where bibliography entries are mentioned
New version: SPv2
There is a new version of science-parse out that works in a completely different way. It has fewer features, but higher quality in the output. Check out the details at https://github.com/allenai/spv2.
There are three different ways to get started with SP. Each has its own document:
- Server: This contains the SP server. It's useful for PDF parsing as a service. It's also probably the easiest way to get going.
- CLI: This contains the command line interface to SP. That's most useful for batch processing.
- Core: This contains SP as a library. It has all the extraction code, plus training and evaluation. Both server and CLI use this to do the actual work.
How to include into your own project
The current version is
3.0.0. If you want to include it in your own project, use this:
libraryDependencies += "org.allenai" %% "science-parse" % "3.0.0"
<dependency> <groupId>org.allenai</groupId> <artifactId>science-parse_2.12</artifactId> <version>3.0.0</version> </dependency>
The first time you run it, SP will download some rather large model files. Don't be alarmed! The model files are cached, and startup is much faster the second time.
For licensing reasons, SP does not include libraries for some image formats. Without these libraries, SP cannot process PDFs that contain images in these formats. If you have no licensing restrictions in your project, we recommend you add these additional dependencies to your project as well:
"com.github.jai-imageio" % "jai-imageio-core" % "1.2.1", "com.github.jai-imageio" % "jai-imageio-jpeg2000" % "1.3.0", // For handling jpeg2000 images "com.levigo.jbig2" % "levigo-jbig2-imageio" % "1.6.5", // For handling jbig2 images
This project is a hybrid between Java and Scala. The interaction between the languages is fairly seamless, and SP can be used as a library in any JVM-based language.
Our build system is sbt. To build science-parse, you have to have sbt installed and working. You can find details about that at https://www.scala-sbt.org.
Once you have sbt set up, just start
sbt in the main project folder to launch sbt's shell. There
are many things you can do in the shell, but here are the most important ones:
+testruns all the tests in all the projects across Scala versions.
cli/assemblybuilds a runnable superjar (i.e., a jar with all dependencies bundled) for the project. You can run it (from bash, not from sbt) with
java -Xmx10g -jar <location of superjar>.
server/assemblybuilds a runnable superjar for the webserver.
server/runstarts the server directly from the sbt shell.
Lombok has a lot of useful annotations that give you some of the nice things in Scala:
valis equivalent to
finaland the right-hand-side class. It gives you type-inference via some tricks
- Check out
Special thanks goes to @kermitt2, whose work on kermitt2/grobid inspired Science Parse, and helped us get started with some labeled data.
Releasing new versions
This project releases to BinTray. To make a release:
- Pull the latest code on the master branch that you want to release
- Tag the release
git tag -a vX.Y.Z -m "Release X.Y.Z"replacing X.Y.Z with the correct version
- Push the tag back to origin
git push origin vX.Y.Z
- Release the build on Bintray
sbt +publish(the "+" is required to cross-compile)
- Verify publication on bintray.com
- Bump the version in
build.sbton master (and push!) with X.Y.Z+1 (e.g., 2.5.1 after releasing 2.5.0)
If you make a mistake you can rollback the release with
sbt bintrayUnpublish and retag the
version to a different commit as necessary.